Develop a policy for testharness.js changes

[annevk]

Every now and then someone looks at it and considers refactoring it. It would be good if we had some amount of guidelines around what changes would be reasonable.

[foolip]

When it comes to depending on JS language features like Promise I think we should have a default conservative policy of allowing anything that's been supported in stable Chrome+Firefox+Safari for 2 years already, as judged by MDN compat tables:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise#browser_compatibility

But I think we could both avoid features that meet this bar if some user of WPT doesn't support it well and use newer features if everybody is fine with it.

[jgraham]

Right, I think the question is how to find out that "everybody is fine with it" or "some user of WPT doesn't support it well".

It sounds like the process is going to be something like "major engines define a baseline requirement based on their flagship product, and we permit other users to veto given features on a case-by-case basis". Then for the vetos we probably need a documentation file with a list of features, the reason for the veto and ideally a link to an issue.

So for Firefox I think we care about supporting the Extended Support Release (ESR) prior to current ESR (or at least, there's a period when both ESR releases are current which seems to correspond to 2 desktop release cycles per the documentation). In practice we can probably afford to be slightly more aggressive; I don't think anything depends on running a new copy of testharness.js in old ESR releases. But the conservative version is pretty close to the two year cutoff that @foolip proposed given around one ESR per year.

[gsnedders]

So historically some of the motivation here was "running tests on IE is occasionally useful when investigating historic behaviour". It may, of course, be the case that we no longer ever have any desire to do so.

See also web-platform-tests/wpt#6266 and web-platform-tests/wpt#26204.

[foolip]

Since testharness.js tests will no longer work on IE, I think we should drop support for it in wptrunner as well. The same goes for Presto actually.

[gsnedders]

Since testharness.js tests will no longer work on IE, I think we should drop support for it in wptrunner as well. The same goes for Presto actually.

We only support IE for iewebdriver today, I think? I don't think we actually realistically support running anything except wd_spec tests in IE. (And wptrunner has never supported Presto, and can't easily!)

[jgraham]

Yeah, we don't support non-wdspec tests in IE. Opera support is only for Blink-based Opera AFAIK (I think one could get it working in Presto with Selenium, but I haven't actually tried or anything).

In any case I think we're at the point where the amount of pain to continue supporting IE for manually run tests outweighs any value we've got from actually doing that in recent history.

[gsnedders]

I think one could get it working in Presto with Selenium, but I haven't actually tried or anything

OperaPrestoDriver doesn't support execute async script, so you can't (at least short of writing an entirely custom executor)