Mobile Testing Results for Interop/WPT

[mtom55]

Description

A significant issue for the Interop project is that currently only the desktop results are reported on the dashboard and provide the misleading impression that the results are universal between desktop and mobile.

Given that mobile devices make up the vast majority of all personal computers and accounts for the majority of global web traffic, the test results for mobile are significantly more important than the desktop results.

      “The web is amazing. It makes collaborating, learning, and connecting easy for billions of people, because it’s
       intentionally designed to run on radically different devices.

      It’s your job as a web developer to ensure your project works in every browser and for every user — and that can be
      hard to do. It’s a far easier undertaking when browsers have identical implementations of the web technology you
      use.”
      WebKit blog on the aim of Interop
      (emphasis added)

It is critical to almost every business involved in the web that web technologies work the same on mobile as desktop. For many businesses mobile is the primary target. Many developers may mistakenly believe that mobile browsers are currently covered by the tests.

In order to fix this, we propose a focus area for Interop 2025 that would include:

1. Building and Running Mobile Test Harnesses for WPT
   Ensuring that the WPT tests are run on the mobile versions of each browser across both Android and iOS. The tested OS version should also be reported.

2. Include Mobile Test Results in Interop
   Ensure that mobile testing of browsers is incorporated into both the WPT test results for each browser and into the Interop Dashboard. The Interop main scores should show the lowest test result between Desktop and Mobile.

3. Upgrading WebDriver
   Implement https://w3c.github.io/webdriver-bidi/ universally to improve both the range and the quality of the existing tests. You can thumbs up WebDriver Bidi in this separate interop issue.

Interop Test Scores

With the understanding that mobile test scores are more important than desktop, the Interop dashboard should then be upgraded to show those results.

Our proposal is that the default view of Interop should be set to Experimental and All, where All is the lowest test result across any platform that the browser vendor ships their browser. Where the testing harness for a browser does not support testing that feature on that operating system, then it should be recorded as a score of 0 for those tests. This will ensure that both desktop and mobile are given focus as part of the interop efforts.

Specification

Interop Team

Additional Signals

1. % of browser traffic on mobile devices
2. Number of mobile devices vs desktop devices

[jgraham]

This is effectively a carryover proposal for the existing mobile testing investigation.

[NiklasMerz]

WebViews are also a big part of mobile browsing and could also be considered.

At the WebView Community Group we are currently working on documentating compatibility of WebViews, comparing them to their upstream browsers and adding them to Baseline.

[RupertBenWiser]

RE WebViews: Android WebView support is currently marked as experimental in the WPT project so there could be some great impact here.

[gsnedders]

WebViews are also a big part of mobile browsing and could also be considered.

At the WebView Community Group we are currently working on documentating compatibility of WebViews, comparing them to their upstream browsers and adding them to Baseline.

The problem with web views is on every platform they rely on the caller to implement various bits of logic which end up with web-exposed consequences — in pretty much unavoidable ways. For example, for the various viewport units to work correctly, the web view needs to know what the minimum and maximum sizes it might be resized to are — or svh/svw and lvh/lvw will be wrong — but it can't internally know what sizes the caller might resize it to, so it is reliant on the caller giving it the correct information.

They're also relatively hard to test: for many of the same reasons, you can't really have a generic WebDriver implementation targeting them — because something has to implement logic like "open a new window" (and certainly many things that use WebViews don't support that!).

That's not to say we can't ever test web views — but it will be testing what happens with a specific use of the class.