Skip to content

Visual regression testing

Jump to bottom
Anton Olkhovskyi edited this page Mar 24, 2021 · 4 revisions

Overview

All components that are covered by regression testing have visual regression testing in place along with functional tests. Visual regression checks that elements on the page have:

  • correct positioning
  • correct colors theme
  • have correct state applied (hover, focused or selected states)

Approved screenshot which are used as expected component/element state are called baseline screenshot.

Storing the baseline screenshots

All baseline screenshots are stored in the e2e/wdio/baselineScreenshots folder of the repository.

When the test executes for the first time, baseline screenshots is generated automatically and saved to the e2e/wdio/baselineScreenshots/platform or core/<component_name>/<browser_name> folder.

For all following executions, new baseline screenshot will not be generated if the baseline screenshot with the same name is already exist in this folder. Instead, a comparison of the new screenshot vs the baseline will be performed.

In case of the tests failure, two screenshot will be generated, for each comparison action:

  • actual screenshot
  • difference screenshot
    This two types of the screenshots will be saved into the .tmp/actual/<browser_name> and .tmp/diff/<browser_name> folders respectively.

All baseline screenshot that are generated for new component needs to be committed to the repository along with the e2e test case code.

Crosbrowser visual regression

For component/element visual regression baseline screenshots is created for each browser + OS combination. For example if one component is under visual regression and tests are executed for Chrome and FF browsers for Win and MacOs, four separate screenshots will be generated and needs to be committed along with the test case code.

Visual regression test cases failure debug.

Visual regression test case indicates the mismatch of the baseline screenshot and actual screenshot (in percentage). If the baseline screenshot and actual screenshot doesn’t match the particular test case will fail with an error of mismatch percentage. (Some test cases needs to be refactored to provide a more meaningful error message.)

In case of the visual regression test case is failing:

  • The test case is needs to be re-executed from the local environment on the Saucelabs side on the env on which tests case failed.
  • The difference screenshot need to be evaluated by PR owner to determine if the difference is expected or not. In the .tmp/diff folder you will be able to find the screenshot with mismatch highlighted in purple color.

In case the PR introduces unexpected changes, it needs to be fixed without merging any unexpected change to the main branch.
In case if change to the component is expected, the baseline screenshot need to be replaced with the new actual screenshot. (You can do this moving the screenshot form the .tmp/actual folder to the e2e/wdio/baselineScreenshots/platform or core/<component_name>/<browser_name> folder). The newly added (newly approved baseline) screenshots need to be committed to this PR.

All baseline screenshots needs to be generated on the Saucelabs side. (the same environment on which actual testing form the CI will be performed). Images generated during the local tests execution can't be used as a baseline screenshots, unless that all environment parameters has complete match (OS version, Browser version, screen resolution)

Clone this wiki locally