How it works?

Visual regression compares the current appearance of a page or element with a stored reference screenshot.

In BugBug, this happens as part of a visual regression step during test execution.

What Happens During a Run

When the test reaches a visual regression step, BugBug:

1. captures a screenshot of the selected element or the current page

2. identifies the current environment

3. looks for a matching reference screenshot

4. compares the new screenshot with the reference

5. marks the step as passed or failed

If the step fails, you can review the difference and decide what to do next.

How BugBug Defines the Environment

Reference screenshots are stored per environment.

An environment is defined by the combination of:

• browser

• operating system

• screen size

• profile

• run mode

This matters because the same page can render differently across environments. For example, a test running in Chrome on Linux in cloud mode may need a different reference than the same test running locally on another operating system.

What Happens on the First Run

If BugBug cannot find a reference screenshot for the current environment, it does not compare anything yet.

Instead, it:

1. saves the current screenshot as the initial reference

2. links that reference to the current environment

3. treats the step as passed

This is how the baseline is created automatically during the first run.

The same behavior also applies when a test later runs in a different environment that does not yet have its own matching reference.

How the Comparison Works

When a matching reference exists, BugBug compares the new screenshot with the stored reference screenshot.

The comparison checks:

• how many pixels are different

• how sensitive the comparison should be to color changes

• whether anti-aliased pixels should be included in the diff

The result is an observed visual difference, expressed as a percentage.

What Decides Whether the Step Passes

Each visual regression step uses a max difference value.

BugBug compares:

• the observed difference from the current run

• the allowed max difference for the step

The step passes when the observed difference stays within the allowed threshold. If the observed difference is higher than the allowed max difference, the step fails.

In short:

• small allowed changes can still pass

• larger differences fail the step

What Artifacts BugBug Creates

For visual regression steps, BugBug can store:

• the reference screenshot

• the screenshot captured during the run

• a diff screenshot that highlights the changed areas

These screenshots are used in the run details and in the review flow so you can see what changed instead of guessing.

What You Can Do After a Failure

When a step fails, open Review & fix and compare:

• the expected reference screenshot

• the observed screenshot from the run

• the visual diff

From there, you can choose one of the following actions:

• Do nothing if the change is not acceptable

• Increase max difference if the change is acceptable and the step needs more tolerance

• Set observed screenshot as reference if the new UI should become the baseline

Why Environment-Specific References Matter

Environment-specific references make visual regression more reliable.

Without them, BugBug would compare screenshots that were captured under different rendering conditions. That would produce false failures for reasons that have nothing to do with an actual regression.

By storing references per environment, BugBug compares like with like:

• desktop with desktop

• one browser with the same browser

• one run context with the same run context

That keeps failures more meaningful and review decisions easier.

Summary

Visual regression in BugBug follows a simple flow:

1. capture the screenshot

2. find the matching environment reference

3. create the reference automatically if none exists

4. compare the screenshots when a reference is available

5. pass or fail the step based on the allowed difference

6. let you review and update the result when needed

That is the whole mechanism. The useful part is not that it compares screenshots. The useful part is that it compares the right screenshot against the right baseline.