Migrating From Percy To Local Visual Regression Testing

by Alex Johnson 56 views

This article outlines the process of replacing Percy, a visual regression testing service, with a local, cost-effective solution using odiff, an image comparison tool. This migration aims to reduce costs, improve feedback speed, and provide greater control over the visual testing process. This approach will be initially tested in a specific repository before being implemented in larger projects like front-end/canvas and other repositories.

Understanding Visual Regression Testing and the Need for Change

Visual regression testing is crucial for ensuring that changes to a user interface (UI) don't introduce unintended visual defects. These defects, often subtle, can negatively impact the user experience. Traditionally, services like Percy have been used to capture snapshots of UI components and compare them against baseline images. However, these services often come with per-snapshot pricing, which can become expensive as projects grow. By transitioning to a local, odiff-based solution, we aim to eliminate these costs while maintaining the integrity of our visual testing process.

The primary motivation for this migration is to achieve cost savings by eliminating the per-snapshot pricing associated with Percy. Additionally, local snapshots offer the advantage of being stored within the repository and version controlled, providing a more transparent and auditable history of UI changes. Furthermore, a local solution offers faster feedback cycles, as there's no need to wait for a third-party service to process snapshots. This speedier feedback loop enables developers to identify and address visual regressions more quickly, ultimately leading to a more efficient development workflow.

Current State: Percy Integration

Currently, the project utilizes Percy for visual regression testing, with 56 snapshot calls across 14 test files. A Percy helper function, located at tests/helpers/percy-snapshot.js, is used to manage snapshot naming. Storybook, a popular UI component development tool, is configured with @storybook/ember. The test environment is set up to launch Chrome with remote debugging enabled via testem.js. This existing infrastructure provides a solid foundation for transitioning to a local solution.

The existing Percy integration includes 56 snapshot calls distributed across 14 test files, highlighting the extensive use of visual regression testing within the project. The tests/helpers/percy-snapshot.js file houses a helper function that streamlines the snapshot naming process, ensuring consistency and clarity. The integration of Storybook with @storybook/ember allows for visual testing of individual components in isolation. Crucially, the testem.js configuration already launches Chrome with the --remote-debugging-port flag, which is a prerequisite for using Puppeteer for local screenshot capture. This existing setup minimizes the initial configuration effort required for the migration.

Proposed Solution: Local odiff-Based Testing

The proposed solution involves a three-pronged approach:

  1. Ember Component Tests → Custom Screenshot Addon: Develop an Ember addon that leverages Puppeteer, a Node library that controls Chrome, to capture screenshots during component tests. These screenshots will be saved locally. This addon will provide a simple API for capturing screenshots within tests.
  2. Storybook → Storycap + odiff: Replace @percy/storybook with Storycap, a tool for capturing screenshots of Storybook stories. These screenshots will then be compared using odiff.
  3. CI Integration → GitHub Action: Create a custom GitHub Action to automate the comparison of snapshots between branches in a continuous integration (CI) environment. This action will utilize odiff for image comparison and generate a visual diff report.

1. Ember Component Tests: Embracing a Custom Screenshot Addon

The first step involves creating a dedicated Ember addon to handle screenshot capture within component tests. This addon will leverage Puppeteer, a powerful Node library that enables programmatic control over Chrome. By using Puppeteer via testem middleware, the addon will be able to capture real Chrome screenshots directly within the test environment. This approach ensures a high degree of fidelity and consistency in the captured images.

The architecture of the addon will consist of the following key components: Testem middleware will host a Puppeteer server on the server-side. This server will connect to Chrome's remote debugging port, enabling Puppeteer to control the browser instance. Tests will interact with the addon via a simple await screenshot(assert) API, making it easy to capture screenshots within test cases. Screenshots will be saved locally in the tests/snapshots/ directory, ensuring they are easily accessible and version controlled.

The required changes to the existing configuration are minimal. The testem.js file needs to be updated to use a fixed remote debugging port (9222) instead of a dynamically assigned port. This ensures that Puppeteer can reliably connect to the Chrome instance. The following diff illustrates the necessary change:

// testem.js
- '--remote-debugging-port=0',
+ '--remote-debugging-port=9222',

To facilitate a smooth transition, a codemod-able migration path will be provided. This will allow developers to automatically update existing test files to use the new screenshot API. The following code snippet demonstrates the migration from the Percy snapshot API to the new screenshot API:

// Before
import percySnapshot from '@percy/ember';
await percySnapshot(assert);
await percySnapshot(assert, 'with label');

// After
import { screenshot } from 'ember-screenshot';
await screenshot(assert);
await screenshot(assert, 'with label');

2. Storybook: Leveraging Storycap and odiff for Visual Consistency

For Storybook, the plan is to replace @percy/storybook with a combination of Storycap and odiff. Storycap is a tool designed specifically for capturing screenshots of Storybook stories, while odiff provides a robust and efficient image comparison engine. This combination offers a flexible and cost-effective solution for visual regression testing within Storybook.

The process of capturing and comparing screenshots involves the following steps: First, Storycap is used to capture screenshots of all stories in the Storybook instance. The captured screenshots are saved locally in a designated directory, such as ./storybook-snapshots. Then, in the CI environment, odiff is used to compare the newly captured screenshots against a set of baseline images. This comparison identifies any visual differences between the current state and the baseline, flagging potential regressions. The following commands illustrate the usage of Storycap and odiff:

# Capture screenshots
npx storycap http://localhost:6006 -o ./storybook-snapshots

# Compare with odiff in CI
npx odiff ./storybook-snapshots/baseline ./storybook-snapshots/current ./storybook-snapshots/diff

3. CI Integration: Automating Visual Regression Testing with GitHub Actions

To fully automate the visual regression testing process, a custom GitHub Action will be created. This action will be responsible for comparing snapshots between the pull request (PR) branch and the baseline branch. It will use odiff for pixel-level comparison, allowing for configurable thresholds to control the sensitivity of the comparison. The action will also generate a visual diff report within the GitHub Actions job summary, providing a clear and concise overview of any visual regressions detected.

In addition to the diff report, the action will upload diff images as artifacts, making it easy to inspect the specific visual differences between the snapshots. This allows developers to quickly identify and address any issues. The custom GitHub Action will be designed for reusability and will be published for use in other projects. This promotes consistency and efficiency across different repositories.

Tasks: A Roadmap for Implementation

The migration from Percy to a local odiff-based solution involves a series of well-defined tasks:

  • [ ] Update testem.js to use a fixed remote debugging port (9222). This ensures consistent connectivity for Puppeteer.
  • [ ] Create/integrate the ember-screenshot addon with Puppeteer middleware. This forms the foundation for local screenshot capture in Ember tests.
  • [ ] Add a screenshot helper with the same naming pattern as the Percy wrapper. This maintains consistency and simplifies the migration process.
  • [ ] Migrate Percy calls to the new screenshot helper (56 calls). This involves updating existing tests to use the new API.
  • [ ] Set up Storycap for Storybook screenshot capture. This enables visual testing of Storybook components.
  • [ ] Integrate the odiff GitHub Action for CI comparison. This automates the visual regression testing process.
  • [ ] Remove @percy/ember and @percy/cli dependencies. This eliminates unnecessary dependencies.
  • [ ] Update the CI workflow to use the new visual testing approach. This ensures that the CI pipeline incorporates the new solution.
  • [ ] Document the migration process for other repos. This facilitates wider adoption of the local testing strategy.

Benefits: Cost Savings, Faster Feedback, and More

The transition to a local odiff-based visual regression testing solution offers numerous benefits:

  • Cost savings: By eliminating per-snapshot pricing, the project can achieve significant cost reductions.
  • Local snapshots: Storing snapshots in the repository provides version control and transparency.
  • Faster feedback: Local processing of snapshots results in quicker feedback cycles, enabling faster identification and resolution of visual regressions.
  • Test runner agnostic: The odiff GitHub Action can be used across different testing frameworks, such as Cypress, Storybook, and Ember, promoting consistency and reusability.

References: Resources for Further Exploration

By migrating from Percy to a local odiff-based solution, the project can achieve significant cost savings, improve feedback speed, and gain greater control over the visual regression testing process. This migration represents a strategic investment in the long-term maintainability and quality of the user interface.