Automated Accessibility Testing at Slack

Slack’s accessibility strategy has traditionally relied on its internal Accessibility Standards, developer guidance from a dedicated accessibility team, and collaboration with external manual testers specializing in accessibility. In 2022, the company began supplementing this approach with automated accessibility tests for desktop, aiming to catch a subset of violations earlier in the development process.

Automated testing is viewed as one layer within a broader strategy that includes involving people with disabilities early in design, conducting prototype reviews, and performing manual testing across supported assistive technologies. Automated tools cannot catch nuanced issues requiring human judgment, such as screen reader usability, and can occasionally flag items that conflict with specific product design decisions. Despite these limitations, integrating an accessibility tool into existing test frameworks was seen as valuable—ideally in a way that let test owners add checks easily, or even without needing to think about adding them at all.

Framework Exploration and Limitations

The team selected Axe for its configurability, extensive capabilities, and compatibility with Slack’s existing end-to-end (E2E) test frameworks. Axe checks against a wide variety of accessibility guidelines, mostly corresponding to WCAG success criteria, with a design that minimizes false positives.

Initial exploration focused on embedding Axe checks directly into the React Testing Library (RTL) framework by wrapping RTL’s render method with a custom function that included the Axe check. This approach would have removed friction from the developer workflow. However, Slack’s customized Jest setup created complications. Running accessibility checks through a separate Jest configuration worked but would have required developers to write dedicated accessibility tests—something the team wanted to avoid. Reworking the custom Jest setup was deemed too complex and resource-intensive, so attention shifted to Playwright.

Why Playwright

Playwright, already used as Slack’s E2E test framework, supports accessibility testing with Axe through the @axe-core/playwright package. Axe Core provides filtering and customization capabilities out of the box, including exclusion methods and accessibility tags such as wcag2a and wcag2aa to specify the type of analysis.

The initial goal was to bake accessibility checks directly into Playwright’s interaction methods, such as clicks and navigation, so Axe would run automatically without explicit calls from test authors. The main obstacle stemmed from Playwright’s Locator object, which simplifies element interaction by managing auto-waiting, loading, and ensuring elements are fully interactable before actions. This automatic behavior is essential for stable tests but complicated embedding Axe into the framework. Accessibility checks should run when the entire page or key components are fully rendered, but Locator only ensures the readiness of individual elements. Modifying Locator risked unreliable audits with undetected issues if checks ran at the wrong time.

Alternative approaches using deprecated methods like waitForElement to control check timing were also problematic. These older methods are less optimized, causing performance degradation, potential error duplication, and conflicts with Playwright’s abstraction model. While embedding Axe into Playwright’s core interaction methods seemed ideal, the framework’s internal complexity required exploring other solutions.

Customizations and Workarounds

Given the roadblocks with embedding accessibility checks directly into frameworks, the team made concessions while prioritizing a simplified developer workflow. Playwright remained the focus because it offered more flexibility in selectively hiding or applying accessibility checks, allowing better management of when and where checks run. Axe Core’s customization features, such as rule filtering and accessibility tags, were also valuable.

Using the @axe-core/playwright package, the accessibility check flow is:

  1. Playwright test lands on a page/view
  2. Axe analyzes the page
  3. Pre-defined exclusions are filtered out
  4. Violations and artifacts are saved to a file

The main function, runAxeAndSaveViolations, was set up with customized scope using the AxeBuilder class. Checks were configured for compliance with WCAG 2.1, Levels A and AA:

constructor(page: Page) { 
     this.page = page; 
     this.defaultTags = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']; 
     this.bodyText = ''; 
     this.baseFileName = `${test.info().title}-violations`.replace(/\//g, '-'); 
     A11y.filenameCounter = 0; 
}

A list of selectors was created to exclude from violation reports, falling into two categories: known accessibility issues already ticketed, and Axe rules that don’t apply to how Slack is designed for accessibility:

// Exclude selectors for known bugs and elements that we do not consider accessibility issues
constants.ACCESSIBILITY.AXE_EXCLUDED_SELECTORS.forEach((excludedSelector) => {
     axe.exclude(excludedSelector);
});

Methods were created to check uniqueness of violations and filter out duplication. Only violations deemed Critical were reported; other severity levels like Serious, Moderate, and Mild may be added in the future:

/**
* Filter violations based on criticality, then ensure we
* are removing any duplicate violations within a single test file
* Please note: this only removes duplicates on a single test, not the entire run
*/
private filterAndRemoveDuplicateViolations(violations: Violation[]) {
   return violations
	.filter((violation) => ['critical'].includes(violation.impact))
	.map(this.mapViolation)
	.filter(this.isUniqueViolation.bind(this));
}

The Playwright fixture model was leveraged. A custom fixture called slack provides access to API calls, UI views, workflows, and utilities. Moving the accessibility helper into this pre-existing fixture allows direct calls in test specs, minimizing overhead for test authors:

// Run accessibility checks and get the violations
await slack.utils.a11y.runAxeAndSaveViolations();

Playwright’s test.step was customized with the label “Running accessibility checks in runAxeAndSaveViolations” to make it easier to detect where violations occur:

Test Steps
- Before Hooks
- apiResponse.json— ../support/api/api.ts:137
- browserContext.waitForEvent— ../support/workflows/login.workflow.ts:273
- Running accessibility checks in runAxeAndSaveViolations— ../support/utils/accessibility.ts:54

Placement in End-to-End Tests

To kick off the project, a test suite was set up mirroring the suite for critical functionality, renamed for clarity as an accessibility suite, and configured as non-blocking. Developers could see test results, but failures would not prevent merging code to production. This initial suite encompassed 91 tests.

Placement of accessibility checks within these critical flow tests was strategic. In general, a check was added for each new view, page, or flow covered in the test. This usually meant placing a check directly after a button click or link that leads to navigation. In other scenarios, checks were placed after signing in as a second user or after a redirect.

Care was taken to avoid analyzing the same view twice within one test or across multiple tests with the same UI flow, as duplication would result in unnecessary error messages, saved artifacts, and slower tests. Axe calls were placed only after pages or views had fully loaded and all content rendered. This approach required deep familiarity with the application and the context of each test case.

Violations Reporting

Reporting evolved through iteration. Initially, results were saved as a simple text file in an artifacts folder. Developers provided early feedback requesting screenshots of pages where violations occurred. Playwright’s screenshot functionality was integrated, and screenshots were saved alongside the text report in the same artifact folder.

To make reports more coherent, the Playwright HTML Reporter was leveraged. This tool aggregates test results and allows attaching artifacts like screenshots and violation reports to the HTML output. Configuring the HTML reporter displayed all accessibility artifacts—screenshots and detailed violation reports—in a single test report.

The violation error message was customized to pull out key pieces of information and parse and condense the error message for improved readability in reports and on the console:

Error - [A11Y]: CRITICAL 
Description: Ensures an element's role supports its ARIA attributes 
Help: Elements must only use supported ARIA attributes Target selector: #add-channel-tab 
Fix all of the following:
ARIA attribute is not allowed: aria-selected="false" 
HTML: <button class="c-button-unstyled addTab__brBMy c-tabs__tab js-tab" data-qa="unstyled-button"

Environment Setup and Running Tests

Once Axe checks were integrated and the test suite set up, the team determined how developers should run them. An environment flag, A11Y_ENABLE, controls activation of accessibility checks within the framework, with a default value of false to prevent unnecessary runs.

This setup offers developers several options:

  • On-Demand Testing: Developers can manually enable the flag to run accessibility checks locally on their branch.
  • Scheduled Runs: Periodic runs during off-peak hours are possible. A daily regression run configured in Buildkite pipes accessibility results into a Slack alert channel.
  • CI Integration: The flag can optionally be enabled in continuous integration pipelines for thorough testing before merging significant changes.

Triage and Ownership

Maintaining tests raises questions of ownership. At Slack, developers own test creation and maintenance for their tests. To help developers understand framework changes and new accessibility automation, documentation was created, and the internal Slack accessibility team was partnered with to develop a comprehensive triage process fitting their existing workflow.

The internal accessibility team already had a process for triaging and labeling issues using internal Slack Accessibility Standards. A new label for “automated accessibility” was created to track issues discovered via automation. A Jira workflow was set up in the alerts channel to spin up tickets with a pre-populated template. Tickets are automatically labeled automated accessibility and placed in a Jira Epic for triaging:

A11Y Automation Bug Ticket Creator - 
Automatically create JIRA bug tickets for A11Y automation violations

Hi there, Would you like to create a new JIRA defect? 
Button clicked. 
A new JIRA bug ticket, A11YAUTO-37, was created. 

What to do next: 
1. Please fill out all of the necessary information listed here: 
https://jira.tinyspeck.com/browse/A11YAUTO-37. 

2. Please add this locator to the list of known issues 
and include the new JIRA bug ticket in the comment.

Conducting Audits

Regular audits of accessibility Playwright calls are performed to check for duplication of Axe calls and ensure proper coverage across tests and suites. A script and environment flag were developed specifically for auditing. Audits can run through sandbox test runs (ideal for suite-wide audits) or locally (for specific tests). Running the script takes a screenshot of every page that performs an Axe call. Screenshots are saved to a folder for easy comparison to spot duplicates.

This process is more manual than preferred. The team is exploring ways to eliminate this step, potentially leveraging AI assistance to perform audits or automatically add accessibility calls to new pages and views, thereby removing the need for audits entirely.

What’s Next

Plans include continuing to partner with the internal accessibility team to design a small blocking test suite dedicated to core feature flows, with a focus on keyboard navigation. The team also wants to explore AI-driven post-processing of accessibility test results and having AI assistants audit suites to determine check placement, further reducing manual effort.

The project required balancing practical limitations of automated tools with the goal of reducing developer burden. While accessibility checks couldn’t be completely integrated into frontend frameworks, significant progress was made: simplified processes for adding checks, easy-to-interpret test results, clear documentation, and streamlined triage through Slack workflows. Automated Axe checks have reduced reliance on manual testing and now complement other essential forms of testing.

Currently, developers need to manually add checks, but groundwork has been laid to make the process straightforward, with potential for AI-driven creation of accessibility tests. Roadblocks like framework complexity or setup difficulties shouldn’t discourage automation as part of a broader accessibility strategy. Even when checks can’t be hidden entirely behind the scenes, focusing on developer experience makes the work impactful. This project strengthened both the accessibility testing approach and the culture of accessibility central to Slack’s product.