test(docs): scan changed pages for WCAG 2.1 A/AA in warn mode (#48727)

Closes DOCS-1233

## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Test coverage. The docs accessibility check now covers the full WCAG 2.1
A/AA rule set instead of two rules.

**Note:** This PR tests _only_ the main article of changed pages
(meaning, the content itself). A follow-up Linear issue is to address
scanning the pieces outside of that: header, navigation, and interactive
elements.

## What is the current behavior?

The `@a11y` test in `e2e/docs` runs two axe rules against each in-scope
page, `heading-order` and `page-has-heading-one`. Both already pass
everywhere, so the check only guards a result we have. Nothing else in
WCAG A/AA is checked.

## What is the new behavior?

The same test runs the full WCAG 2.1 A/AA rule set.

- **Existing debt does not block PRs.** Only the two heading rules fail.
Everything else reports.
- **The check stays fast.** It scans the article only and skips nine
rules that cannot fire there. Scan time drops from 2405ms to 981ms.
- **Findings belong to us.** Legacy mode excludes cross-origin frames.
YouTube embeds were counting against us, 11 of 15 violations on one
page.
- **A pass carries meaning.** A 404 reports as a load failure, not an
a11y bug. A page scanned before it hydrates warns instead of quietly
reporting clean.

## How the findings appear

The test is named `has no blocking accessibility violations`, so a
failure listed by CI is always something to fix. It is not named for the
full rule set, because a green check would then claim more than the
check verifies.

| | Rules | Where you see it |
| --- | --- | --- |
| Blocking | `heading-order`, `page-has-heading-one` | Test failure, so
the runner reports it on the PR |
| Reported | Everything else in WCAG A/AA | `::warning` annotation on
the run |

An annotation looks like this, on a run that still passes:

```
::warning title=Accessibility::/docs/guides/database/functions has 1 non-blocking accessibility finding(s): frame-title (4)
```

The full axe result for each page is attached to the report as
`axe-results.json`.

## Matching the Studio ratchet

This follows the ESLint ratchet in `apps/studio`. That pattern warns on
pre-existing debt rather than blocking on it, surfaces findings as
annotations rather than PR comments, and promotes a rule to an error
once its violations reach zero.

The mechanism here is `ENFORCED_RULES` in `utils/axe-helpers.ts`. The
two heading rules are on it because the heading-hierarchy work drove
them to zero site-wide.

The intent is to migrate rules into that list one at a time. Pick a
rule, fix its violations, then move it into `ENFORCED_RULES` so it
cannot come back. An exhaustive scan of the site groups the current
backlog by root cause to sequence that work, and two fixes cover 99.1%
of it.

Studio keeps per-file baseline counts, which this does not. A whole-rule
list is coarser, and it works here because docs violations reach zero
across the site rather than per file.

## Manual testing

Install the browser once, then run each step from the repo root. Every
command scans production, so you do not need a local docs server.

```bash
pnpm -C e2e/docs exec playwright install chromium
```

1. Confirm a reported finding does not fail the check.

   ```bash
DOCS_E2E_PAGE_PATHS=/docs/guides/database/functions
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y
   ```

Expect `1 passed`, and the `::warning` annotation above in the output.

2. Confirm the scan finds that violation. Same page, now failing on
every rule.

   ```bash
A11Y_ENFORCE_ALL=1 DOCS_E2E_PAGE_PATHS=/docs/guides/database/functions
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y
   ```

Expect `1 failed`, reporting `frame-title (serious, 4 node(s))`. Steps 1
and 2 together are the point of this PR.

3. Confirm the skipped rules stay skipped.

   ```bash
A11Y_ENFORCE_ALL=1
DOCS_E2E_PAGE_PATHS=/docs/guides/getting-started/quickstarts/nextjs
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y
   ```

Expect `button-name (critical, 2 node(s))` and `label (critical, 2
node(s))`, and no `color-contrast`.

4. Confirm a page that does not load reports a load failure.

   ```bash
DOCS_E2E_PAGE_PATHS=/docs/guides/does-not-exist-xyz
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y
   ```

Expect `Expected a successful response for
/docs/guides/does-not-exist-xyz, got 404`, and no axe assertion.

5. Confirm the link checker still passes alongside the a11y test.

   ```bash
DOCS_E2E_PAGE_PATHS=/docs/guides/auth/passwords
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs
   ```

   Expect `3 passed`.

## Known gaps

- `/docs/reference/*` is not scanned. Those routes render client-side
into tens of thousands of elements, where axe exceeds its timeout and
results depend on whether the scan caught the page mid-render.
- Shared chrome is outside the article scope, so nav, sidebar, footer,
menus, and drawers are not covered.
- axe catches roughly 30-40% of WCAG issues. Keyboard navigation, focus
management, and screen reader behavior still need manual testing.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

apps/docs/DEVELOPERS.md

@@ -35,6 +35,26 @@ This creates Markdown files for all routes under the `public/markdown/guides` di
 
 For production this setup runs as a `prebuild` task to allow Vercel to bundle these files with middleware and functions.
 
+## Accessibility checks
+
+Docs pages are scanned for WCAG 2.1 A/AA issues with axe-core, as part of the
+Playwright suite in `e2e/docs`. Pull requests scan the pages your change affects,
+limited to the main article.
+
+To scan the pages your current branch changes:
+
+```bash
+PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y
+```
+
+That resolves which pages to scan from your branch, but reads them from
+production, so it won't see your edits and will 404 on a page you just added.
+Point `PLAYWRIGHT_BASE_URL` at your pull request's preview to scan your own
+content.
+
+See [`e2e/docs/README.md`](https://github.com/supabase/supabase/blob/master/e2e/docs/README.md)
+for coverage and skipped rules.
+
 ## Contributing
 
 For repo organization and style guide, see the [contributing guide](https://github.com/supabase/supabase/blob/master/apps/docs/CONTRIBUTING.md).

e2e/docs/README.md

@@ -15,6 +15,7 @@ This page covers:
 - [Override which pages run](#override-which-pages-run) — when the default git
   scope is wrong
 - [What the suite covers](#what-the-suite-covers) — in-scope paths and limits
+- [Accessibility scans](#accessibility-scans) — WCAG coverage and skipped rules
 - [Debug failures](#debug-failures) — reports and traces
 - [How CI uses this suite](#how-ci-uses-this-suite) — pull request behavior
 
@@ -168,6 +169,30 @@ staged and unstaged working-tree changes.
 } | pnpm -C e2e/docs resolve-docs-scope
 ```
 
+## Accessibility scans
+
+The `@a11y`-tagged test scans each in-scope page for WCAG 2.1 A/AA violations using
+`@axe-core/playwright`, limited to the main article.
+
+```bash
+PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y
+```
+
+Which pages get scanned comes from your branch, but the content comes from
+whatever you point `PLAYWRIGHT_BASE_URL` at. Production won't have your edits and
+will 404 on a page you just added, so use your pull request's preview to scan your
+own content.
+
+`EXCLUDED_RULES` in `utils/axe-helpers.ts` lists the rules the scan skips.
+`color-contrast` is most of the scan time and finds nothing inside an article, since
+docs contrast comes from shared tokens and chrome. The rest target `<html>`, `<head>`,
+and `<body>`, which an article-scoped scan can't reach.
+
+Cross-origin frames are skipped, so a third-party embed isn't reported as ours.
+
+Not covered: `/docs/reference/*`, shared chrome, and most of WCAG. Keyboard
+navigation, focus management, and screen reader behavior need manual testing.
+
 ## Debug failures
 
 1. Open the HTML report after a run:
@@ -186,7 +211,8 @@ owned docs content, partials, or `e2e/docs`.
 1. Diff the pull request against its base branch and resolve in-scope page paths.
 2. Skip Playwright when nothing in scope changed.
 3. When `apps/docs` changed, wait for the Vercel docs preview and set
-   `PLAYWRIGHT_BASE_URL` to that preview. Otherwise use production.
+   `PLAYWRIGHT_BASE_URL` to that preview. When no preview resolves, skip rather than
+   test against production.
 4. Run the suite with `DOCS_E2E_PAGE_PATHS` set to the resolved list.
 
 Draft pull requests stay skipped until you mark them ready for review. Manual

e2e/docs/features/docs-pages.spec.ts

@@ -1,6 +1,17 @@
-import { AxeBuilder } from '@axe-core/playwright'
 import { expect, test } from '@playwright/test'
+import type { TestInfo } from '@playwright/test'
 
+import {
+  attachScanReport,
+  blockingViolations,
+  ENFORCED_RULES,
+  formatViolations,
+  scanArticle,
+  scanLooksEmpty,
+  settleForAxe,
+  shouldEnforceAll,
+  unloadedResult,
+} from '../utils/axe-helpers.js'
 import {
   articleSelectorForPagePath,
   browserLikeUserAgent,
@@ -10,6 +21,11 @@ import {
 
 const pagePaths = parseDocsE2EPagePaths(process.env.DOCS_E2E_PAGE_PATHS)
 
+function annotate(testInfo: TestInfo, description: string) {
+  testInfo.annotations.push({ type: 'warning', description })
+  console.warn(`::warning title=Accessibility::${description}`)
+}
+
 test.describe('Docs owned pages', () => {
   // playwright.config.ts sets fullyParallel: false, and Playwright shards
   // work by file rather than by test in that mode — without this, every test
@@ -63,19 +79,69 @@ test.describe('Docs owned pages', () => {
   }
 
   for (const pagePath of pagePaths) {
-    test(`${pagePath} has a valid heading hierarchy @a11y`, async ({ page }) => {
-      const articleSelector = articleSelectorForPagePath(pagePath)
-      const response = await page.goto(pagePath)
-      expect(response?.ok(), `Expected a successful response for ${pagePath}`).toBeTruthy()
+    test(`${pagePath} has no blocking accessibility violations @a11y`, async ({
+      page,
+    }, testInfo) => {
+      test.setTimeout(120_000)
 
-      const axeResults = await new AxeBuilder({ page })
-        .include(articleSelector)
-        .withRules(['heading-order', 'page-has-heading-one'])
-        .analyze()
+      const include = articleSelectorForPagePath(pagePath)
+
+      let response
+      try {
+        response = await page.goto(pagePath)
+      } catch (error) {
+        await attachScanReport(testInfo, unloadedResult(pagePath, pagePath, null, include))
+        throw error
+      }
+
+      const status = response?.status() ?? null
+
+      if (!response?.ok()) {
+        await attachScanReport(testInfo, unloadedResult(pagePath, page.url(), status, include))
+        expect(
+          response?.ok(),
+          `Expected a successful response for ${pagePath}, got ${status}`
+        ).toBeTruthy()
+        return
+      }
+
+      await settleForAxe(page)
+
+      await expect(
+        page.locator(include),
+        `No article matching "${include}" on ${pagePath}. This suite covers guides and ` +
+          'troubleshooting entries; other routes have no article element to scan.'
+      ).toBeVisible()
+
+      const result = await scanArticle(page, pagePath, include)
+      result.status = status
+
+      await attachScanReport(testInfo, result)
+
+      if (scanLooksEmpty(result)) {
+        annotate(
+          testInfo,
+          `${pagePath} scanned only ${result.elementCount} element(s) in ${include}, so a clean ` +
+            'result here proves nothing. Most likely the page had not finished rendering.'
+        )
+      }
+
+      const blocking = blockingViolations(result)
+
+      const reported = result.violations.filter((violation) => !blocking.includes(violation))
+      if (reported.length) {
+        annotate(
+          testInfo,
+          `${pagePath} has ${reported.length} non-blocking accessibility finding(s): ` +
+            reported.map((v) => `${v.id} (${v.nodes.length})`).join(', ')
+        )
+      }
+
+      const enforced = shouldEnforceAll() ? 'all WCAG 2.1 A/AA rules' : ENFORCED_RULES.join(', ')
 
       expect(
-        axeResults.violations,
-        `Heading hierarchy issues in ${articleSelector}:\n${JSON.stringify(axeResults.violations, null, 2)}`
+        blocking,
+        `${pagePath} has blocking a11y violations (${enforced}):\n${formatViolations(blocking)}`
       ).toEqual([])
     })
   }

e2e/docs/package.json

@@ -16,6 +16,7 @@
   },
   "devDependencies": {
     "@axe-core/playwright": "^4.12.1",
-    "@types/node": "catalog:"
+    "@types/node": "catalog:",
+    "axe-core": "^4.12.1"
   }
 }

e2e/docs/scripts/run-e2e-docs.ts

@@ -122,10 +122,7 @@ async function resolvePagePaths(): Promise<string[] | null> {
 async function main() {
   const rawArgs = process.argv.slice(2)
   const runAll = rawArgs.includes('--all')
-  const withoutAll = rawArgs.filter((arg) => arg !== '--all')
-  // pnpm's `--` separator (from `pnpm run ... -- --list`) can land at index 0
-  // or, once `--all` is stripped, wherever `--all` used to precede it.
-  const playwrightArgs = withoutAll[0] === '--' ? withoutAll.slice(1) : withoutAll
+  const playwrightArgs = rawArgs.filter((arg) => arg !== '--all' && arg !== '--')
 
   const pages = runAll ? await resolveAllPagePaths() : await resolvePagePaths()
   if (pages === null) {

e2e/docs/utils/axe-helpers.ts

@@ -0,0 +1,148 @@
+import { AxeBuilder } from '@axe-core/playwright'
+import type { Page, TestInfo } from '@playwright/test'
+import type { Result } from 'axe-core'
+
+export const WCAG_TAGS = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']
+
+export const ENFORCED_RULES = ['heading-order', 'page-has-heading-one']
+
+export const EXCLUDED_RULES = [
+  'color-contrast',
+  'html-has-lang',
+  'html-lang-valid',
+  'html-xml-lang-mismatch',
+  'document-title',
+  'aria-hidden-body',
+  'meta-viewport',
+  'meta-refresh',
+  'css-orientation-lock',
+]
+
+export interface A11yScanResult {
+  surface: string
+  url: string
+  include: string
+  excludedRules: string[]
+  loaded: boolean
+  status: number | null
+  elementCount: number
+  violations: Result[]
+}
+
+export function shouldEnforceAll(): boolean {
+  return !!process.env.A11Y_ENFORCE_ALL
+}
+
+export async function settleForAxe(page: Page): Promise<void> {
+  await page.waitForLoadState('domcontentloaded')
+  await page.waitForLoadState('networkidle', { timeout: 15_000 }).catch(() => {})
+
+  await page
+    .evaluate(
+      ({ quietMs, capMs }) =>
+        new Promise<void>((resolve) => {
+          let timer: ReturnType<typeof setTimeout>
+          const observer = new MutationObserver(() => {
+            clearTimeout(timer)
+            timer = setTimeout(finish, quietMs)
+          })
+
+          function finish() {
+            clearTimeout(cap)
+            clearTimeout(timer)
+            observer.disconnect()
+            resolve()
+          }
+
+          const cap = setTimeout(finish, capMs)
+          timer = setTimeout(finish, quietMs)
+          observer.observe(document.body, { subtree: true, childList: true, attributes: true })
+        }),
+      { quietMs: 500, capMs: 5_000 }
+    )
+    .catch(() => {})
+}
+
+export async function scanArticle(
+  page: Page,
+  surface: string,
+  include: string
+): Promise<A11yScanResult> {
+  const scan = () => new AxeBuilder({ page }).setLegacyMode(true).include(include)
+
+  const reported = await scan().withTags(WCAG_TAGS).disableRules(EXCLUDED_RULES).analyze()
+
+  const enforced = await scan().withRules(ENFORCED_RULES).analyze()
+
+  const byRule = new Map(
+    [...reported.violations, ...enforced.violations].map((violation) => [violation.id, violation])
+  )
+
+  const elementCount = await page.evaluate(
+    (selector) => document.querySelector(selector)?.querySelectorAll('*').length ?? 0,
+    include
+  )
+
+  return {
+    surface,
+    url: page.url(),
+    include,
+    excludedRules: EXCLUDED_RULES,
+    loaded: true,
+    status: null,
+    elementCount,
+    violations: [...byRule.values()],
+  }
+}
+
+export function unloadedResult(
+  surface: string,
+  url: string,
+  status: number | null,
+  include: string
+): A11yScanResult {
+  return {
+    surface,
+    url,
+    include,
+    excludedRules: EXCLUDED_RULES,
+    loaded: false,
+    status,
+    elementCount: 0,
+    violations: [],
+  }
+}
+
+export const MIN_MEANINGFUL_ELEMENTS = 20
+
+export function scanLooksEmpty(
+  result: A11yScanResult,
+  minElements: number = MIN_MEANINGFUL_ELEMENTS
+): boolean {
+  return result.elementCount < minElements
+}
+
+export async function attachScanReport(testInfo: TestInfo, result: A11yScanResult): Promise<void> {
+  await testInfo.attach('axe-results.json', {
+    body: JSON.stringify(result, null, 2),
+    contentType: 'application/json',
+  })
+}
+
+export function blockingViolations(result: A11yScanResult): Result[] {
+  if (shouldEnforceAll()) return result.violations
+  return result.violations.filter((violation) => ENFORCED_RULES.includes(violation.id))
+}
+
+export function formatViolations(violations: Result[]): string {
+  return violations
+    .map(
+      (violation) =>
+        `${violation.id} (${violation.impact}, ${violation.nodes.length} node(s)): ${violation.help}\n` +
+        violation.nodes
+          .slice(0, 5)
+          .map((node) => `    ${node.target.join(' ')}\n      ${node.html.slice(0, 200)}`)
+          .join('\n')
+    )
+    .join('\n')
+}

package.json

@@ -38,6 +38,7 @@
     "e2e:ui": "pnpm --prefix e2e/studio run e2e:ui",
     "e2e:docs": "pnpm --prefix e2e/docs run e2e:docs",
     "e2e:docs:all": "pnpm --prefix e2e/docs run e2e:docs:all",
+    "e2e:docs:a11y": "pnpm --prefix e2e/docs run e2e:docs:a11y",
     "e2e:docs:ui": "pnpm --prefix e2e/docs run e2e:ui",
     "e2e:docs:local-smoke": "pnpm --prefix e2e/docs run e2e:docs:local-smoke",
     "perf:kong": "ab -t 5 -c 20 -T application/json http://localhost:8000/",

pnpm-lock.yaml

@@ -2027,6 +2027,9 @@ importers:
       '@types/node':
         specifier: 'catalog:'
         version: 22.13.14
+      axe-core:
+        specifier: ^4.12.1
+        version: 4.12.1
 
   e2e/studio:
     dependencies:
@@ -10253,6 +10256,7 @@ packages:
   cron-parser@4.9.0:
     resolution: {integrity: sha512-p0SaNjrHOnQeR8/VnfGbmg9te2kfyYSQ7Sc/j/6DtPL3JQvKxmjO9TSjNFpujqV3vEYYBvNNvXSxzyksBWAx1Q==}
     engines: {node: '>=12.0.0'}
+    deprecated: v4 is no longer maintained, upgrade to v5
 
   croner@10.0.1:
     resolution: {integrity: sha512-ixNtAJndqh173VQ4KodSdJEI6nuioBWI0V1ITNKhZZsO0pEMoDxz539T4FTTbSZ/xIOSuDnzxLVRqBVSvPNE2g==}