mirror of
https://github.com/supabase/supabase.git
synced 2026-10-05 17:35:10 +03:00
Miranda LimonczenkoandClaude Opus 5
777c02c205
Closes DOCS-1233 ## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Test coverage. The docs accessibility check now covers the full WCAG 2.1 A/AA rule set instead of two rules. **Note:** This PR tests _only_ the main article of changed pages (meaning, the content itself). A follow-up Linear issue is to address scanning the pieces outside of that: header, navigation, and interactive elements. ## What is the current behavior? The `@a11y` test in `e2e/docs` runs two axe rules against each in-scope page, `heading-order` and `page-has-heading-one`. Both already pass everywhere, so the check only guards a result we have. Nothing else in WCAG A/AA is checked. ## What is the new behavior? The same test runs the full WCAG 2.1 A/AA rule set. - **Existing debt does not block PRs.** Only the two heading rules fail. Everything else reports. - **The check stays fast.** It scans the article only and skips nine rules that cannot fire there. Scan time drops from 2405ms to 981ms. - **Findings belong to us.** Legacy mode excludes cross-origin frames. YouTube embeds were counting against us, 11 of 15 violations on one page. - **A pass carries meaning.** A 404 reports as a load failure, not an a11y bug. A page scanned before it hydrates warns instead of quietly reporting clean. ## How the findings appear The test is named `has no blocking accessibility violations`, so a failure listed by CI is always something to fix. It is not named for the full rule set, because a green check would then claim more than the check verifies. | | Rules | Where you see it | | --- | --- | --- | | Blocking | `heading-order`, `page-has-heading-one` | Test failure, so the runner reports it on the PR | | Reported | Everything else in WCAG A/AA | `::warning` annotation on the run | An annotation looks like this, on a run that still passes: ``` ::warning title=Accessibility::/docs/guides/database/functions has 1 non-blocking accessibility finding(s): frame-title (4) ``` The full axe result for each page is attached to the report as `axe-results.json`. ## Matching the Studio ratchet This follows the ESLint ratchet in `apps/studio`. That pattern warns on pre-existing debt rather than blocking on it, surfaces findings as annotations rather than PR comments, and promotes a rule to an error once its violations reach zero. The mechanism here is `ENFORCED_RULES` in `utils/axe-helpers.ts`. The two heading rules are on it because the heading-hierarchy work drove them to zero site-wide. The intent is to migrate rules into that list one at a time. Pick a rule, fix its violations, then move it into `ENFORCED_RULES` so it cannot come back. An exhaustive scan of the site groups the current backlog by root cause to sequence that work, and two fixes cover 99.1% of it. Studio keeps per-file baseline counts, which this does not. A whole-rule list is coarser, and it works here because docs violations reach zero across the site rather than per file. ## Manual testing Install the browser once, then run each step from the repo root. Every command scans production, so you do not need a local docs server. ```bash pnpm -C e2e/docs exec playwright install chromium ``` 1. Confirm a reported finding does not fail the check. ```bash DOCS_E2E_PAGE_PATHS=/docs/guides/database/functions PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y ``` Expect `1 passed`, and the `::warning` annotation above in the output. 2. Confirm the scan finds that violation. Same page, now failing on every rule. ```bash A11Y_ENFORCE_ALL=1 DOCS_E2E_PAGE_PATHS=/docs/guides/database/functions PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y ``` Expect `1 failed`, reporting `frame-title (serious, 4 node(s))`. Steps 1 and 2 together are the point of this PR. 3. Confirm the skipped rules stay skipped. ```bash A11Y_ENFORCE_ALL=1 DOCS_E2E_PAGE_PATHS=/docs/guides/getting-started/quickstarts/nextjs PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y ``` Expect `button-name (critical, 2 node(s))` and `label (critical, 2 node(s))`, and no `color-contrast`. 4. Confirm a page that does not load reports a load failure. ```bash DOCS_E2E_PAGE_PATHS=/docs/guides/does-not-exist-xyz PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y ``` Expect `Expected a successful response for /docs/guides/does-not-exist-xyz, got 404`, and no axe assertion. 5. Confirm the link checker still passes alongside the a11y test. ```bash DOCS_E2E_PAGE_PATHS=/docs/guides/auth/passwords PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs ``` Expect `3 passed`. ## Known gaps - `/docs/reference/*` is not scanned. Those routes render client-side into tens of thousands of elements, where axe exceeds its timeout and results depend on whether the scan caught the page mid-render. - Shared chrome is outside the article scope, so nav, sidebar, footer, menus, and drawers are not covered. - axe catches roughly 30-40% of WCAG issues. Keyboard navigation, focus management, and screen reader behavior still need manual testing. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>