mirror of
https://github.com/supabase/supabase.git
synced 2026-10-08 10:55:06 +03:00
a11y/docs-global-elements
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Miranda LimonczenkoandClaude Opus 5
|
d8972f29cb |
test(docs): run global elements in the full-site audit
`pnpm e2e:docs:all` selected the pages project only, so the audit skipped everything outside the article. It now selects both projects and writes one combined report. The per-project commands are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Miranda Limonczenko
|
98db26278b |
test(docs): scan global elements outside the article
The page scan stops at the article wrapper, so nothing checked the top nav, sidebar nav, table-of-contents sidebar, breadcrumbs, or footer. A violation in any of those lands on every docs page. Adds a global-elements Playwright project that scans the document with the article excluded, across six pages, one per layout, at two viewports, plus one pass with the mobile menu open. It runs from a docs_components paths filter in the existing workflow, so a content-only pull request never triggers it. Rules that pass today are enforced, so they catch a newly introduced violation without failing on existing debt. Everything else annotates. Supersedes #48909, which predated e2e/shared. |
||
|
Miranda LimonczenkoandClaude Opus 5
|
6d3a4bcc48 |
feat(www) Add scaffolding for WWW E2E tests and CI check (#48861)
Closes DOCS-1278 ## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Feature. Adds E2E test scaffolding and a CI check for the marketing site. ## What is the current behavior? Closes [FE-4047](https://linear.app/supabase/issue/FE-4047). The marketing site has no E2E coverage. Docs has a suite in `e2e/docs`, but its runner, git helpers and axe reporting are private to that package, so a second site cannot reuse them. ## What is the new behavior? * **A www suite scoped to changed content.** Changed `.mdx` files in `_blog`, `_events`, `_customers` and `_alternatives` map to the URLs they render. Pages with `disable_page_build: true` are skipped because they 404 by design. Capped at 20 pages. Enforces `heading-order` and `page-has-heading-one`, matching docs. * **`e2e/shared` The docs site is also static with similar needs. This folder shares the docs logic with www. * **A CI check that is safe to mark required.** Path scoping lives in a `Detect changed paths` step rather than a `paths:` trigger, so the check reports on every pull request instead of being skipped. `waitForVercelDocsPreview.js` becomes `waitForVercelPreview.js`, shared by both workflows. ## How the check behaves The job always reports a check run, so it is safe to mark required. Path scoping happens in a step rather than a `paths:` trigger, which would leave non-www pull requests waiting on a check that never reports. | Case | Behavior | | --- | --- | | Fork pull request adds new pages | Passes without testing. The Vercel wait is gated on `head.repo.full_name == github.repository`, so forks resolve no preview URL. The job emits a `::warning` and a job summary containing a ready-to-run `gh workflow run www-e2e.yml` command with the resolved page paths, so a maintainer can run it against the preview. | | Vercel preview times out or fails | Passes without testing. The wait step is `continue-on-error: true`, so a 900s timeout or a failed deployment leaves the URL unset and the suite skips. Vercel's own `Vercel – zone-www-dot-com` check already reports the failure. | | Draft pull request | Job does not run at all, gated at the job level on `pull_request.draft == false`. `ready_for_review` is in the trigger's `types`, so marking it ready runs the check. | | Another app changed, www untouched | Job runs and every step skips. The `www` filter matches only the four content directories, `e2e/www`, `e2e/shared`, the lockfile, and this workflow. | | Only the harness changed | Passes without testing. Scope resolves to zero pages, and the Vercel wait is additionally gated on `www_app`, so it does not wait for a preview Vercel skipped. | | No preview resolves, any reason | Skips rather than falling back to production. Production does not serve pages the pull request adds, so testing it would fail a valid change. | ### Not covered Changes to `apps/www` components and routes do not trigger this check — only the four content directories do. A follow-up can check global components such as the navigation and the footer. ## Manual testing 1. Start the site: `pnpm dev:www` 2. Run `pnpm e2e:www` with no www content changed. It should resolve zero pages and skip Playwright, not fail. 3. Touch a post, then run `pnpm e2e:www` again: `echo "" >> apps/www/_blog/2024-01-01-some-post.mdx`. The resolved `/blog/...` path should be listed before Playwright starts. 4. Run against production with no local server: `PLAYWRIGHT_BASE_URL=https://supabase.com WWW_E2E_PAGE_PATHS=/blog/postgres-language-server pnpm e2e:www` 5. Point step 4 at a page with a known heading problem. The failure should name the rule, the CSS selector and the markup. 6. Confirm docs still passes on the shared runner: `pnpm dev:docs`, then `pnpm e2e:docs` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added WWW end-to-end testing for affected content pages, including accessibility checks. * Added standard and full-site test commands, configurable preview testing, and failure reports. * Added shared utilities for page discovery, accessibility scanning, and test execution. * **Documentation** * Documented WWW test setup, coverage, debugging, CI behavior, and running checks against production or preview environments. * **Improvements** * Updated documentation test workflows to better identify affected changes and handle preview environments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Miranda LimonczenkoandClaude Opus 5
|
777c02c205 |
test(docs): scan changed pages for WCAG 2.1 A/AA in warn mode (#48727)
Closes DOCS-1233 ## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Test coverage. The docs accessibility check now covers the full WCAG 2.1 A/AA rule set instead of two rules. **Note:** This PR tests _only_ the main article of changed pages (meaning, the content itself). A follow-up Linear issue is to address scanning the pieces outside of that: header, navigation, and interactive elements. ## What is the current behavior? The `@a11y` test in `e2e/docs` runs two axe rules against each in-scope page, `heading-order` and `page-has-heading-one`. Both already pass everywhere, so the check only guards a result we have. Nothing else in WCAG A/AA is checked. ## What is the new behavior? The same test runs the full WCAG 2.1 A/AA rule set. - **Existing debt does not block PRs.** Only the two heading rules fail. Everything else reports. - **The check stays fast.** It scans the article only and skips nine rules that cannot fire there. Scan time drops from 2405ms to 981ms. - **Findings belong to us.** Legacy mode excludes cross-origin frames. YouTube embeds were counting against us, 11 of 15 violations on one page. - **A pass carries meaning.** A 404 reports as a load failure, not an a11y bug. A page scanned before it hydrates warns instead of quietly reporting clean. ## How the findings appear The test is named `has no blocking accessibility violations`, so a failure listed by CI is always something to fix. It is not named for the full rule set, because a green check would then claim more than the check verifies. | | Rules | Where you see it | | --- | --- | --- | | Blocking | `heading-order`, `page-has-heading-one` | Test failure, so the runner reports it on the PR | | Reported | Everything else in WCAG A/AA | `::warning` annotation on the run | An annotation looks like this, on a run that still passes: ``` ::warning title=Accessibility::/docs/guides/database/functions has 1 non-blocking accessibility finding(s): frame-title (4) ``` The full axe result for each page is attached to the report as `axe-results.json`. ## Matching the Studio ratchet This follows the ESLint ratchet in `apps/studio`. That pattern warns on pre-existing debt rather than blocking on it, surfaces findings as annotations rather than PR comments, and promotes a rule to an error once its violations reach zero. The mechanism here is `ENFORCED_RULES` in `utils/axe-helpers.ts`. The two heading rules are on it because the heading-hierarchy work drove them to zero site-wide. The intent is to migrate rules into that list one at a time. Pick a rule, fix its violations, then move it into `ENFORCED_RULES` so it cannot come back. An exhaustive scan of the site groups the current backlog by root cause to sequence that work, and two fixes cover 99.1% of it. Studio keeps per-file baseline counts, which this does not. A whole-rule list is coarser, and it works here because docs violations reach zero across the site rather than per file. ## Manual testing Install the browser once, then run each step from the repo root. Every command scans production, so you do not need a local docs server. ```bash pnpm -C e2e/docs exec playwright install chromium ``` 1. Confirm a reported finding does not fail the check. ```bash DOCS_E2E_PAGE_PATHS=/docs/guides/database/functions PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y ``` Expect `1 passed`, and the `::warning` annotation above in the output. 2. Confirm the scan finds that violation. Same page, now failing on every rule. ```bash A11Y_ENFORCE_ALL=1 DOCS_E2E_PAGE_PATHS=/docs/guides/database/functions PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y ``` Expect `1 failed`, reporting `frame-title (serious, 4 node(s))`. Steps 1 and 2 together are the point of this PR. 3. Confirm the skipped rules stay skipped. ```bash A11Y_ENFORCE_ALL=1 DOCS_E2E_PAGE_PATHS=/docs/guides/getting-started/quickstarts/nextjs PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y ``` Expect `button-name (critical, 2 node(s))` and `label (critical, 2 node(s))`, and no `color-contrast`. 4. Confirm a page that does not load reports a load failure. ```bash DOCS_E2E_PAGE_PATHS=/docs/guides/does-not-exist-xyz PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y ``` Expect `Expected a successful response for /docs/guides/does-not-exist-xyz, got 404`, and no axe assertion. 5. Confirm the link checker still passes alongside the a11y test. ```bash DOCS_E2E_PAGE_PATHS=/docs/guides/auth/passwords PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs ``` Expect `3 passed`. ## Known gaps - `/docs/reference/*` is not scanned. Those routes render client-side into tens of thousands of elements, where axe exceeds its timeout and results depend on whether the scan caught the page mid-render. - Shared chrome is outside the article scope, so nav, sidebar, footer, menus, and drawers are not covered. - axe catches roughly 30-40% of WCAG issues. Keyboard navigation, focus management, and screen reader behavior still need manual testing. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Miranda LimonczenkoandClaude Sonnet 5
|
52cb1c2600 |
feat(docs) Dynamically E2E test all docs-owned content (#48320)
Closes DOCS-1203 ## Problem The docs E2E workflow only ever tested one hardcoded page: the Next.js quickstart. All other docs content had no E2E coverage. ## Solution This PR expands the initial scaffolding to generalize the Next.js quickstart tests, page runs and checks local links, to all pages affecting Docs content: - Add `resolveDocsScope` (`e2e/docs/utils/resolve-docs-scope.ts`) to map changed guide and troubleshooting `.mdx` files to their `/docs/...` page paths, and to expand changed `_partials` to every page that includes them (including transitively, through partials nested inside other partials). Federated guide sections (`graphql`, `database/extensions/wrappers`, `ai/python`, `deployment/terraform`, `deployment/ci`) and reference docs stay out of scope, and resolution is capped at 20 pages to keep runtime bounded. - Replace the single `quickstarts.spec.ts` test with a generic `docs-pages.spec.ts` that loads whatever pages are resolved, asserting each renders with an `<h1>` and that its docs-owned links resolve. - Add `run-e2e-docs.ts` so `pnpm e2e:docs` resolves scope locally (from commits since `origin/master`, plus staged/unstaged changes) and skips Playwright entirely when nothing in scope changed. - Update `.github/workflows/docs-e2e.yml` to widen the trigger paths to all guides/troubleshooting/partials, resolve scope in a dedicated step, skip the rest of the job when scope is empty, and accept a `page_paths` input for manual `workflow_dispatch` runs. - Rewrite `e2e/docs/README.md` to document the new scoping behavior, the override envs (`DOCS_E2E_PAGE_PATHS`, `DOCS_E2E_BASE_REF`), and how CI uses the suite. - `pnpm e2e:docs:all` is also added to run tests on every page locally. Good for scoping issues but should not be included in CI. ## Manual testing Walk through the following steps to verify this works: - [x] `pnpm e2e:docs` from repo root resolves the expected pages for a local guide edit and can run against local dev **Note:** Challenges with testing on local in part because of the long lag for first page load. Recommendation to use a hosted URL is added to docs. - [x] Editing a shared `_partials` file resolves to every page that includes it (including through nested partials) - [x] `pnpm e2e:docs` exits cleanly with no Playwright run when no in-scope files changed - [x] `git diff --name-only ... | pnpm -C e2e/docs resolve-docs-scope` prints the expected page list for a sample diff - [x] Workflow run on a PR that only touches `e2e/docs`/workflow files skips the Playwright steps - [x] Manual `workflow_dispatch` run with `page_paths` set tests only those pages - [x] Run `pnpm e2e:docs:all` to run the suite on all docs content, which takes awhile ## Next steps After this PR merges, we have the scaffolding to add more fun tests like a11y 😁 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added scoped Docs E2E runs that target eligible doc pages based on changes, plus manual page-targeted runs and an “all eligible pages” mode. * Introduced `DOCS_E2E_PAGE_PATHS` (and updated base ref/base URL behavior) to control which pages are tested. * **Bug Fixes** * Automatically skips Playwright setup when no relevant pages are in scope; Playwright reporting now uploads only on failure. * **Documentation** * Updated the Docs E2E README with new run/CI behavior, troubleshooting notes, and commands to inspect the resolved page list. * **Tests** * Added a Docs-owned pages E2E suite; removed the Next.js quickstart E2E spec. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |