dima/supabase
1
0 Fork 0
mirror of https://github.com/supabase/supabase.git synced 2026-10-06 01:45:10 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
codex/fix-tanstack-e2e
supabase/e2e/www/README.md
T
Miranda LimonczenkoandClaude Opus 5 6d3a4bcc48 feat(www) Add scaffolding for WWW E2E tests and CI check (#48861) 
Closes DOCS-1278

## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Feature. Adds E2E test scaffolding and a CI check for the marketing
site.

## What is the current behavior?

Closes [FE-4047](https://linear.app/supabase/issue/FE-4047).

The marketing site has no E2E coverage. Docs has a suite in `e2e/docs`,
but its
runner, git helpers and axe reporting are private to that package, so a
second
site cannot reuse them.

## What is the new behavior?

* **A www suite scoped to changed content.** Changed `.mdx` files in
`_blog`,
`_events`, `_customers` and `_alternatives` map to the URLs they render.
Pages
with `disable_page_build: true` are skipped because they 404 by design.
Capped
at 20 pages. Enforces `heading-order` and `page-has-heading-one`,
matching docs.
* **`e2e/shared` The docs site is also static with similar needs. This
folder shares the docs logic with www.
* **A CI check that is safe to mark required.** Path scoping lives in a
`Detect changed paths` step rather than a `paths:` trigger, so the check
  reports on every pull request instead of being skipped.
`waitForVercelDocsPreview.js` becomes `waitForVercelPreview.js`, shared
by both
  workflows.

## How the check behaves

The job always reports a check run, so it is safe to mark required. Path
scoping
happens in a step rather than a `paths:` trigger, which would leave
non-www pull
requests waiting on a check that never reports.

| Case | Behavior |
| --- | --- |
| Fork pull request adds new pages | Passes without testing. The Vercel
wait is gated on `head.repo.full_name == github.repository`, so forks
resolve no preview URL. The job emits a `::warning` and a job summary
containing a ready-to-run `gh workflow run www-e2e.yml` command with the
resolved page paths, so a maintainer can run it against the preview. |
| Vercel preview times out or fails | Passes without testing. The wait
step is `continue-on-error: true`, so a 900s timeout or a failed
deployment leaves the URL unset and the suite skips. Vercel's own
`Vercel – zone-www-dot-com` check already reports the failure. |
| Draft pull request | Job does not run at all, gated at the job level
on `pull_request.draft == false`. `ready_for_review` is in the trigger's
`types`, so marking it ready runs the check. |
| Another app changed, www untouched | Job runs and every step skips.
The `www` filter matches only the four content directories, `e2e/www`,
`e2e/shared`, the lockfile, and this workflow. |
| Only the harness changed | Passes without testing. Scope resolves to
zero pages, and the Vercel wait is additionally gated on `www_app`, so
it does not wait for a preview Vercel skipped. |
| No preview resolves, any reason | Skips rather than falling back to
production. Production does not serve pages the pull request adds, so
testing it would fail a valid change. |

### Not covered

Changes to `apps/www` components and routes do not trigger this check —
only the
four content directories do. A follow-up can check global components
such as the navigation and the footer.

## Manual testing

1. Start the site: `pnpm dev:www`
2. Run `pnpm e2e:www` with no www content changed. It should resolve
zero pages
   and skip Playwright, not fail.
3. Touch a post, then run `pnpm e2e:www` again:
`echo "" >> apps/www/_blog/2024-01-01-some-post.mdx`. The resolved
`/blog/...`
   path should be listed before Playwright starts.
4. Run against production with no local server:
`PLAYWRIGHT_BASE_URL=https://supabase.com
WWW_E2E_PAGE_PATHS=/blog/postgres-language-server pnpm e2e:www`
5. Point step 4 at a page with a known heading problem. The failure
should name
   the rule, the CSS selector and the markup.
6. Confirm docs still passes on the shared runner: `pnpm dev:docs`, then
   `pnpm e2e:docs`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added WWW end-to-end testing for affected content pages, including
accessibility checks.
* Added standard and full-site test commands, configurable preview
testing, and failure reports.
* Added shared utilities for page discovery, accessibility scanning, and
test execution.

* **Documentation**
* Documented WWW test setup, coverage, debugging, CI behavior, and
running checks against production or preview environments.

* **Improvements**
* Updated documentation test workflows to better identify affected
changes and handle preview environments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-11 22:06:51 +00:00

170 lines
6.5 KiB
Markdown
Raw Permalink Blame History

# WWW E2E tests
This guide explains how to run Playwright end-to-end checks against marketing
site content pages.
Use this suite when you change blog posts, events, customer stories, or
alternatives pages under `apps/www`. It loads each in-scope page, checks that it
returns a successful status, and scans it for a single accessibility rule.
This page covers:
- [Set up](#set-up) — install the browser once
- [Run the tests](#run-the-tests) — the usual local command
- [Choose a target URL](#choose-a-target-url) — production, preview, or local www
- [Override which pages run](#override-which-pages-run) — when the default git
scope is wrong
- [What the suite covers](#what-the-suite-covers) — in-scope paths and limits
- [Debug failures](#debug-failures) — reports and traces
- [How CI uses this suite](#how-ci-uses-this-suite) — pull request behavior
## Set up
1. From this directory, install the Playwright Chromium browser once:
```bash
cd e2e/www
pnpm exec playwright install chromium
```
## Run the tests
By default, `pnpm e2e:www` tests pages affected by your current changes: commits
since `origin/master`, plus staged and unstaged working-tree files. If nothing
in scope changed, the command exits successfully without starting Playwright.
1. From the repository root, point the suite at a deployed site and run it:
```bash
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:www
```
Extra arguments pass through to Playwright, so `pnpm e2e:www --ui` opens UI mode
and `pnpm e2e:www --grep @a11y` runs only the accessibility assertions.
### Run every in-scope page
To test every content page instead of a changed-files scope — for example, a
periodic full-site check — run:
```bash
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:www:all
```
This ignores `WWW_E2E_PAGE_PATHS` and the 20-page cap described in
[Limits](#limits), and tests every page across all four content directories —
around 480 as of this writing. `--all` runs also default to
`--max-failures=0`, so a full run isn't cut short by `playwright.config.ts`'s
global `maxFailures: 3`. The suite runs one worker by default, so pass
`--workers` to parallelize it:
```bash
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:www:all -- --workers=4
```
Against production, raising workers does not pay off: a serial full run finishes
in about 17 minutes with no failures, while four workers took longer and timed
out on 34 of 480 navigations. Those timeouts are load, not page defects. Prefer
the default single worker unless you are pointed at a preview or a local server.
## Choose a target URL
Tests use `PLAYWRIGHT_BASE_URL`. When unset, they default to the local www dev
server at `http://localhost:3000`.
Prefer a deployed site for day-to-day checks. Use the local server only when you
need unpublished content that production does not serve yet.
For a protected Vercel preview, also set `VERCEL_AUTOMATION_BYPASS_SECRET`.
To use the local server, start it with `pnpm dev:www` from the repository root
and run the suite without `PLAYWRIGHT_BASE_URL`.
## Override which pages run
Leave `WWW_E2E_PAGE_PATHS` unset to keep the default changed-files scope.
To test specific pages instead of the git diff:
```bash
WWW_E2E_PAGE_PATHS=/blog/supabase-steve-chavez \
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:www
```
`WWW_E2E_PAGE_PATHS` accepts a comma- or newline-separated list of
site-relative paths. `WWW_E2E_BASE_REF` overrides the base ref the git diff
compares against.
## What the suite covers
### In scope
| Changed path | Behavior |
| ------------------------------ | --------------------------- |
| `apps/www/_blog/*.mdx` | Test `/blog/<slug>` |
| `apps/www/_events/*.mdx` | Test `/events/<slug>` |
| `apps/www/_customers/*.mdx` | Test `/customers/<slug>` |
| `apps/www/_alternatives/*.mdx` | Test `/alternatives/<slug>` |
Slugs come from the filename, matching `getAllPostSlugs` in
`apps/www/lib/posts.tsx`. Blog and event filenames drop their `YYYY-MM-DD-`
prefix; customers and alternatives use the filename as-is.
Each page gets one test: it must return a successful status, and an axe scan
must report no `page-has-heading-one` violations. That is the only rule enforced
today — add more to `ENFORCED_RULES` in `features/www-pages.spec.ts` once a class
of issue reaches zero across the site.
`ENFORCED_RULES` is deliberately separate from the docs suite's list. Docs
enforces `heading-order` as well; www cannot yet. A full-site scan found
`heading-order` violations on the large majority of content pages, almost all
from the same two shared components — the related-posts card (`h4` under an `h2`)
and a trailing `h6`. Enforcing it here would fail nearly every pull request.
### Out of scope
- Events with `disable_page_build: true`, which return a 404 by design
- Static marketing routes under `apps/www/pages` and `apps/www/app`
- Index and listing pages such as `/blog` and `/customers`
- Link checking, and every accessibility rule other than the one above
### Limits
Resolved scope is capped at 20 pages so a large content drop cannot explode
runtime. If a change resolves to more pages than that, only the first 20 in
sorted order are tested. To test beyond the cap, use `pnpm e2e:www:all` or set
`WWW_E2E_PAGE_PATHS` explicitly.
## Debug failures
1. Open the HTML report after a run:
```bash
pnpm -C e2e/www exec playwright show-report
```
2. Inspect traces and screenshots under `test-results/` for failed runs.
## How CI uses this suite
The workflow at `.github/workflows/www-e2e.yml` runs on pull requests that touch
owned www content, `e2e/www`, or `e2e/shared`.
1. Diff the pull request against its base branch and resolve in-scope page paths.
2. Skip Playwright when nothing in scope changed.
3. When `apps/www` changed, wait for the Vercel www preview and set
`PLAYWRIGHT_BASE_URL` to that preview. When no preview resolves, skip rather
than test against production, which does not have pages the pull request adds.
4. Run the suite with `WWW_E2E_PAGE_PATHS` set to the resolved list.
Draft pull requests stay skipped until you mark them ready for review. Manual
`workflow_dispatch` runs require a `page_paths` input and accept an optional
`base_url`, which defaults to production.
## Shared helpers
`e2e/shared` holds the pieces this suite and `e2e/docs` both use: git diff
collection, page-path parsing, the runner, and the scope-resolver CLI. Suite
directories keep only what is specific to them — for www, that is the
content-file-to-URL mapping in `utils/resolve-www-scope.ts`.
Reference in new issue View Git Blame Copy Permalink