feat(www) Add scaffolding for WWW E2E tests and CI check (#48861)

Closes DOCS-1278

## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Feature. Adds E2E test scaffolding and a CI check for the marketing
site.

## What is the current behavior?

Closes [FE-4047](https://linear.app/supabase/issue/FE-4047).

The marketing site has no E2E coverage. Docs has a suite in `e2e/docs`,
but its
runner, git helpers and axe reporting are private to that package, so a
second
site cannot reuse them.

## What is the new behavior?

* **A www suite scoped to changed content.** Changed `.mdx` files in
`_blog`,
`_events`, `_customers` and `_alternatives` map to the URLs they render.
Pages
with `disable_page_build: true` are skipped because they 404 by design.
Capped
at 20 pages. Enforces `heading-order` and `page-has-heading-one`,
matching docs.
* **`e2e/shared` The docs site is also static with similar needs. This
folder shares the docs logic with www.
* **A CI check that is safe to mark required.** Path scoping lives in a
`Detect changed paths` step rather than a `paths:` trigger, so the check
  reports on every pull request instead of being skipped.
`waitForVercelDocsPreview.js` becomes `waitForVercelPreview.js`, shared
by both
  workflows.

## How the check behaves

The job always reports a check run, so it is safe to mark required. Path
scoping
happens in a step rather than a `paths:` trigger, which would leave
non-www pull
requests waiting on a check that never reports.

| Case | Behavior |
| --- | --- |
| Fork pull request adds new pages | Passes without testing. The Vercel
wait is gated on `head.repo.full_name == github.repository`, so forks
resolve no preview URL. The job emits a `::warning` and a job summary
containing a ready-to-run `gh workflow run www-e2e.yml` command with the
resolved page paths, so a maintainer can run it against the preview. |
| Vercel preview times out or fails | Passes without testing. The wait
step is `continue-on-error: true`, so a 900s timeout or a failed
deployment leaves the URL unset and the suite skips. Vercel's own
`Vercel – zone-www-dot-com` check already reports the failure. |
| Draft pull request | Job does not run at all, gated at the job level
on `pull_request.draft == false`. `ready_for_review` is in the trigger's
`types`, so marking it ready runs the check. |
| Another app changed, www untouched | Job runs and every step skips.
The `www` filter matches only the four content directories, `e2e/www`,
`e2e/shared`, the lockfile, and this workflow. |
| Only the harness changed | Passes without testing. Scope resolves to
zero pages, and the Vercel wait is additionally gated on `www_app`, so
it does not wait for a preview Vercel skipped. |
| No preview resolves, any reason | Skips rather than falling back to
production. Production does not serve pages the pull request adds, so
testing it would fail a valid change. |

### Not covered

Changes to `apps/www` components and routes do not trigger this check —
only the
four content directories do. A follow-up can check global components
such as the navigation and the footer.

## Manual testing

1. Start the site: `pnpm dev:www`
2. Run `pnpm e2e:www` with no www content changed. It should resolve
zero pages
   and skip Playwright, not fail.
3. Touch a post, then run `pnpm e2e:www` again:
`echo "" >> apps/www/_blog/2024-01-01-some-post.mdx`. The resolved
`/blog/...`
   path should be listed before Playwright starts.
4. Run against production with no local server:
`PLAYWRIGHT_BASE_URL=https://supabase.com
WWW_E2E_PAGE_PATHS=/blog/postgres-language-server pnpm e2e:www`
5. Point step 4 at a page with a known heading problem. The failure
should name
   the rule, the CSS selector and the markup.
6. Confirm docs still passes on the shared runner: `pnpm dev:docs`, then
   `pnpm e2e:docs`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added WWW end-to-end testing for affected content pages, including
accessibility checks.
* Added standard and full-site test commands, configurable preview
testing, and failure reports.
* Added shared utilities for page discovery, accessibility scanning, and
test execution.

* **Documentation**
* Documented WWW test setup, coverage, debugging, CI behavior, and
running checks against production or preview environments.

* **Improvements**
* Updated documentation test workflows to better identify affected
changes and handle preview environments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

.github/workflows/docs-e2e.yml

@@ -55,12 +55,8 @@ jobs:
               - 'apps/docs/content/guides/**/*.mdx'
               - 'apps/docs/content/troubleshooting/**/*.mdx'
               - 'apps/docs/content/_partials/**'
-              - 'e2e/docs/features/**'
-              - 'e2e/docs/utils/**'
-              - 'e2e/docs/scripts/**'
-              - 'e2e/docs/playwright.config.ts'
-              - 'e2e/docs/package.json'
-              - 'e2e/docs/tsconfig.json'
+              - 'e2e/docs/**'
+              - 'e2e/shared/**'
               - 'pnpm-lock.yaml'
               - '.github/workflows/docs-e2e.yml'
             docs_app:
@@ -75,6 +71,7 @@ jobs:
           fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
           sparse-checkout: |
             e2e/docs
+            e2e/shared
             scripts
             patches
             apps/docs/content/guides
@@ -138,17 +135,18 @@ jobs:
       # times out polling that API even though the preview builds fine.
       # Poll the "Vercel – docs" commit status instead — Vercel keeps posting
       # those — then resolve the deployment it points to via Vercel's own API
-      # to get the actual preview URL. See scripts/waitForVercelDocsPreview.js.
+      # to get the actual preview URL. See scripts/waitForVercelPreview.js.
       # A Vercel failure or timeout is not the author's problem, and the required
       # "Vercel – docs" check already reports it. Resolve no URL and skip below.
       - name: Wait for Vercel docs preview
         if: steps.scope.outputs.skip == 'false' && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && steps.changes.outputs.docs_app == 'true'
         id: deployment
         continue-on-error: true
-        run: node scripts/waitForVercelDocsPreview.js
+        run: node scripts/waitForVercelPreview.js
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+          VERCEL_STATUS_CONTEXT: 'Vercel – docs'
           VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
           VERCEL_TEAM_ID: ${{ secrets.VERCEL_TEAM_ID }}
 

.github/workflows/www-e2e.yml

@@ -0,0 +1,208 @@
+name: WWW E2E Tests
+
+# Path scoping lives in "Detect changed paths" rather than a `paths` trigger, so
+# this reports a check run on every pull request and stays safe to mark required.
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review, converted_to_draft]
+    branches: ['master']
+  workflow_dispatch:
+    inputs:
+      base_url:
+        description: 'Base URL to test against'
+        required: false
+        default: 'https://supabase.com'
+        type: string
+      page_paths:
+        description: 'Comma-separated site-relative paths to test (required for manual runs)'
+        required: false
+        default: ''
+        type: string
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+  statuses: read
+  pull-requests: read
+
+env:
+  CI: true
+
+jobs:
+  e2e:
+    name: WWW E2E
+    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.draft == false
+    timeout-minutes: 30
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+
+    steps:
+      - name: Detect changed paths
+        id: changes
+        if: github.event_name == 'pull_request'
+        uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
+        with:
+          filters: |
+            www:
+              - 'apps/www/_blog/**/*.mdx'
+              - 'apps/www/_events/**/*.mdx'
+              - 'apps/www/_customers/**/*.mdx'
+              - 'apps/www/_alternatives/**/*.mdx'
+              - 'e2e/www/**'
+              - 'e2e/shared/**'
+              - 'pnpm-lock.yaml'
+              - '.github/workflows/www-e2e.yml'
+            www_app:
+              - 'apps/www/**'
+
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        if: github.event_name == 'workflow_dispatch' || steps.changes.outputs.www == 'true'
+        with:
+          persist-credentials: false
+          # String '0' — numeric 0 is falsy in Actions expressions.
+          fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
+          sparse-checkout: |
+            e2e/www
+            e2e/shared
+            scripts
+            patches
+            apps/www/_blog
+            apps/www/_events
+            apps/www/_customers
+            apps/www/_alternatives
+
+      - name: Use Node.js
+        if: github.event_name == 'workflow_dispatch' || steps.changes.outputs.www == 'true'
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
+        with:
+          node-version-file: '.nvmrc'
+
+      - name: Resolve www E2E scope
+        id: scope
+        if: github.event_name == 'workflow_dispatch' || steps.changes.outputs.www == 'true'
+        env:
+          EVENT_NAME: ${{ github.event_name }}
+          BASE_REF: ${{ github.base_ref }}
+          PAGE_PATHS_INPUT: ${{ inputs.page_paths }}
+        run: |
+          if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
+            if [ -z "$PAGE_PATHS_INPUT" ]; then
+              echo "skip=true" >> "$GITHUB_OUTPUT"
+              echo "paths=" >> "$GITHUB_OUTPUT"
+              echo "Manual run requires the page_paths input."
+              exit 0
+            fi
+            echo "skip=false" >> "$GITHUB_OUTPUT"
+            printf 'paths=%s\n' "$PAGE_PATHS_INPUT" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          git diff --name-only --diff-filter=ACMR "origin/$BASE_REF"...HEAD \
+            | node --experimental-strip-types e2e/www/scripts/resolve-www-scope.ts
+
+      - name: Skip Playwright (no in-scope pages)
+        if: steps.scope.outputs.skip == 'true'
+        run: echo "No in-scope www pages changed; skipping Playwright suite."
+
+      - uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9
+        if: steps.scope.outputs.skip == 'false'
+        name: Install pnpm
+        with:
+          run_install: false
+
+      - name: Enable pnpm store cache
+        if: steps.scope.outputs.skip == 'false'
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
+        with:
+          node-version-file: '.nvmrc'
+          cache: 'pnpm'
+
+      # Vercel skips the preview when only the harness changed, so wait for one
+      # only when apps/www changed. See scripts/waitForVercelPreview.js.
+      - name: Wait for Vercel www preview
+        if: steps.scope.outputs.skip == 'false' && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && steps.changes.outputs.www_app == 'true'
+        id: deployment
+        continue-on-error: true
+        run: node scripts/waitForVercelPreview.js
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+          VERCEL_STATUS_CONTEXT: 'Vercel – zone-www-dot-com'
+          VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
+          VERCEL_TEAM_ID: ${{ secrets.VERCEL_TEAM_ID }}
+
+      - name: Resolve base URL
+        if: steps.scope.outputs.skip == 'false'
+        id: base-url
+        env:
+          EVENT_NAME: ${{ github.event_name }}
+          BASE_URL_INPUT: ${{ inputs.base_url }}
+          DEPLOYMENT_URL: ${{ steps.deployment.outputs.deployment-url }}
+          PAGE_PATHS: ${{ steps.scope.outputs.paths }}
+        run: |
+          set -euo pipefail
+
+          if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
+            printf 'url=%s\n' "$BASE_URL_INPUT" >> "$GITHUB_OUTPUT"
+            if [ "$BASE_URL_INPUT" = "https://supabase.com" ]; then
+              echo "use_bypass=false" >> "$GITHUB_OUTPUT"
+            else
+              echo "use_bypass=true" >> "$GITHUB_OUTPUT"
+            fi
+            echo "should_test=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          if [ -n "$DEPLOYMENT_URL" ]; then
+            printf 'url=%s\n' "$DEPLOYMENT_URL" >> "$GITHUB_OUTPUT"
+            echo "use_bypass=true" >> "$GITHUB_OUTPUT"
+            echo "should_test=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          # Production lacks pages this pull request adds, so it is not a fallback.
+          echo "url=" >> "$GITHUB_OUTPUT"
+          echo "use_bypass=false" >> "$GITHUB_OUTPUT"
+          echo "should_test=false" >> "$GITHUB_OUTPUT"
+          echo "::warning::No Vercel www preview URL for this pull request. Skipping Playwright rather than testing production, which does not have pages this pull request adds."
+          {
+            echo "### WWW E2E skipped: no preview to test against"
+            echo
+            echo "Fork pull requests reach this path because they run without repository"
+            echo "secrets. A maintainer can run the suite against the preview manually:"
+            echo
+            echo '```'
+            echo "gh workflow run www-e2e.yml \\"
+            echo "  -f base_url=<preview-url> \\"
+            echo "  -f page_paths=$PAGE_PATHS"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Install dependencies
+        if: steps.base-url.outputs.should_test == 'true'
+        run: pnpm install --frozen-lockfile --filter=e2e-www...
+
+      - name: Install Playwright Chromium
+        if: steps.base-url.outputs.should_test == 'true'
+        run: pnpm -C e2e/www exec playwright install chromium --with-deps --only-shell
+
+      - name: Run www E2E
+        if: steps.base-url.outputs.should_test == 'true'
+        working-directory: e2e/www
+        run: pnpm run e2e:www
+        env:
+          PLAYWRIGHT_BASE_URL: ${{ steps.base-url.outputs.url }}
+          WWW_E2E_PAGE_PATHS: ${{ steps.scope.outputs.paths }}
+          VERCEL_AUTOMATION_BYPASS_SECRET: ${{ steps.base-url.outputs.use_bypass == 'true' && secrets.VERCEL_AUTOMATION_BYPASS_WWW || '' }}
+
+      - name: Upload Playwright report
+        if: failure() && steps.scope.outputs.skip == 'false'
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: www-playwright-report
+          path: |
+            e2e/www/playwright-report/
+            e2e/www/test-results/
+          retention-days: 7