feat(docs) Dynamically E2E test all docs-owned content (#48320)

Closes DOCS-1203

## Problem

The docs E2E workflow only ever tested one hardcoded page: the Next.js
quickstart. All other docs content had no E2E coverage.

## Solution

This PR expands the initial scaffolding to generalize the Next.js
quickstart tests, page runs and checks local links, to all pages
affecting Docs content:


- Add `resolveDocsScope` (`e2e/docs/utils/resolve-docs-scope.ts`) to map
changed guide and troubleshooting `.mdx` files to their `/docs/...` page
paths, and to expand changed `_partials` to every page that includes
them (including transitively, through partials nested inside other
partials). Federated guide sections (`graphql`,
`database/extensions/wrappers`, `ai/python`, `deployment/terraform`,
`deployment/ci`) and reference docs stay out of scope, and resolution is
capped at 20 pages to keep runtime bounded.
- Replace the single `quickstarts.spec.ts` test with a generic
`docs-pages.spec.ts` that loads whatever pages are resolved, asserting
each renders with an `<h1>` and that its docs-owned links resolve.
- Add `run-e2e-docs.ts` so `pnpm e2e:docs` resolves scope locally (from
commits since `origin/master`, plus staged/unstaged changes) and skips
Playwright entirely when nothing in scope changed.
- Update `.github/workflows/docs-e2e.yml` to widen the trigger paths to
all guides/troubleshooting/partials, resolve scope in a dedicated step,
skip the rest of the job when scope is empty, and accept a `page_paths`
input for manual `workflow_dispatch` runs.
- Rewrite `e2e/docs/README.md` to document the new scoping behavior, the
override envs (`DOCS_E2E_PAGE_PATHS`, `DOCS_E2E_BASE_REF`), and how CI
uses the suite.
- `pnpm e2e:docs:all` is also added to run tests on every page locally.
Good for scoping issues but should not be included in CI.

## Manual testing

Walk through the following steps to verify this works:

- [x] `pnpm e2e:docs` from repo root resolves the expected pages for a
local guide edit and can run against local dev
**Note:** Challenges with testing on local in part because of the long
lag for first page load. Recommendation to use a hosted URL is added to
docs.
- [x] Editing a shared `_partials` file resolves to every page that
includes it (including through nested partials)
- [x] `pnpm e2e:docs` exits cleanly with no Playwright run when no
in-scope files changed
- [x] `git diff --name-only ... | pnpm -C e2e/docs resolve-docs-scope`
prints the expected page list for a sample diff
- [x] Workflow run on a PR that only touches `e2e/docs`/workflow files
skips the Playwright steps
- [x] Manual `workflow_dispatch` run with `page_paths` set tests only
those pages
- [x] Run `pnpm e2e:docs:all` to run the suite on all docs content,
which takes awhile

## Next steps

After this PR merges, we have the scaffolding to add more fun tests like
a11y 😁

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added scoped Docs E2E runs that target eligible doc pages based on
changes, plus manual page-targeted runs and an “all eligible pages”
mode.
* Introduced `DOCS_E2E_PAGE_PATHS` (and updated base ref/base URL
behavior) to control which pages are tested.
* **Bug Fixes**
* Automatically skips Playwright setup when no relevant pages are in
scope; Playwright reporting now uploads only on failure.
* **Documentation**
* Updated the Docs E2E README with new run/CI behavior, troubleshooting
notes, and commands to inspect the resolved page list.
* **Tests**
* Added a Docs-owned pages E2E suite; removed the Next.js quickstart E2E
spec.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

e2e/docs/README.md

@@ -1,83 +1,194 @@
-# Supabase Docs E2E Tests
+# Docs E2E tests
 
-Playwright end-to-end tests for the docs site under `apps/docs`.
-Add new docs journeys under `features/`. They all run together as one suite.
+This guide explains how to run Playwright end-to-end checks against docs pages
+this repo owns.
 
-## Setup
+Use this suite when you change guides, troubleshooting entries, or shared
+partials under `apps/docs/content`. It loads each in-scope page, checks that the
+article renders, and verifies that docs-owned links in the article resolve.
 
-Install the Playwright browser once from this directory:
+This page covers:
+
+- [Set up](#set-up) — install the browser once
+- [Run the tests](#run-the-tests) — the usual local command
+- [Choose a target URL](#choose-a-target-url) — production, preview, or local docs
+- [Override which pages run](#override-which-pages-run) — when the default git
+  scope is wrong
+- [What the suite covers](#what-the-suite-covers) — in-scope paths and limits
+- [Debug failures](#debug-failures) — reports and traces
+- [How CI uses this suite](#how-ci-uses-this-suite) — pull request behavior
+
+## Set up
+
+1. From this directory, install the Playwright Chromium browser once:
+
+   ```bash
+   cd e2e/docs
+   pnpm exec playwright install chromium
+   ```
+
+## Run the tests
+
+By default, `pnpm e2e:docs` tests pages affected by your current changes:
+commits since `origin/master`, plus staged and unstaged working-tree files. If
+nothing in scope changed, the command exits successfully without starting
+Playwright.
+
+1. From the repository root, point the suite at a deployed docs site and run it:
+
+   ```bash
+   PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs
+   ```
+
+2. Optional: open Playwright UI mode for the same scoped run:
+
+   ```bash
+   PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:ui
+   ```
+
+You can also run from `e2e/docs` with `pnpm run e2e:docs`.
+
+## Choose a target URL
+
+Tests use `PLAYWRIGHT_BASE_URL`. When unset, they default to the local docs
+dev server at `http://localhost:3001`.
+
+Prefer a deployed site for day-to-day checks. Use the local server only when you
+need unpublished content that production does not serve yet.
+
+### Deployed site
 
 ```bash
-cd e2e/docs
-pnpm exec playwright install chromium
+PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs
 ```
 
-## Choosing a target URL
+For a protected Vercel preview, also set `VERCEL_AUTOMATION_BYPASS_SECRET`.
 
-Tests run against whatever `PLAYWRIGHT_BASE_URL` points to, defaulting to the
-local docs dev server at `http://localhost:3001`.
+### Local docs server
 
-- **Local docs server** — in a separate terminal, start docs from the repo root:
+1. From the repository root, start docs in a separate terminal:
 
-  ```bash
-  pnpm dev:docs
-  ```
+   ```bash
+   pnpm dev:docs
+   ```
 
-- **A deployed site** — set the base URL inline:
+2. Run the suite without `PLAYWRIGHT_BASE_URL`, or set it to
+   `http://localhost:3001`.
 
-  ```bash
+The local server needs a full monorepo install and credentials for some content.
+
+Local runs are unreliable for pages whose docs-owned links point into
+`/docs/reference/*` or `/docs/guides/auth/server-side/*`: reference pages can
+take over a minute to compile on first request in dev mode, which exceeds the
+suite's per-test timeout, and `server-side` auth guides have a known local-only
+routing issue that 404s even though the page serves correctly in production.
+Prefer a deployed site for pages that link into either of those sections.
+
+## Override which pages run
+
+Leave `DOCS_E2E_PAGE_PATHS` unset to keep the default changed-files scope.
+
+To test specific pages instead of the git diff:
+
+```bash
+DOCS_E2E_PAGE_PATHS=/docs/guides/getting-started/quickstarts/nextjs \
   PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs
-  ```
-
-If the target is a protected Vercel preview, also set
-`VERCEL_AUTOMATION_BYPASS_SECRET` so the tests can bypass deployment protection.
-
-The local docs server requires a full monorepo install and credentials for some
-content, so the quickest way to run the suite is against a deployed site. Reach
-for the local server only when you need to test unpublished content changes.
-
-## Running the tests
-
-From the repo root:
-
-```bash
-pnpm e2e:docs
 ```
 
-Or from this directory:
+To compare against a different base ref:
 
 ```bash
-pnpm run e2e:docs
+DOCS_E2E_BASE_REF=origin/develop \
+  PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs
 ```
 
-### UI mode for debugging
+`DOCS_E2E_PAGE_PATHS` accepts a comma- or newline-separated list of `/docs/...`
+paths.
+
+### Run every in-scope page
+
+To test every guide and troubleshooting entry instead of a changed-files scope
+— for example, a periodic full-site check — run:
 
 ```bash
-pnpm e2e:docs:ui
+PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:all
 ```
 
-### Run a single file
+This ignores `DOCS_E2E_PAGE_PATHS` and the 20-page cap described in
+[Limits](#limits), and tests every page listed by
+`pnpm -C e2e/docs resolve-docs-scope` across the whole `guides` and
+`troubleshooting` trees — several hundred pages as of this writing. `--all`
+runs also default to `--max-failures=0`, so a full run isn't cut short by
+`playwright.config.ts`'s global `maxFailures: 3`. Expect a long run: the suite
+runs one worker by default, so pass `--workers` to parallelize it, for
+example:
 
 ```bash
-pnpm run e2e:docs -- features/<name>.spec.ts
+PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:all -- --workers=4
 ```
 
-## Debugging
+Run this against a deployed site, not the local dev server — see
+[Local docs server](#local-docs-server) for why local runs are unreliable for
+pages linking into reference docs or server-side auth guides.
 
-- View the HTML report after a run:
+## What the suite covers
 
-  ```bash
-  pnpm -C e2e/docs exec playwright show-report
-  ```
+### In scope
 
-- Traces and screenshots for failures are saved in `test-results/`.
+| Changed path                                 | Behavior                                                 |
+| -------------------------------------------- | -------------------------------------------------------- |
+| `apps/docs/content/guides/**/*.mdx`          | Test `/docs/guides/<slug>`, excluding federated sections |
+| `apps/docs/content/troubleshooting/**/*.mdx` | Test `/docs/guides/troubleshooting/<slug>`               |
+| `apps/docs/content/_partials/**`             | Test owned pages that include that partial               |
 
-## CI
+### Out of scope
 
-`.github/workflows/docs-e2e.yml` runs this suite on pull requests that touch the
-tested docs pages or `e2e/docs`. When the PR changes `apps/docs`, the workflow
-waits for the matching Vercel preview and points `PLAYWRIGHT_BASE_URL` at it.
-When only the harness or workflow changes, Vercel skips the docs preview, so
-the suite falls back to production. Draft PRs are skipped until marked ready
-for review. The workflow can also be triggered manually with an optional
-`base_url` input that defaults to production.
+- Federated guide sections: `graphql`, `database/extensions/wrappers`,
+  `ai/python`, `deployment/terraform`, `deployment/ci`
+- Reference docs under `/docs/reference`
+- Non-docs routes such as `/dashboard` and `/ui`, which the link checker skips
+
+### Limits
+
+Resolved scope is capped at 20 pages so a widely shared partial cannot explode
+runtime. If a change resolves to more pages than that, only the first 20 in
+sorted order are tested and the rest are silently dropped from that run. To
+test beyond the cap, use `pnpm e2e:docs:all` instead of raising it. See
+[Run every in-scope page](#run-every-in-scope-page).
+
+To inspect the resolved list without running Playwright, replicate the same
+scope `pnpm e2e:docs` uses by default: commits since `origin/master`, plus
+staged and unstaged working-tree changes.
+
+```bash
+{
+  git diff --name-only --diff-filter=ACMR origin/master...HEAD
+  git diff --name-only --diff-filter=ACMR
+  git diff --name-only --diff-filter=ACMR --cached
+} | pnpm -C e2e/docs resolve-docs-scope
+```
+
+## Debug failures
+
+1. Open the HTML report after a run:
+
+   ```bash
+   pnpm -C e2e/docs exec playwright show-report
+   ```
+
+2. Inspect traces and screenshots under `test-results/` for failed runs.
+
+## How CI uses this suite
+
+The workflow at `.github/workflows/docs-e2e.yml` runs on pull requests that touch
+owned docs content, partials, or `e2e/docs`.
+
+1. Diff the pull request against its base branch and resolve in-scope page paths.
+2. Skip Playwright when nothing in scope changed.
+3. When `apps/docs` changed, wait for the Vercel docs preview and set
+   `PLAYWRIGHT_BASE_URL` to that preview. Otherwise use production.
+4. Run the suite with `DOCS_E2E_PAGE_PATHS` set to the resolved list.
+
+Draft pull requests stay skipped until you mark them ready for review. Manual
+`workflow_dispatch` runs require a `page_paths` input and accept an optional
+`base_url`, which defaults to production.

e2e/docs/features/docs-pages.spec.ts

@@ -0,0 +1,67 @@
+import { expect, test } from '@playwright/test'
+
+import {
+  articleSelectorForPagePath,
+  browserLikeUserAgent,
+  collectDocsOwnedLinks,
+  parseDocsE2EPagePaths,
+} from '../utils/docs-links.js'
+
+const pagePaths = parseDocsE2EPagePaths(process.env.DOCS_E2E_PAGE_PATHS)
+
+test.describe('Docs owned pages', () => {
+  // playwright.config.ts sets fullyParallel: false, and Playwright shards
+  // work by file rather than by test in that mode — without this, every test
+  // in this single spec file runs on one worker no matter what --workers is
+  // passed. Opt this describe block into parallel scheduling explicitly.
+  test.describe.configure({ mode: 'parallel' })
+
+  test('resolved page list must not be empty', () => {
+    expect(
+      pagePaths.length,
+      'No pages to test. `pnpm e2e:docs` resolves pages from git changes by default, ' +
+        'or set DOCS_E2E_PAGE_PATHS explicitly.'
+    ).toBeGreaterThan(0)
+  })
+
+  for (const pagePath of pagePaths) {
+    test(`${pagePath} loads and docs-owned article links resolve`, async ({ page }, testInfo) => {
+      const baseURL = testInfo.project.use.baseURL
+      expect(baseURL, 'A Playwright base URL should be configured').toBeTruthy()
+
+      const articleSelector = articleSelectorForPagePath(pagePath)
+      const response = await page.goto(pagePath)
+      expect(response, `Expected a response for ${pagePath}`).not.toBeNull()
+      expect(
+        response!.ok(),
+        `Page should return a successful status, got ${response!.status()}`
+      ).toBeTruthy()
+
+      const article = page.locator(articleSelector)
+      await expect(article, 'Page article should be present').toBeVisible()
+      await expect(
+        article.getByRole('heading', { level: 1 }),
+        'Page article should include an h1'
+      ).toBeVisible()
+
+      const links = await collectDocsOwnedLinks(page, baseURL!, articleSelector)
+      const userAgent = await browserLikeUserAgent(page)
+
+      for (const url of links) {
+        try {
+          const linkResponse = await page.request.get(url, { headers: { 'user-agent': userAgent } })
+          expect
+            .soft(linkResponse.ok(), `${url} should resolve (status ${linkResponse.status()})`)
+            .toBeTruthy()
+        } catch (error) {
+          expect
+            .soft(
+              null,
+              `${url} should be reachable (${error instanceof Error ? error.message : error})`
+            )
+            .toBeTruthy()
+        }
+      }
+    })
+  }
+})

e2e/docs/features/quickstarts.spec.ts

@@ -1,45 +0,0 @@
-import { expect, test } from '@playwright/test'
-
-import { collectDocsOwnedLinks } from '../utils/docs-links.js'
-
-const QUICKSTART_PATH = '/docs/guides/getting-started/quickstarts/nextjs'
-const ARTICLE_SELECTOR = '#sb-docs-guide-main-article'
-
-test.describe('Next.js quickstart', () => {
-  test('loads and docs-owned article links resolve', async ({ page }, testInfo) => {
-    const baseURL = testInfo.project.use.baseURL
-    expect(baseURL, 'A Playwright base URL should be configured').toBeTruthy()
-
-    const response = await page.goto(QUICKSTART_PATH)
-    expect(response, `Expected a response for ${QUICKSTART_PATH}`).not.toBeNull()
-    expect(
-      response!.ok(),
-      `Quickstart page should return a successful status, got ${response!.status()}`
-    ).toBeTruthy()
-
-    const article = page.locator(ARTICLE_SELECTOR)
-    await expect(article, 'Guide article should be present').toBeVisible()
-    await expect(
-      article.getByRole('heading', { level: 1 }),
-      'Guide article should include an h1'
-    ).toBeVisible()
-
-    const links = await collectDocsOwnedLinks(page, baseURL!)
-
-    for (const url of links) {
-      try {
-        const linkResponse = await page.request.get(url)
-        expect
-          .soft(linkResponse.ok(), `${url} should resolve (status ${linkResponse.status()})`)
-          .toBeTruthy()
-      } catch (error) {
-        expect
-          .soft(
-            null,
-            `${url} should be reachable (${error instanceof Error ? error.message : error})`
-          )
-          .toBeTruthy()
-      }
-    }
-  })
-})

e2e/docs/package.json

@@ -4,11 +4,16 @@
   "private": true,
   "type": "module",
   "scripts": {
-    "e2e:docs": "playwright test",
-    "e2e:ui": "playwright test --ui",
-    "e2e:docs:local-smoke": "playwright test --config=playwright.local-smoke.config.ts"
+    "e2e:docs": "node --experimental-strip-types scripts/run-e2e-docs.ts",
+    "e2e:docs:all": "node --experimental-strip-types scripts/run-e2e-docs.ts --all",
+    "e2e:ui": "node --experimental-strip-types scripts/run-e2e-docs.ts --ui",
+    "e2e:docs:local-smoke": "playwright test --config=playwright.local-smoke.config.ts",
+    "resolve-docs-scope": "node --experimental-strip-types scripts/resolve-docs-scope.ts"
   },
   "dependencies": {
     "@playwright/test": "^1.59.1"
+  },
+  "devDependencies": {
+    "@types/node": "catalog:"
   }
 }

e2e/docs/scripts/resolve-docs-scope.ts

@@ -0,0 +1,82 @@
+#!/usr/bin/env node
+/**
+ * Resolve docs E2E page scope from changed files.
+ *
+ * Usage:
+ *   git diff --name-only origin/master...HEAD | node --experimental-strip-types scripts/resolve-docs-scope.ts
+ *   node --experimental-strip-types scripts/resolve-docs-scope.ts --files a.mdx,b.mdx
+ *
+ * Outputs (GitHub Actions friendly):
+ *   skip=true|false
+ *   paths=<comma-separated /docs/... paths>
+ * Also prints each path on its own line to stderr for debugging.
+ */
+import { dirname, join } from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+import { parseChangedFilesList, resolveDocsScope } from '../utils/resolve-docs-scope.ts'
+
+const __dirname = dirname(fileURLToPath(import.meta.url))
+const REPO_ROOT = join(__dirname, '../../..')
+
+function readChangedFilesFromArgv(argv: string[]): string[] | null {
+  const filesIdx = argv.indexOf('--files')
+  if (filesIdx !== -1 && argv[filesIdx + 1]) {
+    return parseChangedFilesList(argv[filesIdx + 1])
+  }
+  return null
+}
+
+async function readStdin(): Promise<string> {
+  const chunks: Buffer[] = []
+  for await (const chunk of process.stdin) {
+    chunks.push(typeof chunk === 'string' ? Buffer.from(chunk) : chunk)
+  }
+  return Buffer.concat(chunks).toString('utf8')
+}
+
+async function main() {
+  const argv = process.argv.slice(2)
+  let changedFiles = readChangedFilesFromArgv(argv)
+
+  if (!changedFiles) {
+    if (process.stdin.isTTY) {
+      console.error('Pass changed files via stdin or --files path1,path2')
+      process.exit(2)
+    }
+    changedFiles = parseChangedFilesList(await readStdin())
+  }
+
+  const result = await resolveDocsScope({
+    changedFiles,
+    repoRoot: REPO_ROOT,
+  })
+
+  // GitHub Actions step outputs
+  const githubOutput = process.env.GITHUB_OUTPUT
+  const skipLine = `skip=${result.skip}`
+  const pathsLine = `paths=${result.pages.join(',')}`
+
+  if (githubOutput) {
+    // appendFile via sync to keep the CLI dependency-free
+    const { appendFileSync } = await import('node:fs')
+    appendFileSync(githubOutput, `${skipLine}\n${pathsLine}\n`)
+  } else {
+    console.log(skipLine)
+    console.log(pathsLine)
+  }
+
+  if (result.pages.length > 0) {
+    console.error(`Resolved ${result.pages.length} docs page(s):`)
+    for (const page of result.pages) {
+      console.error(`  ${page}`)
+    }
+  } else {
+    console.error('No in-scope docs pages — skipping Playwright suite.')
+  }
+}
+
+main().catch((error) => {
+  console.error(error instanceof Error ? error.message : error)
+  process.exit(1)
+})

e2e/docs/scripts/run-e2e-docs.ts

@@ -0,0 +1,180 @@
+#!/usr/bin/env node
+/**
+ * Default entry for `pnpm e2e:docs`.
+ *
+ * If DOCS_E2E_PAGE_PATHS is already set (CI, or an explicit local override),
+ * runs Playwright with that list. Otherwise resolves pages from files changed
+ * vs DOCS_E2E_BASE_REF (default origin/master), including the working tree.
+ *
+ * Pass `--all` to test every in-scope guide and troubleshooting page instead
+ * (hundreds of pages — expect a long run against a deployed site).
+ *
+ * Extra CLI args are forwarded to Playwright (e.g. --ui, a spec file path).
+ */
+import { spawn, spawnSync } from 'node:child_process'
+import { dirname, join } from 'node:path'
+import { fileURLToPath } from 'node:url'
+
+import {
+  parseChangedFilesList,
+  resolveAllDocsPages,
+  resolveDocsScope,
+} from '../utils/resolve-docs-scope.ts'
+
+const __dirname = dirname(fileURLToPath(import.meta.url))
+const E2E_DOCS_ROOT = join(__dirname, '..')
+
+// Mirrors the default in playwright.config.ts.
+const DEFAULT_BASE_URL = 'http://localhost:3001'
+const PREFLIGHT_TIMEOUT_MS = 3_000
+
+async function isBaseUrlReachable(baseUrl: string): Promise<boolean> {
+  const controller = new AbortController()
+  const timeout = setTimeout(() => controller.abort(), PREFLIGHT_TIMEOUT_MS)
+  try {
+    await fetch(baseUrl, { signal: controller.signal })
+    return true
+  } catch {
+    return false
+  } finally {
+    clearTimeout(timeout)
+  }
+}
+
+function git(args: string[], cwd: string): string {
+  const result = spawnSync('git', args, {
+    cwd,
+    encoding: 'utf8',
+    env: process.env,
+  })
+  if (result.status !== 0) {
+    const detail = (result.stderr || result.stdout || '').trim()
+    throw new Error(`git ${args.join(' ')} failed${detail ? `: ${detail}` : ''}`)
+  }
+  return result.stdout
+}
+
+function repoRootFromCwd(): string {
+  return git(['rev-parse', '--show-toplevel'], E2E_DOCS_ROOT).trim()
+}
+
+function collectChangedFiles(repoRoot: string, baseRef: string): string[] {
+  const ranges: string[][] = [
+    // Commits on this branch since diverging from the base
+    ['diff', '--name-only', '--diff-filter=ACMR', `${baseRef}...HEAD`],
+    // Unstaged working tree
+    ['diff', '--name-only', '--diff-filter=ACMR'],
+    // Staged working tree
+    ['diff', '--name-only', '--diff-filter=ACMR', '--cached'],
+  ]
+
+  const files = new Set<string>()
+  for (const args of ranges) {
+    try {
+      for (const file of parseChangedFilesList(git([...args], repoRoot))) {
+        files.add(file)
+      }
+    } catch (error) {
+      if (args.includes(`${baseRef}...HEAD`)) {
+        throw error
+      }
+      // Working-tree diffs can be empty / fail in odd git states; ignore those.
+    }
+  }
+  return [...files].sort()
+}
+
+async function resolveAllPagePaths(): Promise<string[]> {
+  const repoRoot = repoRootFromCwd()
+  const pages = await resolveAllDocsPages(repoRoot)
+  console.error(`Resolved all ${pages.length} in-scope docs page(s) (guides + troubleshooting).`)
+  return pages
+}
+
+async function resolvePagePaths(): Promise<string[] | null> {
+  const existing = process.env.DOCS_E2E_PAGE_PATHS?.trim()
+  if (existing) {
+    return existing
+      .split(/[\n,]/)
+      .map((p) => p.trim())
+      .filter(Boolean)
+  }
+
+  const baseRef = process.env.DOCS_E2E_BASE_REF?.trim() || 'origin/master'
+  const repoRoot = repoRootFromCwd()
+  const changedFiles = collectChangedFiles(repoRoot, baseRef)
+  const result = await resolveDocsScope({ changedFiles, repoRoot })
+
+  if (result.skip) {
+    console.error(
+      `No in-scope docs pages changed vs ${baseRef} (including working tree). Skipping Playwright.`
+    )
+    return null
+  }
+
+  console.error(`Resolved ${result.pages.length} docs page(s) from changes vs ${baseRef}:`)
+  for (const page of result.pages) {
+    console.error(`  ${page}`)
+  }
+  return result.pages
+}
+
+async function main() {
+  const rawArgs = process.argv.slice(2)
+  const runAll = rawArgs.includes('--all')
+  const withoutAll = rawArgs.filter((arg) => arg !== '--all')
+  // pnpm's `--` separator (from `pnpm run ... -- --list`) can land at index 0
+  // or, once `--all` is stripped, wherever `--all` used to precede it.
+  const playwrightArgs = withoutAll[0] === '--' ? withoutAll.slice(1) : withoutAll
+
+  const pages = runAll ? await resolveAllPagePaths() : await resolvePagePaths()
+  if (pages === null) {
+    process.exit(0)
+  }
+
+  // playwright.config.ts sets a global maxFailures: 3, which would otherwise
+  // abort an exhaustive --all run after just 3 failing pages out of hundreds.
+  // -x is Playwright's shorthand for --max-failures=1.
+  const hasMaxFailuresArg = playwrightArgs.some(
+    (arg) => arg === '-x' || arg.startsWith('--max-failures')
+  )
+  const finalPlaywrightArgs =
+    runAll && !hasMaxFailuresArg ? [...playwrightArgs, '--max-failures=0'] : playwrightArgs
+
+  const baseUrl = process.env.PLAYWRIGHT_BASE_URL?.trim() || DEFAULT_BASE_URL
+  if (!(await isBaseUrlReachable(baseUrl))) {
+    console.error(`No docs server responding at ${baseUrl}.`)
+    if (!process.env.PLAYWRIGHT_BASE_URL) {
+      console.error('Start it with `pnpm dev:docs`, or point at a deployed site:')
+      console.error('  PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs')
+    } else {
+      console.error('Check that the URL is correct and reachable.')
+    }
+    process.exit(1)
+  }
+
+  const env = {
+    ...process.env,
+    DOCS_E2E_PAGE_PATHS: pages.join(','),
+  }
+
+  const child = spawn('pnpm', ['exec', 'playwright', 'test', ...finalPlaywrightArgs], {
+    cwd: E2E_DOCS_ROOT,
+    env,
+    stdio: 'inherit',
+    shell: process.platform === 'win32',
+  })
+
+  child.on('exit', (code, signal) => {
+    if (signal) {
+      process.kill(process.pid, signal)
+      return
+    }
+    process.exit(code ?? 1)
+  })
+}
+
+main().catch((error) => {
+  console.error(error instanceof Error ? error.message : error)
+  process.exit(1)
+})

e2e/docs/tsconfig.json

@@ -4,6 +4,9 @@
     "module": "nodenext",
     "skipLibCheck": true,
     "esModuleInterop": true,
-    "strict": true
+    "strict": true,
+    "noEmit": true,
+    "allowImportingTsExtensions": true,
+    "types": ["node"]
   }
 }

e2e/docs/utils/docs-links.ts

@@ -1,18 +1,41 @@
 import type { Page } from '@playwright/test'
 
-const ARTICLE_SELECTOR = '#sb-docs-guide-main-article'
+export const GUIDE_ARTICLE_SELECTOR = '#sb-docs-guide-main-article'
+export const TROUBLESHOOTING_ARTICLE_SELECTOR = 'article.prose'
 const DOCS_PATH_PREFIX = '/docs'
+const TROUBLESHOOTING_PATH_PREFIX = '/docs/guides/troubleshooting/'
 
 /**
- * Collect unique docs-owned links from the main guide article.
+ * Pick the main article selector for a docs page path.
+ * Guides use a stable id; troubleshooting entries use a plain prose article.
+ */
+export function articleSelectorForPagePath(pagePath: string): string {
+  const pathname = pagePath.startsWith('http') ? new URL(pagePath).pathname : pagePath
+
+  if (
+    pathname === TROUBLESHOOTING_PATH_PREFIX.slice(0, -1) ||
+    pathname.startsWith(TROUBLESHOOTING_PATH_PREFIX)
+  ) {
+    return TROUBLESHOOTING_ARTICLE_SELECTOR
+  }
+
+  return GUIDE_ARTICLE_SELECTOR
+}
+
+/**
+ * Collect unique docs-owned links from the main article.
  *
  * Cross-app paths such as `/ui` and `/dashboard` are excluded because the
  * docs preview does not own those routes.
  */
-export async function collectDocsOwnedLinks(page: Page, baseURL: string): Promise<string[]> {
+export async function collectDocsOwnedLinks(
+  page: Page,
+  baseURL: string,
+  articleSelector: string = GUIDE_ARTICLE_SELECTOR
+): Promise<string[]> {
   const origin = new URL(baseURL).origin
   const hrefs = await page
-    .locator(`${ARTICLE_SELECTOR} a[href]`)
+    .locator(`${articleSelector} a[href]`)
     .evaluateAll((anchors) =>
       anchors.map((anchor) => (anchor as HTMLAnchorElement).getAttribute('href') ?? '')
     )
@@ -40,3 +63,27 @@ export async function collectDocsOwnedLinks(page: Page, baseURL: string): Promis
 
   return [...links].sort()
 }
+
+/**
+ * Playwright's headless Chromium reports a `HeadlessChrome` UA string, which
+ * Vercel's bot protection blocks on some routes (notably /docs/reference/*)
+ * even though the same page loads fine for a real browser. Stripping
+ * `Headless` avoids that false positive when checking links out-of-band via
+ * page.request rather than an actual navigation.
+ */
+export async function browserLikeUserAgent(page: Page): Promise<string> {
+  const userAgent = await page.evaluate(() => navigator.userAgent)
+  return userAgent.replace('HeadlessChrome', 'Chrome')
+}
+
+/**
+ * Parse DOCS_E2E_PAGE_PATHS (comma- or newline-separated /docs/... paths).
+ */
+export function parseDocsE2EPagePaths(raw: string | undefined): string[] {
+  if (!raw?.trim()) return []
+  return raw
+    .split(/[\n,]/)
+    .map((path) => path.trim())
+    .filter(Boolean)
+    .map((path) => (path.startsWith('/') ? path : `/${path}`))
+}

e2e/docs/utils/resolve-docs-scope.ts

@@ -0,0 +1,353 @@
+import { readdir, readFile } from 'node:fs/promises'
+import { basename, join, relative, sep } from 'node:path'
+
+/**
+ * Federated guide section prefixes — mirrors
+ * apps/docs/scripts/federated-content/sources/*.ts. Checked automatically
+ * against those files by assertFederatedSectionsInSync below.
+ */
+export const FEDERATED_SECTIONS = [
+  'graphql',
+  'database/extensions/wrappers',
+  'ai/python',
+  'deployment/terraform',
+  'deployment/ci',
+] as const
+
+export const MAX_SCOPED_PAGES = 20
+
+const GUIDES_PREFIX = 'apps/docs/content/guides/'
+const TROUBLESHOOTING_PREFIX = 'apps/docs/content/troubleshooting/'
+const PARTIALS_PREFIX = 'apps/docs/content/_partials/'
+const DOCS_GUIDES_URL_PREFIX = '/docs/guides/'
+const DOCS_TROUBLESHOOTING_URL_PREFIX = '/docs/guides/troubleshooting/'
+const FEDERATED_CONTENT_SOURCES_DIR = 'apps/docs/scripts/federated-content/sources'
+
+const PARTIAL_PATH_RE = /<\$Partial\b[\s\S]*?\bpath\s*=\s*"([^"]+)"[\s\S]*?\/?>/g
+const SOURCE_SECTION_RE = /\bsection:\s*'([^']+)'/g
+
+/**
+ * Compares FEDERATED_SECTIONS against the `section:` field declared in each
+ * apps/docs/scripts/federated-content/sources/*.ts file, so a section added
+ * or removed there can't silently drift from what this suite treats as
+ * out-of-scope.
+ */
+async function assertFederatedSectionsInSync(repoRoot: string): Promise<void> {
+  const sourcesDir = join(repoRoot, FEDERATED_CONTENT_SOURCES_DIR)
+  const sourceFiles = (await readdir(sourcesDir)).filter((file) => file.endsWith('.ts'))
+
+  const actualSections = new Set<string>()
+  for (const file of sourceFiles) {
+    const source = await readFile(join(sourcesDir, file), 'utf8')
+    const matches = [...source.matchAll(SOURCE_SECTION_RE)].map((match) => match[1])
+    const distinctMatches = new Set(matches)
+
+    if (distinctMatches.size > 1) {
+      throw new Error(
+        `${FEDERATED_CONTENT_SOURCES_DIR}/${file} declares multiple distinct 'section:' ` +
+          `values (${[...distinctMatches].join(', ')}); expected exactly one per source file.`
+      )
+    }
+    if (distinctMatches.size === 1) actualSections.add(matches[0])
+  }
+
+  const expectedSections = new Set<string>(FEDERATED_SECTIONS)
+  const missing = [...actualSections].filter((section) => !expectedSections.has(section))
+  const stale = [...expectedSections].filter((section) => !actualSections.has(section))
+
+  if (missing.length > 0 || stale.length > 0) {
+    const details = [
+      missing.length > 0 ? `missing from FEDERATED_SECTIONS: ${missing.join(', ')}` : null,
+      stale.length > 0 ? `no longer a federated-content source: ${stale.join(', ')}` : null,
+    ]
+      .filter(Boolean)
+      .join('; ')
+    throw new Error(
+      `FEDERATED_SECTIONS in e2e/docs/utils/resolve-docs-scope.ts is out of sync with ` +
+        `${FEDERATED_CONTENT_SOURCES_DIR}/*.ts (${details}). Update FEDERATED_SECTIONS to match.`
+    )
+  }
+}
+
+export type ResolveDocsScopeOptions = {
+  /** Repo-root-relative changed file paths */
+  changedFiles: string[]
+  /** Absolute path to the monorepo root */
+  repoRoot: string
+  /** Max pages to test before the rest are truncated (default MAX_SCOPED_PAGES) */
+  maxPages?: number
+}
+
+export type ResolveDocsScopeResult = {
+  pages: string[]
+  skip: boolean
+}
+
+function normalizeRepoPath(filePath: string): string {
+  return filePath.replaceAll('\\', '/')
+}
+
+function isFederatedGuideSlug(slug: string): boolean {
+  return FEDERATED_SECTIONS.some((section) => slug === section || slug.startsWith(`${section}/`))
+}
+
+function isHiddenMdx(filePath: string): boolean {
+  return basename(filePath).startsWith('_')
+}
+
+/**
+ * Map a changed content file to a docs URL, or null if not a testable page.
+ */
+export function changedFileToPagePath(filePath: string): string | null {
+  const normalized = normalizeRepoPath(filePath)
+
+  if (normalized.startsWith(GUIDES_PREFIX) && normalized.endsWith('.mdx')) {
+    if (isHiddenMdx(normalized)) return null
+    const slug = normalized.slice(GUIDES_PREFIX.length, -'.mdx'.length)
+    if (!slug || isFederatedGuideSlug(slug)) return null
+    return `${DOCS_GUIDES_URL_PREFIX}${slug}`
+  }
+
+  if (normalized.startsWith(TROUBLESHOOTING_PREFIX) && normalized.endsWith('.mdx')) {
+    if (isHiddenMdx(normalized)) return null
+    const slug = basename(normalized, '.mdx')
+    return `${DOCS_TROUBLESHOOTING_URL_PREFIX}${slug}`
+  }
+
+  return null
+}
+
+function partialRelPathFromChangedFile(filePath: string): string | null {
+  const normalized = normalizeRepoPath(filePath)
+  if (!normalized.startsWith(PARTIALS_PREFIX) || !normalized.endsWith('.mdx')) {
+    return null
+  }
+  if (normalized.includes('/_fixtures/')) return null
+  const rel = normalized.slice(PARTIALS_PREFIX.length)
+  if (!rel || basename(rel).startsWith('_')) return null
+  return rel
+}
+
+function normalizePartialRef(pathAttr: string): string | null {
+  // $Partial paths are relative to content/_partials (see Partial.ts).
+  // Leading-slash example-code paths are not real partials.
+  if (!pathAttr || pathAttr.startsWith('/') || pathAttr.startsWith('http')) {
+    return null
+  }
+  if (!pathAttr.endsWith('.md') && !pathAttr.endsWith('.mdx')) {
+    return null
+  }
+  return normalizeRepoPath(pathAttr).replace(/^\.\//, '')
+}
+
+function extractPartialRefs(source: string): string[] {
+  const refs: string[] = []
+  PARTIAL_PATH_RE.lastIndex = 0
+  let match: RegExpExecArray | null
+  while ((match = PARTIAL_PATH_RE.exec(source)) !== null) {
+    const normalized = normalizePartialRef(match[1])
+    if (normalized) refs.push(normalized)
+  }
+  return refs
+}
+
+async function walkMdxFiles(dir: string): Promise<string[]> {
+  const entries = await readdir(dir, { withFileTypes: true })
+  const files: string[] = []
+
+  for (const entry of entries) {
+    const fullPath = join(dir, entry.name)
+    if (entry.isDirectory()) {
+      if (entry.name.startsWith('_')) continue
+      files.push(...(await walkMdxFiles(fullPath)))
+      continue
+    }
+    if (!entry.isFile()) continue
+    if (!entry.name.endsWith('.mdx')) continue
+    if (entry.name.startsWith('_')) continue
+    files.push(fullPath)
+  }
+
+  return files
+}
+
+type PartialIndex = {
+  /** partial rel path → set of partial rel paths that include it */
+  includedByPartials: Map<string, Set<string>>
+  /** page URL → set of direct partial rel paths */
+  pagePartials: Map<string, Set<string>>
+}
+
+async function buildPartialIndex(repoRoot: string): Promise<PartialIndex> {
+  const includedByPartials = new Map<string, Set<string>>()
+  const pagePartials = new Map<string, Set<string>>()
+
+  const partialsDir = join(repoRoot, 'apps/docs/content/_partials')
+  const guidesDir = join(repoRoot, 'apps/docs/content/guides')
+  const troubleshootingDir = join(repoRoot, 'apps/docs/content/troubleshooting')
+
+  const [partialFiles, guideFiles, troubleshootingFiles] = await Promise.all([
+    walkMdxFiles(partialsDir).catch(() => [] as string[]),
+    walkMdxFiles(guidesDir).catch(() => [] as string[]),
+    walkMdxFiles(troubleshootingDir).catch(() => [] as string[]),
+  ])
+
+  for (const file of partialFiles) {
+    const rel = normalizeRepoPath(relative(partialsDir, file))
+    const source = await readFile(file, 'utf8')
+    for (const ref of extractPartialRefs(source)) {
+      let parents = includedByPartials.get(ref)
+      if (!parents) {
+        parents = new Set()
+        includedByPartials.set(ref, parents)
+      }
+      parents.add(rel)
+    }
+  }
+
+  for (const file of guideFiles) {
+    const relFromGuides = normalizeRepoPath(relative(guidesDir, file)).replace(/\.mdx$/, '')
+    if (isFederatedGuideSlug(relFromGuides)) continue
+    const pagePath = `${DOCS_GUIDES_URL_PREFIX}${relFromGuides}`
+    const source = await readFile(file, 'utf8')
+    const refs = extractPartialRefs(source)
+    if (refs.length > 0) pagePartials.set(pagePath, new Set(refs))
+  }
+
+  for (const file of troubleshootingFiles) {
+    const slug = basename(file, '.mdx')
+    const pagePath = `${DOCS_TROUBLESHOOTING_URL_PREFIX}${slug}`
+    const source = await readFile(file, 'utf8')
+    const refs = extractPartialRefs(source)
+    if (refs.length > 0) pagePartials.set(pagePath, new Set(refs))
+  }
+
+  return { includedByPartials, pagePartials }
+}
+
+/**
+ * Expand a changed partial to all partials that transitively include it
+ * (including itself).
+ */
+function expandPartialClosure(
+  seed: string,
+  includedByPartials: Map<string, Set<string>>
+): Set<string> {
+  const result = new Set<string>([seed])
+  const queue = [seed]
+
+  while (queue.length > 0) {
+    const current = queue.pop()!
+    const parents = includedByPartials.get(current)
+    if (!parents) continue
+    for (const parent of parents) {
+      if (result.has(parent)) continue
+      result.add(parent)
+      queue.push(parent)
+    }
+  }
+
+  return result
+}
+
+function pagesUsingPartials(
+  targetPartials: Set<string>,
+  pagePartials: Map<string, Set<string>>
+): string[] {
+  const pages: string[] = []
+  for (const [page, refs] of pagePartials) {
+    for (const ref of refs) {
+      if (targetPartials.has(ref)) {
+        pages.push(page)
+        break
+      }
+    }
+  }
+  return pages
+}
+
+/**
+ * Resolve which docs pages to E2E-test from a list of changed repo files.
+ * Truncates to maxPages (sorted) so a widely shared partial can't blow up
+ * runtime; use resolveAllDocsPages / `pnpm e2e:docs:all` to cover everything.
+ */
+export async function resolveDocsScope(
+  options: ResolveDocsScopeOptions
+): Promise<ResolveDocsScopeResult> {
+  await assertFederatedSectionsInSync(options.repoRoot)
+
+  const maxPages = options.maxPages ?? MAX_SCOPED_PAGES
+  const pages = new Set<string>()
+  const changedPartials: string[] = []
+
+  for (const file of options.changedFiles) {
+    const page = changedFileToPagePath(file)
+    if (page) {
+      pages.add(page)
+      continue
+    }
+    const partial = partialRelPathFromChangedFile(file)
+    if (partial) changedPartials.push(partial)
+  }
+
+  if (changedPartials.length > 0) {
+    const index = await buildPartialIndex(options.repoRoot)
+    const targets = new Set<string>()
+    for (const partial of changedPartials) {
+      for (const expanded of expandPartialClosure(partial, index.includedByPartials)) {
+        targets.add(expanded)
+      }
+    }
+    for (const page of pagesUsingPartials(targets, index.pagePartials)) {
+      pages.add(page)
+    }
+  }
+
+  const sorted = [...pages].sort().slice(0, maxPages)
+
+  return {
+    pages: sorted,
+    skip: sorted.length === 0,
+  }
+}
+
+/**
+ * List every in-scope docs page — all guides (excluding federated sections)
+ * and all troubleshooting entries — regardless of what changed. Used for
+ * full-suite runs rather than the default changed-files scope.
+ */
+export async function resolveAllDocsPages(repoRoot: string): Promise<string[]> {
+  await assertFederatedSectionsInSync(repoRoot)
+
+  const guidesDir = join(repoRoot, 'apps/docs/content/guides')
+  const troubleshootingDir = join(repoRoot, 'apps/docs/content/troubleshooting')
+
+  const [guideFiles, troubleshootingFiles] = await Promise.all([
+    walkMdxFiles(guidesDir).catch(() => [] as string[]),
+    walkMdxFiles(troubleshootingDir).catch(() => [] as string[]),
+  ])
+
+  const pages = new Set<string>()
+
+  for (const file of guideFiles) {
+    const relFromGuides = normalizeRepoPath(relative(guidesDir, file)).replace(/\.mdx$/, '')
+    if (isFederatedGuideSlug(relFromGuides)) continue
+    pages.add(`${DOCS_GUIDES_URL_PREFIX}${relFromGuides}`)
+  }
+
+  for (const file of troubleshootingFiles) {
+    const slug = basename(file, '.mdx')
+    pages.add(`${DOCS_TROUBLESHOOTING_URL_PREFIX}${slug}`)
+  }
+
+  return [...pages].sort()
+}
+
+/** Parse changed-file list from stdin or newline/comma-separated string. */
+export function parseChangedFilesList(input: string): string[] {
+  return input
+    .split(/[\n,]/)
+    .map((line) => line.trim())
+    .filter(Boolean)
+    .map((line) => normalizeRepoPath(line.split(sep).join('/')))
+}