fix(docs ci): stop Docs E2E blocking pull requests it shouldn't (#48726)

Supersedes #48725, which GitHub closed when its head branch was renamed.
Same commits, same diff.

Fixes
[DOCS-1270](https://linear.app/supabase/issue/DOCS-1270/fail-the-e2e-pipeline-if-the-docs-preview-never-loads).

`Docs E2E` is a required check on `master`, so anything that turns it
red blocks a merge. It had three ways of going red that had nothing to
do with whether the author's docs were correct.

## Problem

**1. Every troubleshooting page could fail, with nothing actionable.**
Troubleshooting entries were selected by `article.prose`. That class is
not unique — `apps/docs/app/not-found.tsx` renders `<article
className="prose …">` too — and nothing guaranteed it matched the
entry's article at all. When it missed, the link test failed with `Page
article should be present` and the a11y test failed inside axe with `No
elements found for include in page Context` plus a stack trace. Neither
tells the author what to do.

This is what DOCS-1270 actually was. The ticket describes tests running
"against a preview build that was never created", but the [failing
run](https://github.com/supabase/supabase/actions/runs/30949924515/job/92132543658)
for #48719 shows the preview resolved fine and `response.ok()` passed —
it broke at the article assertion. **Blocked:** anyone adding or editing
a troubleshooting entry.

**2. Fork pull requests failed for being forks.** Fork runs get no
`VERCEL_TOKEN`, so no preview URL resolves, and the base-URL step fell
back to `https://supabase.com`. The page paths under test can include
pages the pull request *adds*, which do not exist on production, so they
404. **Blocked:** every external contributor adding a docs page,
unconditionally, with no action available to them.

**3. A Vercel problem failed the docs check.**
`waitForVercelDocsPreview.js` throws when Vercel reports a failed
deployment, omits a `target_url`, or does not post a status within 900s.
The step had no `continue-on-error`, so any of those turned `Docs E2E`
red. **Blocked:** any author whose pull request coincided with a Vercel
incident. This is live right now — two Vercel checks on this very pull
request are failing with "unable to fetch required git information", a
git-integration auth error that happens before any build runs.

## Solution

**1. Select on a stable, purpose-named attribute.** Add
`id="sb-docs-troubleshooting-main-article"` on the troubleshooting
article, mirroring `#sb-docs-guide-main-article` on guides, and select
on that instead of the class. Per review feedback, a plain id doesn't
say it's a test hook, so both articles also get `data-testid` with the
same value — matching the convention `apps/studio` already uses with
Playwright's `getByTestId` — and the e2e selectors target that attribute
instead. Guides keep their `id` — `GuidesMdx.client.tsx` and
`GuidesSidebar.tsx` both query it directly for the table of contents and
the "copy article" fallback — and gain `data-testid` alongside it.

**2 and 3. Resolve a preview or skip — never substitute production,
never fail on Vercel.** The production fallback is gone.
`continue-on-error: true` on the preview wait means a Vercel failure
resolves no URL instead of failing the job, which lands in the same path
as a fork: `should_test=false`, so Playwright is skipped and the check
passes. Both cases emit a `::warning::` and a job summary with the exact
`gh workflow run` command to test the preview by hand, and manual runs
against a non-production base URL now send the protection bypass so that
command actually works.

Skipping does not let a broken preview through: `Vercel – docs` is
itself a required check on `master`, so a genuine preview failure still
blocks the merge — via the check that describes the real problem.


## Manual test

**1. The selector matches the markup, and it needs this pull request's
preview.** `data-testid` isn't deployed anywhere yet — not on
production, not on any other branch — so this is the one claim in this
PR that production cannot confirm. Verified directly against this
branch's own Vercel preview:

```bash
curl -s https://docs-git-docs-e2e-stop-false-blocks-supabase.vercel.app/docs/guides/database/overview \
  | grep -o 'data-testid="[^"]*"'
curl -s https://docs-git-docs-e2e-stop-false-blocks-supabase.vercel.app/docs/guides/troubleshooting/42501--permission-denied-for-table-httprequestqueue-KnozmQ \
  | grep -o 'data-testid="[^"]*"'
```

Expect `data-testid="sb-docs-guide-main-article"` and
`data-testid="sb-docs-troubleshooting-main-article"` respectively. Then
run the suite against that same preview — expect all page/link/a11y
checks to pass:

```bash
PLAYWRIGHT_BASE_URL=https://docs-git-docs-e2e-stop-false-blocks-supabase.vercel.app \
DOCS_E2E_PAGE_PATHS=/docs/guides/database/overview,/docs/guides/troubleshooting/42501--permission-denied-for-table-httprequestqueue-KnozmQ \
pnpm -C e2e/docs exec playwright test --reporter=list
```

Running the same command with `PLAYWRIGHT_BASE_URL=https://supabase.com`
fails both pages right now — expected until this merges, not a
regression. Once merged, exercise it through the real pipeline:

```bash
gh workflow run docs-e2e.yml --ref docs-e2e/stop-false-blocks \
  -f base_url=<preview-url> \
  -f page_paths=/docs/guides/troubleshooting/42501--permission-denied-for-table-httprequestqueue-KnozmQ
```

**2. No preview means skip, not a run against production.** Exercise the
base-URL step's three paths from the repository root:

```bash
export GITHUB_OUTPUT=$(mktemp) GITHUB_STEP_SUMMARY=$(mktemp) PAGE_PATHS=/docs/guides/a
script=$(python3 -c "import yaml;print([s for s in yaml.safe_load(open('.github/workflows/docs-e2e.yml'))['jobs']['e2e']['steps'] if s.get('name')=='Resolve base URL'][0]['run'])")
for c in "workflow_dispatch|https://supabase.com|" "pull_request||https://docs-abc.vercel.app" "pull_request||"; do
  IFS='|' read -r ev url dep <<< "$c"
  : > "$GITHUB_OUTPUT"
  EVENT_NAME="$ev" BASE_URL_INPUT="${url:-https://supabase.com}" DEPLOYMENT_URL="$dep" bash -c "$script" >/dev/null 2>&1
  echo "$ev deployment=[${dep:-none}] -> $(tr '\n' ' ' < "$GITHUB_OUTPUT")"
done
tail -4 "$GITHUB_STEP_SUMMARY"
```

Expected:

```
workflow_dispatch deployment=[none] -> url=https://supabase.com use_bypass=false should_test=true
pull_request deployment=[https://docs-abc.vercel.app] -> url=https://docs-abc.vercel.app use_bypass=true should_test=true
pull_request deployment=[none] -> url= use_bypass=false should_test=false
```

followed by a runnable `gh workflow run docs-e2e.yml` command in the job
summary. The third line covers both the fork case and the Vercel-failure
case: no base URL, no test, no block.

**3. A Vercel failure no longer fails the job.** `continue-on-error:
true` on the wait step is what routes a throw into that third line:

```bash
python3 -c "
import yaml
s=[x for x in yaml.safe_load(open('.github/workflows/docs-e2e.yml'))['jobs']['e2e']['steps'] if x.get('name')=='Wait for Vercel docs preview'][0]
print('continue-on-error:', s.get('continue-on-error'))
for n in ('Install dependencies','Install Playwright Chromium','Run docs E2E'):
    print(n, '->', [x for x in yaml.safe_load(open('.github/workflows/docs-e2e.yml'))['jobs']['e2e']['steps'] if x.get('name')==n][0]['if'])
"
```

Expect `continue-on-error: True` and all three run steps gated on
`steps.base-url.outputs.should_test == 'true'`.

**Note on this pull request's own check.** The scope resolver only maps
`apps/docs/content/**` to pages, and this pull request changes none, so
`Docs E2E` resolves zero pages and skips — which is correct, and why the
dispatch above is the real test.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved documentation preview checks so unavailable or delayed
previews no longer cause unnecessary workflow failures.
* Added clearer handling for manual documentation checks and missing
preview deployments.

* **Tests**
* Improved end-to-end documentation testing reliability across preview
and production environments.
* Added stable targeting for the troubleshooting article to reduce test
failures caused by page structure changes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

.github/workflows/docs-e2e.yml

@@ -139,9 +139,12 @@ jobs:
       # Poll the "Vercel – docs" commit status instead — Vercel keeps posting
       # those — then resolve the deployment it points to via Vercel's own API
       # to get the actual preview URL. See scripts/waitForVercelDocsPreview.js.
+      # A Vercel failure or timeout is not the author's problem, and the required
+      # "Vercel – docs" check already reports it. Resolve no URL and skip below.
       - name: Wait for Vercel docs preview
         if: steps.scope.outputs.skip == 'false' && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && steps.changes.outputs.docs_app == 'true'
         id: deployment
+        continue-on-error: true
         run: node scripts/waitForVercelDocsPreview.js
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -156,30 +159,61 @@ jobs:
           EVENT_NAME: ${{ github.event_name }}
           BASE_URL_INPUT: ${{ inputs.base_url }}
           DEPLOYMENT_URL: ${{ steps.deployment.outputs.deployment-url }}
-          DOCS_APP_CHANGED: ${{ steps.changes.outputs.docs_app }}
+          PAGE_PATHS: ${{ steps.scope.outputs.paths }}
         run: |
+          set -euo pipefail
+
           if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
             printf 'url=%s\n' "$BASE_URL_INPUT" >> "$GITHUB_OUTPUT"
-            echo "use_bypass=false" >> "$GITHUB_OUTPUT"
-          elif [ "$DOCS_APP_CHANGED" = "true" ] && [ -n "$DEPLOYMENT_URL" ]; then
-            printf 'url=%s\n' "$DEPLOYMENT_URL" >> "$GITHUB_OUTPUT"
-            echo "use_bypass=true" >> "$GITHUB_OUTPUT"
-          else
-            # Harness-only PRs have no docs preview; test against production.
-            echo "url=https://supabase.com" >> "$GITHUB_OUTPUT"
-            echo "use_bypass=false" >> "$GITHUB_OUTPUT"
+            # Non-production targets are previews, which may need the bypass.
+            if [ "$BASE_URL_INPUT" = "https://supabase.com" ]; then
+              echo "use_bypass=false" >> "$GITHUB_OUTPUT"
+            else
+              echo "use_bypass=true" >> "$GITHUB_OUTPUT"
+            fi
+            echo "should_test=true" >> "$GITHUB_OUTPUT"
+            exit 0
           fi
 
+          if [ -n "$DEPLOYMENT_URL" ]; then
+            printf 'url=%s\n' "$DEPLOYMENT_URL" >> "$GITHUB_OUTPUT"
+            echo "use_bypass=true" >> "$GITHUB_OUTPUT"
+            echo "should_test=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+
+          # Production is not a substitute: it lacks pages this pull request
+          # adds, so testing it fails a required check for a valid change.
+          echo "url=" >> "$GITHUB_OUTPUT"
+          echo "use_bypass=false" >> "$GITHUB_OUTPUT"
+          echo "should_test=false" >> "$GITHUB_OUTPUT"
+          echo "::warning::No Vercel docs preview URL for this pull request, so there is nothing serving its content to test. Skipping Playwright rather than testing production, which does not have pages this pull request adds."
+          {
+            echo "### Docs E2E skipped: no preview to test against"
+            echo
+            echo "Nothing is serving this pull request's content, and production is not a"
+            echo "substitute — pages it adds do not exist there yet."
+            echo
+            echo "Fork pull requests reach this path because they run without repository"
+            echo "secrets. A maintainer can run the suite against the preview manually:"
+            echo
+            echo '```'
+            echo "gh workflow run docs-e2e.yml \\"
+            echo "  -f base_url=<preview-url> \\"
+            echo "  -f page_paths=$PAGE_PATHS"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"
+
       - name: Install dependencies
-        if: steps.scope.outputs.skip == 'false'
+        if: steps.base-url.outputs.should_test == 'true'
         run: pnpm install --frozen-lockfile --filter=e2e-docs...
 
       - name: Install Playwright Chromium
-        if: steps.scope.outputs.skip == 'false'
+        if: steps.base-url.outputs.should_test == 'true'
         run: pnpm -C e2e/docs exec playwright install chromium --with-deps --only-shell
 
       - name: Run docs E2E
-        if: steps.scope.outputs.skip == 'false'
+        if: steps.base-url.outputs.should_test == 'true'
         working-directory: e2e/docs
         run: pnpm run e2e:docs
         env: