mirror of
https://github.com/supabase/supabase.git
synced 2026-10-10 03:45:06 +03:00
Pamela ChiaandAlaister Young
20290c71bd
## Summary Since the guides UA-redirect shipped (GROWTH-811), named LLM bots requesting `/docs/guides/*` get rewritten to the markdown handler, which returns a 404 when no `.md` file exists. About 90K of those 404s per day land on real pages that serve HTML 200 fine: the bot gets nothing on a page that works. The root cause is that the docs middleware hardcoded `hasMarkdownVariant: true` for every guide path, so it never checked whether a `.md` actually existed. I fixed it in two layers: 1. A build-time slug manifest makes `hasMarkdownVariant` truthful. Guide pages with no `.md` now fall through to HTML 200 instead of a 404. This is content-source-agnostic and future-proof: a new content source can never silently regress to a 404. 2. A second generator pass emits real markdown for the troubleshooting collection (the largest source, ~70% of the 404 volume), so those bots get clean markdown rather than just HTML. ## Changes - Add a shared `markdown-sources` module: a single source of truth for which slugs get a `.md` (guides + troubleshooting), so the generator output and the manifest cannot drift. - Generate markdown for the troubleshooting collection (196 pages, TOML frontmatter parsed via `smol-toml`), written under `public/markdown/guides/troubleshooting/`. - Emit a build-time slug manifest (a gitignored generated `.ts` module, regenerated in `prebuild`, `predev`, and `pretypecheck`, mirroring the existing `__generated__/graphql.ts` lifecycle). - Gate the middleware's `hasMarkdownVariant` on the manifest: serve HTML 200 instead of a 404 for guide paths with no markdown variant. This PR intentionally does not generate markdown for the ai-prompts, YAML config, and externally-fetched (splinter) sources. The HTML fallback covers them now; generating their markdown is follow-up work. ## Testing Local verification (deterministic, against the real manifest and the real negotiation function): - Manifest invariant holds: 744 manifest slugs equal 744 generated `.md` files. - Generator emits 196 troubleshooting files with zero warnings, frontmatter stripped, no leaked delimiters. - Negotiation decision matrix, 6/6: covered slug + bot UA to markdown; uncovered real page + bot UA to pass (HTML 200); nonexistent + bot UA to pass; browser to HTML; covered + `.md` suffix to markdown; uncovered + `.md` suffix to pass. Verified on the Vercel preview deploy: - [x] `User-Agent: ChatGPT-User` on a troubleshooting page returns `200 text/markdown` (real markdown body, frontmatter stripped). - [x] `User-Agent: ChatGPT-User` on an uncovered real page (`ai-tools/ai-prompts/code-format-sql`) returns `200 text/html` (was 404). - [x] Browser request to the same uncovered page returns `200 text/html` (unchanged for humans). - [x] `User-Agent: ChatGPT-User` on a covered standard guide returns `200 text/markdown` (no regression). - [x] `User-Agent: ChatGPT-User` on a nonexistent guide URL returns `404` (correct). Known limitation: an explicit `.md`-suffix request on an uncovered page still 404s by design (an explicit markdown request for a page that has no markdown). The ~90K/day volume is plain-URL UA-based, so it is unaffected. Post-deploy, I will re-run the request-grain 404 reclassification in the GROWTH-915 BQ workspace to confirm fixable guide markdown 404s drop to near zero. ## Linear - fixes GROWTH-946 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added generated markdown slug tracking for docs guides, improving markdown availability detection. * Added automated manifest generation and validation during docs build and CI workflows. * **Bug Fixes** * Improved guide markdown negotiation so only supported guide slugs are treated as having a markdown variant. * Standardized markdown source handling for guides and troubleshooting pages. * **Tests** * Added coverage for guide and troubleshooting slug generation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com>
56 lines
1.5 KiB
YAML
56 lines
1.5 KiB
YAML
name: Docs Markdown Manifest Sync
|
|
|
|
on:
|
|
pull_request:
|
|
branches: ['master']
|
|
paths:
|
|
- 'apps/docs/content/guides/**/*.mdx'
|
|
- 'apps/docs/content/troubleshooting/**/*.mdx'
|
|
- 'apps/docs/internals/generate-markdown-manifest.ts'
|
|
- 'apps/docs/internals/markdown-sources.ts'
|
|
- 'apps/docs/lib/markdown-manifest.ts'
|
|
|
|
# Cancel old builds on new commit for same workflow + branch/PR
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
env:
|
|
CI: true
|
|
|
|
jobs:
|
|
check:
|
|
name: Check apps/docs/lib/markdown-manifest.ts is in sync
|
|
runs-on: blacksmith-4vcpu-ubuntu-2404
|
|
|
|
steps:
|
|
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
|
|
with:
|
|
persist-credentials: false
|
|
sparse-checkout: |
|
|
apps/docs
|
|
examples
|
|
packages
|
|
supabase
|
|
patches
|
|
|
|
- uses: pnpm/action-setup@41ff72655975bd51cab0327fa583b6e92b6d3061 # v4.2.0
|
|
name: Install pnpm
|
|
with:
|
|
run_install: false
|
|
|
|
- name: Use Node.js
|
|
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
|
|
with:
|
|
node-version-file: '.nvmrc'
|
|
cache: 'pnpm'
|
|
|
|
- name: Install deps
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Verify markdown manifest is up to date
|
|
run: pnpm --filter docs check:markdown-manifest
|