dima/supabase
1
0 Fork 0
mirror of https://github.com/supabase/supabase.git synced 2026-10-05 17:35:10 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
98cfe3307eb99436bbf10fa74fa2d495e40ce483
supabase/apps/docs/package.json
T
Pamela ChiaandAlaister Young 20290c71bd fix(docs): stop named-bot markdown 404s on guides (#47337) 
## Summary

Since the guides UA-redirect shipped (GROWTH-811), named LLM bots
requesting `/docs/guides/*` get rewritten to the markdown handler, which
returns a 404 when no `.md` file exists. About 90K of those 404s per day
land on real pages that serve HTML 200 fine: the bot gets nothing on a
page that works.

The root cause is that the docs middleware hardcoded
`hasMarkdownVariant: true` for every guide path, so it never checked
whether a `.md` actually existed. I fixed it in two layers:

1. A build-time slug manifest makes `hasMarkdownVariant` truthful. Guide
pages with no `.md` now fall through to HTML 200 instead of a 404. This
is content-source-agnostic and future-proof: a new content source can
never silently regress to a 404.
2. A second generator pass emits real markdown for the troubleshooting
collection (the largest source, ~70% of the 404 volume), so those bots
get clean markdown rather than just HTML.

## Changes

- Add a shared `markdown-sources` module: a single source of truth for
which slugs get a `.md` (guides + troubleshooting), so the generator
output and the manifest cannot drift.
- Generate markdown for the troubleshooting collection (196 pages, TOML
frontmatter parsed via `smol-toml`), written under
`public/markdown/guides/troubleshooting/`.
- Emit a build-time slug manifest (a gitignored generated `.ts` module,
regenerated in `prebuild`, `predev`, and `pretypecheck`, mirroring the
existing `__generated__/graphql.ts` lifecycle).
- Gate the middleware's `hasMarkdownVariant` on the manifest: serve HTML
200 instead of a 404 for guide paths with no markdown variant.

This PR intentionally does not generate markdown for the ai-prompts,
YAML config, and externally-fetched (splinter) sources. The HTML
fallback covers them now; generating their markdown is follow-up work.

## Testing

Local verification (deterministic, against the real manifest and the
real negotiation function):
- Manifest invariant holds: 744 manifest slugs equal 744 generated `.md`
files.
- Generator emits 196 troubleshooting files with zero warnings,
frontmatter stripped, no leaked delimiters.
- Negotiation decision matrix, 6/6: covered slug + bot UA to markdown;
uncovered real page + bot UA to pass (HTML 200); nonexistent + bot UA to
pass; browser to HTML; covered + `.md` suffix to markdown; uncovered +
`.md` suffix to pass.

Verified on the Vercel preview deploy:
- [x] `User-Agent: ChatGPT-User` on a troubleshooting page returns `200
text/markdown` (real markdown body, frontmatter stripped).
- [x] `User-Agent: ChatGPT-User` on an uncovered real page
(`ai-tools/ai-prompts/code-format-sql`) returns `200 text/html` (was
404).
- [x] Browser request to the same uncovered page returns `200 text/html`
(unchanged for humans).
- [x] `User-Agent: ChatGPT-User` on a covered standard guide returns
`200 text/markdown` (no regression).
- [x] `User-Agent: ChatGPT-User` on a nonexistent guide URL returns
`404` (correct).

Known limitation: an explicit `.md`-suffix request on an uncovered page
still 404s by design (an explicit markdown request for a page that has
no markdown). The ~90K/day volume is plain-URL UA-based, so it is
unaffected.

Post-deploy, I will re-run the request-grain 404 reclassification in the
GROWTH-915 BQ workspace to confirm fixable guide markdown 404s drop to
near zero.

## Linear
- fixes GROWTH-946


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added generated markdown slug tracking for docs guides, improving
markdown availability detection.
* Added automated manifest generation and validation during docs build
and CI workflows.

* **Bug Fixes**
* Improved guide markdown negotiation so only supported guide slugs are
treated as having a markdown variant.
* Standardized markdown source handling for guides and troubleshooting
pages.

* **Tests**
  * Added coverage for guide and troubleshooting slug generation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com>
2026-06-27 22:00:14 -07:00

186 lines
7.6 KiB
JSON
Raw Blame History

{
"name": "docs",
"version": "0.0.0",
"private": true,
"type": "module",
"scripts": {
"build": "next build",
"build:analyze": "ANALYZE=true next build",
"build:guides-markdown": "tsx ./internals/generate-guides-markdown.ts",
"prebuild:reference-markdown": "pnpm run codegen:references",
"build:reference-markdown": "tsx ./internals/generate-reference-markdown.ts",
"build:markdown": "pnpm build:guides-markdown && pnpm build:reference-markdown",
"build:markdown-manifest": "tsx ./internals/generate-markdown-manifest.ts",
"check:markdown-manifest": "tsx ./internals/generate-markdown-manifest.ts --check",
"build:gz-archive": "tsx ./internals/generate-gz-archive.ts",
"build:sitemap": "tsx ./internals/generate-sitemap.ts",
"clean": "rimraf .next .turbo node_modules features/docs/generated examples __generated__",
"codegen:examples": "shx cp -r ../../examples ./examples",
"codegen:graphql": "tsx --conditions=react-server ./scripts/graphqlSchema.ts && graphql-codegen --config codegen.ts",
"codegen:references:new:ensure": "test -f spec/reference/javascript/v2/supabase.json || (cd spec && make download.tsdoc.v2)",
"codegen:references:new": "pnpm run codegen:references:new:ensure && tsx scripts/build-reference-content.ts",
"codegen:references:legacy": "tsx features/docs/Reference.generated.script.ts",
"codegen:references": "pnpm codegen:references:legacy && pnpm codegen:references:new",
"codemod:frontmatter": "node ./scripts/codemod/mdx-meta.mjs && prettier --cache --write \"content/**/*.mdx\"",
"dev": "concurrently --kill-others \"next dev --port 3001\" \"pnpm run dev:watch:troubleshooting\"",
"dev:secrets:pull": "AWS_PROFILE=supa-dev node ../../scripts/getSecrets.js -n local/docs",
"dev:watch:troubleshooting": "node ./scripts/troubleshooting/watch.mjs",
"embeddings": "tsx --conditions=react-server scripts/search/generate-embeddings.ts",
"embeddings:nimbus": "ENABLED_FEATURES_OVERRIDE_DISABLE_ALL=true pnpm run embeddings",
"embeddings:refresh": "pnpm run embeddings --refresh",
"embeddings:nimbus:refresh": "ENABLED_FEATURES_OVERRIDE_DISABLE_ALL=true pnpm run embeddings:refresh",
"last-changed": "tsx scripts/last-changed.ts",
"last-changed:reset": "pnpm run last-changed -- --reset",
"lint": "eslint .",
"lint:mdx": "supa-mdx-lint content --config ../../supa-mdx-lint.config.toml",
"postbuild": "pnpm run build:sitemap && ./../../scripts/upload-static-assets.sh",
"prebuild": "pnpm run codegen:graphql && pnpm run codegen:references && pnpm run codegen:examples && pnpm run build:markdown && pnpm run build:markdown-manifest && pnpm run build:gz-archive",
"predev": "pnpm run codegen:graphql && pnpm run codegen:references && pnpm run codegen:examples && pnpm run build:markdown-manifest",
"preembeddings": "pnpm run codegen:references",
"preinstall": "npx only-allow pnpm",
"presync": "pnpm run codegen:graphql",
"pretest": "pnpm run codegen:examples",
"pretypecheck": "pnpm run codegen:graphql && pnpm run build:markdown-manifest && next typegen",
"start": "next start",
"sync": "tsx --conditions=react-server ./resources/rootSync.ts",
"test": "pnpm supabase start && pnpm run test:local && pnpm supabase stop",
"test:local": "vitest --exclude \"**/*.smoke.test.ts\"",
"test:local:unwatch": "vitest --exclude \"**/*.smoke.test.ts\" --run",
"test:smoke": "pnpm run codegen:references && vitest -t \"prod smoke test\"",
"troubleshooting:sync": "node features/docs/Troubleshooting.script.mjs",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"@har-sdk/openapi-sampler": "^2.2.0",
"@hookform/resolvers": "^3.1.1",
"@mdx-js/loader": "^3.0.0",
"@mdx-js/react": "^3.0.0",
"@next/bundle-analyzer": "15.3.1",
"@next/mdx": "15.3.1",
"@octokit/auth-app": "^7.0.0",
"@octokit/core": "^7.0.6",
"@octokit/graphql": "^8.0.0",
"@octokit/plugin-paginate-graphql": "^4.0.0",
"@octokit/plugin-retry": "^8.1.0",
"@sentry/nextjs": "catalog:",
"@supabase/supabase-js": "catalog:",
"@tanstack/react-query": "~5.83.0",
"acorn": "^8.11.3",
"acorn-typescript": "^1.4.13",
"ai-commands": "workspace:*",
"beautiful-mermaid": "^1.1.3",
"common": "workspace:*",
"common-tags": "^1.8.2",
"config": "workspace:*",
"dev-tools": "workspace:*",
"eslint-config-supabase": "workspace:*",
"framer-motion": "^11.18.2",
"github-slugger": "^2.0.0",
"graphql": "^16.10.0",
"graphql-validation-complexity": "^0.4.2",
"gray-matter": "^4.0.3",
"hast-util-has-property": "^2.0.1",
"hast-util-to-html": "^9.0.5",
"icons": "workspace:*",
"isbot": "^5.1.2",
"js-yaml": "^3.14.2",
"jsrsasign": "^11.0.0",
"katex": "^0.16.21",
"libpg-query": "15.2.0",
"lodash-es": "catalog:",
"lucide-react": "*",
"mdast": "^3.0.0",
"mdast-util-from-markdown": "^1.2.0",
"mdast-util-gfm": "^2.0.2",
"mdast-util-mdx": "^2.0.0",
"mdast-util-mdx-jsx": "^2.0.0",
"mdast-util-to-hast": "^12.3.0",
"mdast-util-to-markdown": "^1.5.0",
"mdast-util-to-string": "^3.1.1",
"micromark-extension-gfm": "^2.0.3",
"micromark-extension-mdxjs": "^1.0.0",
"next": "^15.5.18",
"next-mdx-remote-client": "^1.1.7",
"next-plugin-yaml": "^1.0.1",
"next-themes": "catalog:",
"nuqs": "^1.19.1",
"openai": "^4.75.1",
"openapi-fetch": "0.12.4",
"radix-ui": "catalog:",
"react": "catalog:",
"react-copy-to-clipboard": "^5.1.0",
"react-dom": "catalog:",
"react-error-boundary": "^4.0.12",
"react-hook-form": "^7.71.2",
"react-intersection-observer": "^9.5.3",
"react-markdown": "^10.1.0",
"react-use": "^17.5.0",
"rehype-katex": "^7.0.0",
"rehype-slug": "^5.1.0",
"remark": "^14.0.2",
"remark-emoji": "^3.1.2",
"remark-gfm": "^4.0.0",
"remark-math": "^6.0.0",
"server-only": "^0.0.1",
"shared-data": "workspace:*",
"ui": "workspace:*",
"ui-patterns": "workspace:*",
"unist-builder": "^3.0.1",
"unist-util-filter": "^4.0.1",
"unist-util-visit": "^4.1.2",
"uuid": "^14.0.0",
"valtio": "catalog:",
"yaml": "^2.8.1",
"zod": "catalog:"
},
"devDependencies": {
"@babel/core": "*",
"@graphiql/toolkit": "^0.9.1",
"@graphql-codegen/cli": "5.0.5",
"@graphql-codegen/typescript": "4.1.6",
"@graphql-codegen/typescript-resolvers": "4.5.0",
"@supabase/supa-mdx-lint": "0.3.2",
"@testing-library/react": "^16.0.0",
"@types/common-tags": "^1.8.4",
"@types/estree": "1.0.5",
"@types/graphql-validation-complexity": "^0.4.4",
"@types/hast": "^2.3.4",
"@types/jsrsasign": "^10.5.12",
"@types/lodash-es": "^4.17.12",
"@types/mdast": "^3.0.0",
"@types/node": "catalog:",
"@types/react": "catalog:",
"@types/react-copy-to-clipboard": "^5.0.4",
"@types/react-dom": "catalog:",
"@types/unist": "^2.0.6",
"@redocly/cli": "^2.26.0",
"amaro": "^1.1.5",
"api-types": "workspace:*",
"cheerio": "^1.0.0-rc.12",
"chokidar": "^4.0.3",
"concurrently": "^9.1.2",
"config": "workspace:*",
"dotenv": "^16.0.3",
"ejs": "^3.1.10",
"globby": "^13.2.2",
"graphiql": "^4.0.2",
"openapi-types": "^12.1.3",
"postcss": "catalog:",
"shiki": "^3.2.1",
"shx": "^0.4.0",
"simple-git": "^3.24.0",
"slugify": "^1.6.6",
"smol-toml": "^1.3.1",
"tailwindcss": "catalog:",
"tar": "^7.5.13",
"tsconfig": "workspace:*",
"tsx": "catalog:",
"twoslash": "^0.3.1",
"typescript": "catalog:",
"unist-util-visit-parents": "5.1.3",
"vite": "catalog:",
"vite-tsconfig-paths": "catalog:",
"vitest": "catalog:"
}
}
Reference in new issue View Git Blame Copy Permalink