Files
supabase/apps/www/scripts/generateMdContent.mjs
Pamela Chia d409836ca7 feat(www,docs): serve marketing pages as .md, advertise via link rel=alternate (#45277)
## Summary

Adds `/<page>.md` routes for 10 marketing/product pages (homepage, auth,
database, edge-functions, realtime, storage, vector, pricing,
modules/cron, modules/queues) so AI agents can fetch clean markdown
instead of parsing JS-rendered HTML. Also advertises the markdown
alternate via `<link rel="alternate" type="text/markdown">` on marketing
and docs pages so agents can discover it.

Pricing is generated dynamically via `generatePricingContent()` (single
source of truth with `/llms.txt` and `/llms-full.txt`); the other nine
slugs are bundled at build time from `content/md/*.md` into a
`MD_CONTENT` map.

Supersedes #44891 (rebased fresh off current master to avoid a 9-commit
replay over rename/rename conflicts created by #44897).

## Changes

- New `/api-v2/md/[...slug]` route handler returns the bundled markdown
(or dynamic pricing) with `Content-Type: text/markdown`,
`X-Content-Type-Options: nosniff`, and appropriate cache headers
- Middleware rewrites `/<slug>.md` and `Accept: text/markdown` to the
API route for the `MD_PAGES` allowlist; trailing-slash variants
(`/auth/`) are normalized so they resolve the same as `/auth`
- Build-time codegen `scripts/generateMdContent.mjs` scans `content/md/`
and emits `app/api-v2/md/content.generated.ts` exporting both
`MD_CONTENT` (Map) and `MD_PAGES` (Set, incl. dynamic `pricing`). Fails
the build on slug collision between `content/md/` and `DYNAMIC_SLUGS`.
Adding a new marketing `.md` is just dropping a file in `content/md/`
(also update `PRODUCT_OVERVIEW_LINKS` in `/llms.txt` since that list is
editorial).
- 8 permanent redirects `/llms/<product>.txt` → `/<product>.md` so
legacy URLs in caches and downstream `llms.txt` copies keep working
- `/llms.txt` product overview now references `.md` URLs (incl.
`modules/cron`, `modules/queues`); `/llms-full.txt` iterates
`MD_CONTENT.values()` (homepage first, then alphabetical) and appends
dynamic pricing
- `/llms/[slug]` route slimmed to proxy SDK reference files (`js.txt`,
`dart.txt`, etc.) since redirects handle product slugs and pricing;
pricing branch retained as fallback in case redirects are bypassed
- `apps/www/pages/_app.tsx` injects the alternate link conditionally
based on `MD_PAGES`; `/pricing` (app router) sets it via page metadata
- `apps/docs/app/page.tsx` (the `/docs` root) sets the text/markdown
alternate to `/llms-full.txt`; per-guide pages override with their
specific `.md` URL via `genGuideMeta` in `GuidesMdx.utils.tsx`. Other
docs pages (reference, troubleshooting) inherit nothing.
- `apps/www/.vercelignore`: replaces the prior `*.md`/`README.md` rules
with `*.md` + `!content/md/**/*.md` so Edge Function READMEs and future
scratch `.md` files aren't silently shipped to the build artifact
- Drops `apps/www/data/llms/*.txt` and the related
`outputFileTracingIncludes`
- Test coverage for the new middleware branches: `.md` suffix rewrite
(allowlisted vs. fall-through), `Accept: text/markdown` content
negotiation, trailing-slash normalization

## Testing (Vercel preview)

Local dev server smoke tests passing on `:3771` after each iteration.
Re-verified on the preview URL after the latest hardening commit:

- [x] `curl -I https://<preview>/llms/auth.txt` — expect `308 Permanent
Redirect` to `/auth.md`
- [x] `curl https://<preview>/auth.md | head -3` — expect `# Supabase
Auth`
- [x] `curl https://<preview>/pricing.md | head -3` — expect `# Supabase
Pricing` with current tier values
- [x] `curl https://<preview>/modules/cron.md | head -3` — expect `#
Supabase Cron`
- [x] `curl -H 'Accept: text/markdown' https://<preview>/ | head -3` —
expect `# Supabase` (homepage.md)
- [x] `curl https://<preview>/llms.txt` — Product Overview section lists
`.md` URLs and includes Cron + Queues
- [x] `curl https://<preview>/llms-full.txt | grep -E '^# Supabase
(Cron\|Queues\|Pricing)'` — Cron and Pricing each match once; Queues
matches twice (marketing module + existing docs guide)
- [x] View source on `/`, `/pricing`, `/database` — expect `<link
rel="alternate" type="text/markdown" href="/<slug>.md">`
- [x] View source on `/docs` — expect `<link rel="alternate"
type="text/markdown" href="/llms-full.txt">`
- [x] View source on a docs guide page (e.g., `/docs/guides/auth`) —
expect per-guide `.md` alternate; reference/troubleshooting pages should
NOT emit a markdown alternate
- [x] `curl -I https://<preview>/auth.md` — expect
`X-Content-Type-Options: nosniff`
- [x] `curl -I -L -H 'Accept: text/markdown' https://<preview>/auth/` —
should resolve to markdown content (trailing-slash normalization, with
Vercel's auto-redirect)

## Linear

- fixes GROWTH-760

## Follow-up (separate PR)

GROWTH-760 also asks about extending `.md` to blog/customers/events.
Different mechanism (path-prefix middleware, MDX read at request time
via `gray-matter`) so it deserves its own review. Will open a follow-up
PR after this lands.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Serve prebuilt and dynamic Markdown docs via new markdown endpoints
and routing; pages now advertise markdown alternates (including
pricing).
  * Added Cron and Queues module documentation pages.

* **Documentation**
  * Minor formatting tweaks to Realtime and Storage docs.

* **Chores**
* Added build-time Markdown content generation and adjusted
ignore/deploy rules for generated files.
* Added redirects from legacy text-based product URLs to new markdown
pages.

* **Tests**
* Expanded tests for markdown routing and content-negotiation behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-28 16:41:03 +09:00

99 lines
3.2 KiB
JavaScript

// @ts-check
/**
* Scans content/md/ and emits a TypeScript module exporting the markdown
* content as a Map plus the slug allowlist as a Set. Static imports of the
* generated file make the content traceable by @vercel/nft so it ends up in
* the serverless bundle without runtime fs.readFile.
*
* Drop a new .md file in content/md/ and the route handler picks it up on
* the next content:build — no constant to edit.
*/
import { promises as fs } from 'fs'
import path from 'path'
import { fileURLToPath } from 'url'
const __dirname = path.dirname(fileURLToPath(import.meta.url))
const contentDir = path.join(__dirname, '../content/md')
const outputPath = path.join(__dirname, '../app/api-v2/md/content.generated.ts')
// 'pricing' is served dynamically via generatePricingContent() instead of
// from a .md file; it still needs to be in MD_PAGES so middleware rewrites
// /pricing.md to the API route.
const DYNAMIC_SLUGS = ['pricing']
async function collectMdFiles(dir, prefix = '') {
const results = []
const dirents = await fs.readdir(dir, { withFileTypes: true })
for (const dirent of dirents) {
const slug = prefix ? `${prefix}/${dirent.name}` : dirent.name
if (dirent.isDirectory()) {
results.push(...(await collectMdFiles(path.join(dir, dirent.name), slug)))
} else if (dirent.name.endsWith('.md')) {
results.push(slug.replace(/\.md$/, ''))
}
}
return results
}
// homepage first (it's the site overview), then everything else alphabetical.
function sortSlugs(a, b) {
if (a === 'homepage') return -1
if (b === 'homepage') return 1
return a.localeCompare(b)
}
const slugs = (await collectMdFiles(contentDir)).sort(sortSlugs)
if (slugs.length === 0) {
console.error('❌ No .md files found in content/md/')
process.exit(1)
}
// A static file with the same slug as a dynamic generator would land in both
// MD_CONTENT and the dynamic append path, so /llms-full.txt would emit it
// twice. Fail the build instead of shipping a duplicate.
const collisions = slugs.filter((s) => DYNAMIC_SLUGS.includes(s))
if (collisions.length > 0) {
console.error(
`❌ Slug collision: [${collisions.join(', ')}] is reserved for a dynamic generator. ` +
`Remove the corresponding file from content/md/ or update DYNAMIC_SLUGS.`
)
process.exit(1)
}
const entries = []
const errors = []
for (const slug of slugs) {
const filePath = path.join(contentDir, `${slug}.md`)
try {
const content = await fs.readFile(filePath, 'utf-8')
entries.push(` [${JSON.stringify(slug)}, ${JSON.stringify(content)}]`)
} catch (err) {
errors.push(`${slug}.md: ${err.message}`)
}
}
if (errors.length > 0) {
console.error('❌ Failed to read .md files:')
errors.forEach((e) => console.error(` ${e}`))
process.exit(1)
}
const allSlugs = [...slugs, ...DYNAMIC_SLUGS]
const pageEntries = allSlugs.map((s) => ` ${JSON.stringify(s)}`).join(',\n')
const output = `// AUTO-GENERATED by scripts/generateMdContent.mjs — do not edit
export const MD_CONTENT = new Map<string, string>([
${entries.join(',\n')},
])
export const MD_PAGES = new Set<string>([
${pageEntries},
])
`
await fs.writeFile(outputPath, output, 'utf-8')
console.log(`✅ Generated ${outputPath} (${entries.length} files, ${allSlugs.length} pages)`)