supabase

dima/supabase

0 Fork 0

mirror of https://github.com/supabase/supabase.git synced 2026-10-06 18:05:11 +03:00

Commit Graph

Author	SHA1	Message	Date
Pamela Chia	9113c2ba04	feat: markdown alternate tags + llms.txt cleanup (#48287 ) The June/July marketing redesign (#47271, #47228) rebuilt the homepage and product pages off the Pages Router, silently dropping their `<link rel="alternate" type="text/markdown">` head tags, and llms-full.txt has been accidentally embedding every blog/customer/event page via an `MD_CONTENT` spread. I restored the tags behind a shared helper, added a CI drift test so a future redesign can't drop them silently again, and trimmed both llms files to the agreed docs-index shape. Changed: - Markdown siblings advertised again: homepage, the 5 product pages, pricing, and blog emit absolute `.md` alternate URLs via a new `mdAlternates(slug)` helper (the one documented consumer of the tag parses it from `<head>` and fetches the `.md` sibling, so tags must point at the sibling, never the page itself). - Drift test: a vitest file walks `content/md/` and asserts every markdown-served slug's page wires the helper (or is covered by the Pages Router `_app.tsx` mechanism, whose alternate-link wiring the test also asserts directly so removing it fails CI too). Source-level assertions by design: page modules can't be imported under www's vitest config. Fails correctly when wiring is removed (verified by hiding a page and by altering the `_app.tsx` tag). - Vector orphan fixed: `content/md/vector.md` moved to `modules/vector` matching the live route (the page previously had no negotiation or tag, and `/modules/vector.md` 404'd); `/vector.md` now 308s to `/modules/vector.md` and the legacy `/llms/vector.txt` redirect no longer chains. - llms.txt + llms-full.txt: the `## Product Overview` sections are gone from both, each keeps a `## Pricing` section. This deletes the hand-maintained links array (a drift trap) and fixes the accidental ~470-page embed, shrinking llms-full.txt from ~9.8MB to ~4.9MB and dropping the 4.1MB generated content module from that route's serverless bundle. Note:** this PR is scoped to apps/www only. The docs side (troubleshooting pages and the rest of the docs surface) is handled separately through a consolidated manifest-gated mechanism; an earlier troubleshooting-tag commit was reverted out of this branch to keep the scopes clean. <details> <summary>Why alternate tags matter (background)</summary> Agents ingest markdown far more efficiently than our rendered HTML: a fraction of the tokens and no extraction step. Since #47770 removed UA-based serving (UA sniffing broke a major AI app's fetcher and poisoned CDN caches), markdown is served only on explicit request: a `.md` suffix URL, an `Accept: text/markdown` header, or llms.txt. That's the right serving model, but it makes the markdown twin invisible to any agent that doesn't already know our URL convention, and the major AI fetchers send browser/wildcard Accept headers, so bare URLs hand them HTML. The `<link rel="alternate" type="text/markdown">` head tag is the standards-based advertisement of the sibling. It has a documented consumer today: an agent CLI that parses the tag from `<head>` and then fetches the `.md` sibling, which is also why the tag must point at a real sibling URL and never at the page itself. Peer docs sites ship this tag as table stakes. These www pages used to carry it until the June/July marketing redesign silently dropped it; the drift test in this PR turns that regression class into a CI failure. </details> ## To test Tested locally (www + docs dev servers): - [x] `/llms.txt` renders `## Documentation` + single-link `## Pricing`, no Product Overview - [x] `/llms-full.txt` renders `# Supabase` → `## Pricing` → `## Documentation`, no Product Overview, ~4.9MB - [x] Full www suite: 6 files / 71 tests green; drift test fails correctly when a page is removed or the `_app.tsx` wiring is altered - [x] `generateMdContent.mjs` emits `modules/vector`, bare `vector` slug gone On the Vercel preview (browser-verified with Playwright): - [x] Alternate tag present on `/`, `/auth`, `/database`, `/storage`, `/edge-functions`, `/realtime`, `/pricing`, and a blog post: exactly one tag each, href = preview origin + `.md` sibling - [x] `/vector.md` → 308 → `/modules/vector.md`, renders as markdown (`# Supabase Vector`) - [x] `/llms.txt` shows single-link `## Pricing`, no Product Overview - [x] Coverage sweep: all 482 `MD_PAGES` slugs + changelog index/entry curled on the preview; 471 pages carry exactly one tag, all `.md` siblings 200 as `text/markdown`. The 11 misses are legacy blog slugs whose HTML 308-redirects away (stale `MD_PAGES` entries predating this PR, no head to tag; follow-up tracked in Linear) Post-merge prod: - [ ] Full llms.txt link sweep (every linked URL 200s; previews can't cover the docs-hosted links) ## Linear - fixes GROWTH-1013 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * New Features * Added Markdown alternate links across key product, pricing, blog, and troubleshooting pages. * Added Supabase Vector documentation covering features, use cases, workflows, and technical details. * Updated AI-focused documentation indexes with dedicated pricing content. * Added redirects for updated Vector documentation URLs. * Tests * Added coverage to verify Markdown documentation links stay aligned with available pages. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-07-27 17:06:16 +08:00
Pamela Chia	d409836ca7	feat(www,docs): serve marketing pages as .md, advertise via link rel=alternate (#45277 ) ## Summary Adds `/<page>.md` routes for 10 marketing/product pages (homepage, auth, database, edge-functions, realtime, storage, vector, pricing, modules/cron, modules/queues) so AI agents can fetch clean markdown instead of parsing JS-rendered HTML. Also advertises the markdown alternate via `<link rel="alternate" type="text/markdown">` on marketing and docs pages so agents can discover it. Pricing is generated dynamically via `generatePricingContent()` (single source of truth with `/llms.txt` and `/llms-full.txt`); the other nine slugs are bundled at build time from `content/md/.md` into a `MD_CONTENT` map. Supersedes #44891 (rebased fresh off current master to avoid a 9-commit replay over rename/rename conflicts created by #44897). ## Changes - New `/api-v2/md/[...slug]` route handler returns the bundled markdown (or dynamic pricing) with `Content-Type: text/markdown`, `X-Content-Type-Options: nosniff`, and appropriate cache headers - Middleware rewrites `/<slug>.md` and `Accept: text/markdown` to the API route for the `MD_PAGES` allowlist; trailing-slash variants (`/auth/`) are normalized so they resolve the same as `/auth` - Build-time codegen `scripts/generateMdContent.mjs` scans `content/md/` and emits `app/api-v2/md/content.generated.ts` exporting both `MD_CONTENT` (Map) and `MD_PAGES` (Set, incl. dynamic `pricing`). Fails the build on slug collision between `content/md/` and `DYNAMIC_SLUGS`. Adding a new marketing `.md` is just dropping a file in `content/md/` (also update `PRODUCT_OVERVIEW_LINKS` in `/llms.txt` since that list is editorial). - 8 permanent redirects `/llms/<product>.txt` → `/<product>.md` so legacy URLs in caches and downstream `llms.txt` copies keep working - `/llms.txt` product overview now references `.md` URLs (incl. `modules/cron`, `modules/queues`); `/llms-full.txt` iterates `MD_CONTENT.values()` (homepage first, then alphabetical) and appends dynamic pricing - `/llms/[slug]` route slimmed to proxy SDK reference files (`js.txt`, `dart.txt`, etc.) since redirects handle product slugs and pricing; pricing branch retained as fallback in case redirects are bypassed - `apps/www/pages/_app.tsx` injects the alternate link conditionally based on `MD_PAGES`; `/pricing` (app router) sets it via page metadata - `apps/docs/app/page.tsx` (the `/docs` root) sets the text/markdown alternate to `/llms-full.txt`; per-guide pages override with their specific `.md` URL via `genGuideMeta` in `GuidesMdx.utils.tsx`. Other docs pages (reference, troubleshooting) inherit nothing. - `apps/www/.vercelignore`: replaces the prior `.md`/`README.md` rules with `.md` + `!content/md//.md` so Edge Function READMEs and future scratch `.md` files aren't silently shipped to the build artifact - Drops `apps/www/data/llms/.txt` and the related `outputFileTracingIncludes` - Test coverage for the new middleware branches: `.md` suffix rewrite (allowlisted vs. fall-through), `Accept: text/markdown` content negotiation, trailing-slash normalization ## Testing (Vercel preview) Local dev server smoke tests passing on `:3771` after each iteration. Re-verified on the preview URL after the latest hardening commit: - [x] `curl -I https://<preview>/llms/auth.txt` — expect `308 Permanent Redirect` to `/auth.md` - [x] `curl https://<preview>/auth.md \| head -3` — expect `# Supabase Auth` - [x] `curl https://<preview>/pricing.md \| head -3` — expect `# Supabase Pricing` with current tier values - [x] `curl https://<preview>/modules/cron.md \| head -3` — expect `# Supabase Cron` - [x] `curl -H 'Accept: text/markdown' https://<preview>/ \| head -3` — expect `# Supabase` (homepage.md) - [x] `curl https://<preview>/llms.txt` — Product Overview section lists `.md` URLs and includes Cron + Queues - [x] `curl https://<preview>/llms-full.txt \| grep -E '^# Supabase (Cron\\|Queues\\|Pricing)'` — Cron and Pricing each match once; Queues matches twice (marketing module + existing docs guide) - [x] View source on `/`, `/pricing`, `/database` — expect `<link rel="alternate" type="text/markdown" href="/<slug>.md">` - [x] View source on `/docs` — expect `<link rel="alternate" type="text/markdown" href="/llms-full.txt">` - [x] View source on a docs guide page (e.g., `/docs/guides/auth`) — expect per-guide `.md` alternate; reference/troubleshooting pages should NOT emit a markdown alternate - [x] `curl -I https://<preview>/auth.md` — expect `X-Content-Type-Options: nosniff` - [x] `curl -I -L -H 'Accept: text/markdown' https://<preview>/auth/` — should resolve to markdown content (trailing-slash normalization, with Vercel's auto-redirect) ## Linear - fixes GROWTH-760 ## Follow-up (separate PR) GROWTH-760 also asks about extending `.md` to blog/customers/events. Different mechanism (path-prefix middleware, MDX read at request time via `gray-matter`) so it deserves its own review. Will open a follow-up PR after this lands. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit New Features * Serve prebuilt and dynamic Markdown docs via new markdown endpoints and routing; pages now advertise markdown alternates (including pricing). * Added Cron and Queues module documentation pages. * Documentation * Minor formatting tweaks to Realtime and Storage docs. * Chores * Added build-time Markdown content generation and adjusted ignore/deploy rules for generated files. * Added redirects from legacy text-based product URLs to new markdown pages. * Tests * Expanded tests for markdown routing and content-negotiation behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-04-28 16:41:03 +09:00

Author

SHA1

Message

Date

Pamela Chia

9113c2ba04

feat: markdown alternate tags + llms.txt cleanup (#48287 )

The June/July marketing redesign (#47271, #47228) rebuilt the homepage
and product pages off the Pages Router, silently dropping their `<link
rel="alternate" type="text/markdown">` head tags, and llms-full.txt has
been accidentally embedding every blog/customer/event page via an
`MD_CONTENT` spread. I restored the tags behind a shared helper, added a
CI drift test so a future redesign can't drop them silently again, and
trimmed both llms files to the agreed docs-index shape.

**Changed:**

- **Markdown siblings advertised again**: homepage, the 5 product pages,
pricing, and blog emit absolute `.md` alternate URLs via a new
`mdAlternates(slug)` helper (the one documented consumer of the tag
parses it from `<head>` and fetches the `.md` sibling, so tags must
point at the sibling, never the page itself).
- **Drift test**: a vitest file walks `content/md/**` and asserts every
markdown-served slug's page wires the helper (or is covered by the Pages
Router `_app.tsx` mechanism, whose alternate-link wiring the test also
asserts directly so removing it fails CI too). Source-level assertions
by design: page modules can't be imported under www's vitest config.
Fails correctly when wiring is removed (verified by hiding a page and by
altering the `_app.tsx` tag).
- **Vector orphan fixed**: `content/md/vector.md` moved to
`modules/vector` matching the live route (the page previously had no
negotiation or tag, and `/modules/vector.md` 404'd); `/vector.md` now
308s to `/modules/vector.md` and the legacy `/llms/vector.txt` redirect
no longer chains.
- **llms.txt + llms-full.txt**: the `## Product Overview` sections are
gone from both, each keeps a `## Pricing` section. This deletes the
hand-maintained links array (a drift trap) and fixes the accidental
~470-page embed, shrinking llms-full.txt from ~9.8MB to ~4.9MB and
dropping the 4.1MB generated content module from that route's serverless
bundle.

**Note:** this PR is scoped to apps/www only. The docs side
(troubleshooting pages and the rest of the docs surface) is handled
separately through a consolidated manifest-gated mechanism; an earlier
troubleshooting-tag commit was reverted out of this branch to keep the
scopes clean.

<details>
<summary>Why alternate tags matter (background)</summary>

Agents ingest markdown far more efficiently than our rendered HTML: a
fraction of the tokens and no extraction step. Since #47770 removed
UA-based serving (UA sniffing broke a major AI app's fetcher and
poisoned CDN caches), markdown is served only on explicit request: a
`.md` suffix URL, an `Accept: text/markdown` header, or llms.txt. That's
the right serving model, but it makes the markdown twin invisible to any
agent that doesn't already know our URL convention, and the major AI
fetchers send browser/wildcard Accept headers, so bare URLs hand them
HTML.

The `<link rel="alternate" type="text/markdown">` head tag is the
standards-based advertisement of the sibling. It has a documented
consumer today: an agent CLI that parses the tag from `<head>` and then
fetches the `.md` sibling, which is also why the tag must point at a
real sibling URL and never at the page itself. Peer docs sites ship this
tag as table stakes. These www pages used to carry it until the
June/July marketing redesign silently dropped it; the drift test in this
PR turns that regression class into a CI failure.

</details>

## To test

Tested locally (www + docs dev servers):
- [x] `/llms.txt` renders `## Documentation` + single-link `## Pricing`,
no Product Overview
- [x] `/llms-full.txt` renders `# Supabase` → `## Pricing` → `##
Documentation`, no Product Overview, ~4.9MB
- [x] Full www suite: 6 files / 71 tests green; drift test fails
correctly when a page is removed or the `_app.tsx` wiring is altered
- [x] `generateMdContent.mjs` emits `modules/vector`, bare `vector` slug
gone

On the Vercel preview (browser-verified with Playwright):
- [x] Alternate tag present on `/`, `/auth`, `/database`, `/storage`,
`/edge-functions`, `/realtime`, `/pricing`, and a blog post: exactly one
tag each, href = preview origin + `.md` sibling
- [x] `/vector.md` → 308 → `/modules/vector.md`, renders as markdown (`#
Supabase Vector`)
- [x] `/llms.txt` shows single-link `## Pricing`, no Product Overview
- [x] Coverage sweep: all 482 `MD_PAGES` slugs + changelog index/entry
curled on the preview; 471 pages carry exactly one tag, all `.md`
siblings 200 as `text/markdown`. The 11 misses are legacy blog slugs
whose HTML 308-redirects away (stale `MD_PAGES` entries predating this
PR, no head to tag; follow-up tracked in Linear)

Post-merge prod:
- [ ] Full llms.txt link sweep (every linked URL 200s; previews can't
cover the docs-hosted links)

## Linear

- fixes GROWTH-1013


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added Markdown alternate links across key product, pricing, blog, and
troubleshooting pages.
* Added Supabase Vector documentation covering features, use cases,
workflows, and technical details.
* Updated AI-focused documentation indexes with dedicated pricing
content.
  * Added redirects for updated Vector documentation URLs.

* **Tests**
* Added coverage to verify Markdown documentation links stay aligned
with available pages.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

2026-07-27 17:06:16 +08:00

Pamela Chia

d409836ca7

feat(www,docs): serve marketing pages as .md, advertise via link rel=alternate (#45277 )

## Summary

Adds `/<page>.md` routes for 10 marketing/product pages (homepage, auth,
database, edge-functions, realtime, storage, vector, pricing,
modules/cron, modules/queues) so AI agents can fetch clean markdown
instead of parsing JS-rendered HTML. Also advertises the markdown
alternate via `<link rel="alternate" type="text/markdown">` on marketing
and docs pages so agents can discover it.

Pricing is generated dynamically via `generatePricingContent()` (single
source of truth with `/llms.txt` and `/llms-full.txt`); the other nine
slugs are bundled at build time from `content/md/*.md` into a
`MD_CONTENT` map.

Supersedes #44891 (rebased fresh off current master to avoid a 9-commit
replay over rename/rename conflicts created by #44897).

## Changes

- New `/api-v2/md/[...slug]` route handler returns the bundled markdown
(or dynamic pricing) with `Content-Type: text/markdown`,
`X-Content-Type-Options: nosniff`, and appropriate cache headers
- Middleware rewrites `/<slug>.md` and `Accept: text/markdown` to the
API route for the `MD_PAGES` allowlist; trailing-slash variants
(`/auth/`) are normalized so they resolve the same as `/auth`
- Build-time codegen `scripts/generateMdContent.mjs` scans `content/md/`
and emits `app/api-v2/md/content.generated.ts` exporting both
`MD_CONTENT` (Map) and `MD_PAGES` (Set, incl. dynamic `pricing`). Fails
the build on slug collision between `content/md/` and `DYNAMIC_SLUGS`.
Adding a new marketing `.md` is just dropping a file in `content/md/`
(also update `PRODUCT_OVERVIEW_LINKS` in `/llms.txt` since that list is
editorial).
- 8 permanent redirects `/llms/<product>.txt` → `/<product>.md` so
legacy URLs in caches and downstream `llms.txt` copies keep working
- `/llms.txt` product overview now references `.md` URLs (incl.
`modules/cron`, `modules/queues`); `/llms-full.txt` iterates
`MD_CONTENT.values()` (homepage first, then alphabetical) and appends
dynamic pricing
- `/llms/[slug]` route slimmed to proxy SDK reference files (`js.txt`,
`dart.txt`, etc.) since redirects handle product slugs and pricing;
pricing branch retained as fallback in case redirects are bypassed
- `apps/www/pages/_app.tsx` injects the alternate link conditionally
based on `MD_PAGES`; `/pricing` (app router) sets it via page metadata
- `apps/docs/app/page.tsx` (the `/docs` root) sets the text/markdown
alternate to `/llms-full.txt`; per-guide pages override with their
specific `.md` URL via `genGuideMeta` in `GuidesMdx.utils.tsx`. Other
docs pages (reference, troubleshooting) inherit nothing.
- `apps/www/.vercelignore`: replaces the prior `*.md`/`README.md` rules
with `*.md` + `!content/md/**/*.md` so Edge Function READMEs and future
scratch `.md` files aren't silently shipped to the build artifact
- Drops `apps/www/data/llms/*.txt` and the related
`outputFileTracingIncludes`
- Test coverage for the new middleware branches: `.md` suffix rewrite
(allowlisted vs. fall-through), `Accept: text/markdown` content
negotiation, trailing-slash normalization

## Testing (Vercel preview)

Local dev server smoke tests passing on `:3771` after each iteration.
Re-verified on the preview URL after the latest hardening commit:

- [x] `curl -I https://<preview>/llms/auth.txt` — expect `308 Permanent
Redirect` to `/auth.md`
- [x] `curl https://<preview>/auth.md | head -3` — expect `# Supabase
Auth`
- [x] `curl https://<preview>/pricing.md | head -3` — expect `# Supabase
Pricing` with current tier values
- [x] `curl https://<preview>/modules/cron.md | head -3` — expect `#
Supabase Cron`
- [x] `curl -H 'Accept: text/markdown' https://<preview>/ | head -3` —
expect `# Supabase` (homepage.md)
- [x] `curl https://<preview>/llms.txt` — Product Overview section lists
`.md` URLs and includes Cron + Queues
- [x] `curl https://<preview>/llms-full.txt | grep -E '^# Supabase
(Cron\|Queues\|Pricing)'` — Cron and Pricing each match once; Queues
matches twice (marketing module + existing docs guide)
- [x] View source on `/`, `/pricing`, `/database` — expect `<link
rel="alternate" type="text/markdown" href="/<slug>.md">`
- [x] View source on `/docs` — expect `<link rel="alternate"
type="text/markdown" href="/llms-full.txt">`
- [x] View source on a docs guide page (e.g., `/docs/guides/auth`) —
expect per-guide `.md` alternate; reference/troubleshooting pages should
NOT emit a markdown alternate
- [x] `curl -I https://<preview>/auth.md` — expect
`X-Content-Type-Options: nosniff`
- [x] `curl -I -L -H 'Accept: text/markdown' https://<preview>/auth/` —
should resolve to markdown content (trailing-slash normalization, with
Vercel's auto-redirect)

## Linear

- fixes GROWTH-760

## Follow-up (separate PR)

GROWTH-760 also asks about extending `.md` to blog/customers/events.
Different mechanism (path-prefix middleware, MDX read at request time
via `gray-matter`) so it deserves its own review. Will open a follow-up
PR after this lands.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Serve prebuilt and dynamic Markdown docs via new markdown endpoints
and routing; pages now advertise markdown alternates (including
pricing).
  * Added Cron and Queues module documentation pages.

* **Documentation**
  * Minor formatting tweaks to Realtime and Storage docs.

* **Chores**
* Added build-time Markdown content generation and adjusted
ignore/deploy rules for generated files.
* Added redirects from legacy text-based product URLs to new markdown
pages.

* **Tests**
* Expanded tests for markdown routing and content-negotiation behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

2026-04-28 16:41:03 +09:00