feat(www): AI content via Content-Signal in robots.txt (#44875)

Add
[Content-Signal](https://www.ietf.org/archive/id/draft-romm-aipref-contentsignals-00.txt)
directive to robots.txt opting into AI training, search, and inference
use.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated website and documentation robots.txt to enable AI training,
search indexing, and AI input processing for site content.
* Minor formatting fix: ensured trailing newline at end of both
robots.txt files.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Greg Richardson authored and GitHub committed 2026-04-15 18:05:35 +00:00
1 parent 03d44c390d
commit 82f55217a2
2 files changed
+4 -2

No files matched your search

+2 -1
View File
@@ -1,3 +1,4 @@
User-agent: *
Allow: /
Sitemap: https://supabase.com/docs/sitemap.xml
Sitemap: https://supabase.com/docs/sitemap.xml
Content-Signal: ai-train=yes, search=yes, ai-input=yes
+2 -1
View File
@@ -1,2 +1,3 @@
User-agent: *
Sitemap: https://supabase.com/sitemap.xml
Sitemap: https://supabase.com/sitemap.xml
Content-Signal: ai-train=yes, search=yes, ai-input=yes