Preview sites are being crawled because the rewrite causes the docs production
content (which naturally has `index, follow`) to be served under the www
preview domain. We shouldn’t be rewriting in any environment except production.
* feat: llms.txt
* feat: split llms.txt into multiple files
We have too many docs, so the concatenated text file uses an unreasonable amount of tokens. Chunk it up a little so it's more usable.
Charis