Preview sites are being crawled because the rewrite causes the docs production content (which naturally has `index, follow`) to be served under the www preview domain. We shouldn’t be rewriting in any environment except production.