feat(studio-evals): custom search_docs tool for the eval harness (no token / no PAT) (#50092)

- Eval harness's only live tool, `search_docs`, no longer needs the
in-process MCP client or its dummy token — it now calls the public docs
GraphQL API (`https://supabase.com/docs/api/graphql`) directly. Low risk
as this is an eval-harness change only. Production assistant path
(`mcp-tools.ts`) untouched.

**Update:** per [@mattrossman's
review](https://github.com/supabase/supabase/pull/50092#discussion_r3980396341),
the eval tool's description embeds the Content API's own GraphQL schema
(fetched via a `{ schema }` query and minified with `gqlmin`), mirroring
how `@supabase/mcp-server-supabase`'s `docs-tools.ts`/`loadSchema`
populates production's `search_docs` description. Without it, the model
had no schema to work from and issued malformed queries, which caused
the 218 `search_docs` errors and the -25pp Docs Faithfulness regression
in the first eval run on this PR. Schema loading is required:
`createSearchDocsTool()` rejects if the schema fetch fails, so preflight
and the gated eval job fail loudly instead of producing untrustworthy
fallback results. `createSearchDocsTool` is async because the `ai`
package's `tool()` only accepts a plain string `description`, unlike the
MCP SDK's async description support; both callers (`getMockTools`,
`evals/preflight.ts`) await it. `gqlmin` is a direct `apps/studio`
dependency and was already transitive via
`@supabase/mcp-server-supabase`.

### Verification
- `pnpm -C apps/studio exec -- tsc --noEmit` reaches the compiler; it
reports only the pre-existing unrelated
`packages/ui-patterns/src/McpUrlBuilder/components/InstructionBlocks.tsx`
`StaticImageData` error.
- `pnpm -C apps/studio exec -- vitest run
lib/ai/tools/mock-tools.test.ts lib/ai/tools/mcp-tools.test.ts` — 21/21
passed.
- `pnpm exec tsx evals/preflight.ts` — live docs API schema fetch and
search_docs call passed.
- `NEXT_PUBLIC_CONTENT_API_URL=http://127.0.0.1:1/graphql pnpm -C
apps/studio exec -- tsx evals/preflight.ts` — failed fast as expected,
proving schema/API failures gate evals.
- Fresh `run-evals` pass: Docs Faithfulness 55.7% (0pp), with no
systemic `search_docs` regression.

Risk: eval-harness-only; schema/API outage now fails the eval job before
scoring rather than allowing fallback descriptions.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added documentation search powered by the public Supabase
documentation GraphQL API.
* Documentation search results now include live schema information and
clearer error handling for failed or invalid requests.

* **Bug Fixes**
* Improved evaluation tooling reliability by removing unnecessary
connection-abort behavior.
* Updated validation to detect missing search tools and malformed
documentation responses.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Pedro RodriguesandClaude Sonnet 5 authored and GitHub committed 2026-09-14 12:51:14 +02:00
1 parent 25f411657d
commit 22d7bc0cfd
9 files changed
+233 -192

No files matched your search

+1
View File
@@ -108,6 +108,7 @@
"framer-motion": "^11.18.2",
"fuse.js": "^7.4.0",
"generate-password-browser": "^1.1.0",
"gqlmin": "^0.3.1",
"graphiql": "^5.2.2",
"html-to-image": "^1.11.13",
"http-status": "^2.1.0",