Files
supabase/apps/studio/lib/ai/tools/mcp-tools.ts
T
Pedro RodriguesandClaude Sonnet 5 22d7bc0cfd feat(studio-evals): custom search_docs tool for the eval harness (no token / no PAT) (#50092)
- Eval harness's only live tool, `search_docs`, no longer needs the
in-process MCP client or its dummy token — it now calls the public docs
GraphQL API (`https://supabase.com/docs/api/graphql`) directly. Low risk
as this is an eval-harness change only. Production assistant path
(`mcp-tools.ts`) untouched.

**Update:** per [@mattrossman's
review](https://github.com/supabase/supabase/pull/50092#discussion_r3980396341),
the eval tool's description embeds the Content API's own GraphQL schema
(fetched via a `{ schema }` query and minified with `gqlmin`), mirroring
how `@supabase/mcp-server-supabase`'s `docs-tools.ts`/`loadSchema`
populates production's `search_docs` description. Without it, the model
had no schema to work from and issued malformed queries, which caused
the 218 `search_docs` errors and the -25pp Docs Faithfulness regression
in the first eval run on this PR. Schema loading is required:
`createSearchDocsTool()` rejects if the schema fetch fails, so preflight
and the gated eval job fail loudly instead of producing untrustworthy
fallback results. `createSearchDocsTool` is async because the `ai`
package's `tool()` only accepts a plain string `description`, unlike the
MCP SDK's async description support; both callers (`getMockTools`,
`evals/preflight.ts`) await it. `gqlmin` is a direct `apps/studio`
dependency and was already transitive via
`@supabase/mcp-server-supabase`.

### Verification
- `pnpm -C apps/studio exec -- tsc --noEmit` reaches the compiler; it
reports only the pre-existing unrelated
`packages/ui-patterns/src/McpUrlBuilder/components/InstructionBlocks.tsx`
`StaticImageData` error.
- `pnpm -C apps/studio exec -- vitest run
lib/ai/tools/mock-tools.test.ts lib/ai/tools/mcp-tools.test.ts` — 21/21
passed.
- `pnpm exec tsx evals/preflight.ts` — live docs API schema fetch and
search_docs call passed.
- `NEXT_PUBLIC_CONTENT_API_URL=http://127.0.0.1:1/graphql pnpm -C
apps/studio exec -- tsx evals/preflight.ts` — failed fast as expected,
proving schema/API failures gate evals.
- Fresh `run-evals` pass: Docs Faithfulness 55.7% (0pp), with no
systemic `search_docs` regression.

Risk: eval-harness-only; schema/API outage now fails the eval job before
scoring rather than allowing fallback descriptions.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added documentation search powered by the public Supabase
documentation GraphQL API.
* Documentation search results now include live schema information and
clearer error handling for failed or invalid requests.

* **Bug Fixes**
* Improved evaluation tooling reliability by removing unnecessary
connection-abort behavior.
* Updated validation to detect missing search tools and malformed
documentation responses.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-14 12:51:14 +02:00

124 lines
5.0 KiB
TypeScript

// Type-only import (erased at build time — pulls no runtime code into this route).
import type * as SupabaseMcp from '@supabase/mcp-server-supabase'
import type { ToolSet } from 'ai'
import { createSupabaseMCPClient } from '../supabase-mcp'
import { filterToolsByOptInLevel, toolSetValidationSchema } from '../tool-filter'
import type { AiOptInLevel } from '@/hooks/misc/useOrgOptedIntoAi'
/**
* Union of the tool names exposed by the pinned `@supabase/mcp-server-supabase`
* version. Studio's dependency is bumped in lockstep with the remote MCP server
* (via automated bump PRs), so typing our tool-name lists against this makes an
* upstream rename/removal a **compile-time** failure (`pnpm typecheck`) in that
* PR, instead of a silent capability loss at runtime.
*/
type SupabaseMcpToolName = keyof typeof SupabaseMcp.supabaseMcpToolSchemas
// UI-executed tools handled locally by Studio (see getStudioTools); the remote
// MCP server's versions are removed so the UI-controlled Studio versions win.
const UI_EXECUTED_TOOLS = [
'execute_sql',
'deploy_edge_function',
] as const satisfies readonly SupabaseMcpToolName[]
// Read-only tools the assistant relies on from the remote MCP server — the
// MCP-sourced subset of the allowlist in tool-filter.ts (TOOL_CATEGORY_MAP).
// `satisfies` gives the compile-time drift guard; the runtime check below also
// catches a deployed server that returns fewer tools (feature flags / version
// skew). The allowlist remains the source of truth for what is allowed.
const EXPECTED_MCP_TOOLS = [
'search_docs',
'list_tables',
'list_extensions',
'list_edge_functions',
'list_branches',
'get_advisors',
'query_logs',
] as const satisfies readonly SupabaseMcpToolName[]
export const getMcpTools = async ({
accessToken,
projectRef,
aiOptInLevel,
signal,
}: {
accessToken: string
projectRef: string
aiOptInLevel: AiOptInLevel
// Required: the remote client holds an HTTP connection that must be torn down
// when the request ends. The caller owns that lifecycle via this signal.
signal: AbortSignal
}) => {
// Connect to the remote MCP server over HTTP and fetch its tools, replacing
// the local tools. A remote failure (outage, timeout, auth) degrades to the
// remaining tools in `getTools` rather than breaking the assistant.
const mcpClient = await createSupabaseMCPClient({
accessToken,
projectRef,
})
// The remote client keeps an HTTP connection open. The tools' `execute`
// functions are invoked later, while the response is streaming, so the
// connection must stay open until the request ends. Close it exactly once when
// the request is done (normal completion or abort) to avoid leaking a
// connection per request.
let closed = false
const closeClient = () => {
if (closed) return
closed = true
void mcpClient.close().catch(() => {})
}
// The request already ended before we could fetch tools; don't bother.
if (signal.aborted) {
closeClient()
return {} as ToolSet
}
signal.addEventListener('abort', closeClient, { once: true })
try {
const availableMcpTools = (await mcpClient.tools()) as ToolSet
// Runtime drift detection: `EXPECTED_MCP_TOOLS` is compile-time-checked
// against the pinned package (see its declaration), but the *deployed* remote
// server can still return fewer tools than the pinned types — feature flags,
// killswitches, or version skew during a bump. `filterToolsByOptInLevel`
// drops missing tools silently, so warn to make that observable.
const missingExpectedTools = EXPECTED_MCP_TOOLS.filter((name) => !(name in availableMcpTools))
if (missingExpectedTools.length > 0) {
console.error(
`Remote MCP server is missing expected tools: ${missingExpectedTools.join(', ')}. ` +
'The tool contract may have drifted; the assistant will operate without them.'
)
}
// Safety gate: `filterToolsByOptInLevel` keeps only tools in the allowlist
// (tool-filter.ts TOOL_CATEGORY_MAP) and drops everything else. This — not
// the `read_only` query param — is what prevents the remote server's
// write/destructive tools (apply_migration, create_branch, ...) from reaching
// the assistant. `read_only` is defense-in-depth (those tools throw at
// runtime). Do not remove this filter on the assumption `read_only` suffices.
const allowedMcpTools = filterToolsByOptInLevel(availableMcpTools, aiOptInLevel)
// Remove UI-executed tools handled locally
const filteredMcpTools: ToolSet = { ...allowedMcpTools }
UI_EXECUTED_TOOLS.forEach((toolName) => {
delete filteredMcpTools[toolName]
})
// Validate that only known tools are provided
const validation = toolSetValidationSchema.safeParse(filteredMcpTools)
if (!validation.success) {
console.error('MCP tools validation error:', validation.error)
throw new Error('Internal error: MCP tools validation failed')
}
return validation.data
} catch (error) {
// Don't leak the connection if fetching or validating tools fails
closeClient()
throw error
}
}