mirror of
https://github.com/supabase/supabase.git
synced 2026-10-05 09:25:06 +03:00
- Eval harness's only live tool, `search_docs`, no longer needs the in-process MCP client or its dummy token — it now calls the public docs GraphQL API (`https://supabase.com/docs/api/graphql`) directly. Low risk as this is an eval-harness change only. Production assistant path (`mcp-tools.ts`) untouched. **Update:** per [@mattrossman's review](https://github.com/supabase/supabase/pull/50092#discussion_r3980396341), the eval tool's description embeds the Content API's own GraphQL schema (fetched via a `{ schema }` query and minified with `gqlmin`), mirroring how `@supabase/mcp-server-supabase`'s `docs-tools.ts`/`loadSchema` populates production's `search_docs` description. Without it, the model had no schema to work from and issued malformed queries, which caused the 218 `search_docs` errors and the -25pp Docs Faithfulness regression in the first eval run on this PR. Schema loading is required: `createSearchDocsTool()` rejects if the schema fetch fails, so preflight and the gated eval job fail loudly instead of producing untrustworthy fallback results. `createSearchDocsTool` is async because the `ai` package's `tool()` only accepts a plain string `description`, unlike the MCP SDK's async description support; both callers (`getMockTools`, `evals/preflight.ts`) await it. `gqlmin` is a direct `apps/studio` dependency and was already transitive via `@supabase/mcp-server-supabase`. ### Verification - `pnpm -C apps/studio exec -- tsc --noEmit` reaches the compiler; it reports only the pre-existing unrelated `packages/ui-patterns/src/McpUrlBuilder/components/InstructionBlocks.tsx` `StaticImageData` error. - `pnpm -C apps/studio exec -- vitest run lib/ai/tools/mock-tools.test.ts lib/ai/tools/mcp-tools.test.ts` — 21/21 passed. - `pnpm exec tsx evals/preflight.ts` — live docs API schema fetch and search_docs call passed. - `NEXT_PUBLIC_CONTENT_API_URL=http://127.0.0.1:1/graphql pnpm -C apps/studio exec -- tsx evals/preflight.ts` — failed fast as expected, proving schema/API failures gate evals. - Fresh `run-evals` pass: Docs Faithfulness 55.7% (0pp), with no systemic `search_docs` regression. Risk: eval-harness-only; schema/API outage now fails the eval job before scoring rather than allowing fallback descriptions. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added documentation search powered by the public Supabase documentation GraphQL API. * Documentation search results now include live schema information and clearer error handling for failed or invalid requests. * **Bug Fixes** * Improved evaluation tooling reliability by removing unnecessary connection-abort behavior. * Updated validation to detect missing search tools and malformed documentation responses. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
124 lines
5.0 KiB
TypeScript
124 lines
5.0 KiB
TypeScript
// Type-only import (erased at build time — pulls no runtime code into this route).
|
|
import type * as SupabaseMcp from '@supabase/mcp-server-supabase'
|
|
import type { ToolSet } from 'ai'
|
|
|
|
import { createSupabaseMCPClient } from '../supabase-mcp'
|
|
import { filterToolsByOptInLevel, toolSetValidationSchema } from '../tool-filter'
|
|
import type { AiOptInLevel } from '@/hooks/misc/useOrgOptedIntoAi'
|
|
|
|
/**
|
|
* Union of the tool names exposed by the pinned `@supabase/mcp-server-supabase`
|
|
* version. Studio's dependency is bumped in lockstep with the remote MCP server
|
|
* (via automated bump PRs), so typing our tool-name lists against this makes an
|
|
* upstream rename/removal a **compile-time** failure (`pnpm typecheck`) in that
|
|
* PR, instead of a silent capability loss at runtime.
|
|
*/
|
|
type SupabaseMcpToolName = keyof typeof SupabaseMcp.supabaseMcpToolSchemas
|
|
|
|
// UI-executed tools handled locally by Studio (see getStudioTools); the remote
|
|
// MCP server's versions are removed so the UI-controlled Studio versions win.
|
|
const UI_EXECUTED_TOOLS = [
|
|
'execute_sql',
|
|
'deploy_edge_function',
|
|
] as const satisfies readonly SupabaseMcpToolName[]
|
|
|
|
// Read-only tools the assistant relies on from the remote MCP server — the
|
|
// MCP-sourced subset of the allowlist in tool-filter.ts (TOOL_CATEGORY_MAP).
|
|
// `satisfies` gives the compile-time drift guard; the runtime check below also
|
|
// catches a deployed server that returns fewer tools (feature flags / version
|
|
// skew). The allowlist remains the source of truth for what is allowed.
|
|
const EXPECTED_MCP_TOOLS = [
|
|
'search_docs',
|
|
'list_tables',
|
|
'list_extensions',
|
|
'list_edge_functions',
|
|
'list_branches',
|
|
'get_advisors',
|
|
'query_logs',
|
|
] as const satisfies readonly SupabaseMcpToolName[]
|
|
|
|
export const getMcpTools = async ({
|
|
accessToken,
|
|
projectRef,
|
|
aiOptInLevel,
|
|
signal,
|
|
}: {
|
|
accessToken: string
|
|
projectRef: string
|
|
aiOptInLevel: AiOptInLevel
|
|
// Required: the remote client holds an HTTP connection that must be torn down
|
|
// when the request ends. The caller owns that lifecycle via this signal.
|
|
signal: AbortSignal
|
|
}) => {
|
|
// Connect to the remote MCP server over HTTP and fetch its tools, replacing
|
|
// the local tools. A remote failure (outage, timeout, auth) degrades to the
|
|
// remaining tools in `getTools` rather than breaking the assistant.
|
|
const mcpClient = await createSupabaseMCPClient({
|
|
accessToken,
|
|
projectRef,
|
|
})
|
|
|
|
// The remote client keeps an HTTP connection open. The tools' `execute`
|
|
// functions are invoked later, while the response is streaming, so the
|
|
// connection must stay open until the request ends. Close it exactly once when
|
|
// the request is done (normal completion or abort) to avoid leaking a
|
|
// connection per request.
|
|
let closed = false
|
|
const closeClient = () => {
|
|
if (closed) return
|
|
closed = true
|
|
void mcpClient.close().catch(() => {})
|
|
}
|
|
|
|
// The request already ended before we could fetch tools; don't bother.
|
|
if (signal.aborted) {
|
|
closeClient()
|
|
return {} as ToolSet
|
|
}
|
|
signal.addEventListener('abort', closeClient, { once: true })
|
|
|
|
try {
|
|
const availableMcpTools = (await mcpClient.tools()) as ToolSet
|
|
|
|
// Runtime drift detection: `EXPECTED_MCP_TOOLS` is compile-time-checked
|
|
// against the pinned package (see its declaration), but the *deployed* remote
|
|
// server can still return fewer tools than the pinned types — feature flags,
|
|
// killswitches, or version skew during a bump. `filterToolsByOptInLevel`
|
|
// drops missing tools silently, so warn to make that observable.
|
|
const missingExpectedTools = EXPECTED_MCP_TOOLS.filter((name) => !(name in availableMcpTools))
|
|
if (missingExpectedTools.length > 0) {
|
|
console.error(
|
|
`Remote MCP server is missing expected tools: ${missingExpectedTools.join(', ')}. ` +
|
|
'The tool contract may have drifted; the assistant will operate without them.'
|
|
)
|
|
}
|
|
|
|
// Safety gate: `filterToolsByOptInLevel` keeps only tools in the allowlist
|
|
// (tool-filter.ts TOOL_CATEGORY_MAP) and drops everything else. This — not
|
|
// the `read_only` query param — is what prevents the remote server's
|
|
// write/destructive tools (apply_migration, create_branch, ...) from reaching
|
|
// the assistant. `read_only` is defense-in-depth (those tools throw at
|
|
// runtime). Do not remove this filter on the assumption `read_only` suffices.
|
|
const allowedMcpTools = filterToolsByOptInLevel(availableMcpTools, aiOptInLevel)
|
|
|
|
// Remove UI-executed tools handled locally
|
|
const filteredMcpTools: ToolSet = { ...allowedMcpTools }
|
|
UI_EXECUTED_TOOLS.forEach((toolName) => {
|
|
delete filteredMcpTools[toolName]
|
|
})
|
|
|
|
// Validate that only known tools are provided
|
|
const validation = toolSetValidationSchema.safeParse(filteredMcpTools)
|
|
if (!validation.success) {
|
|
console.error('MCP tools validation error:', validation.error)
|
|
throw new Error('Internal error: MCP tools validation failed')
|
|
}
|
|
|
|
return validation.data
|
|
} catch (error) {
|
|
// Don't leak the connection if fetching or validating tools fails
|
|
closeClient()
|
|
throw error
|
|
}
|
|
}
|