mirror of
https://github.com/supabase/supabase.git
synced 2026-10-05 09:25:06 +03:00
Pedro RodriguesandClaude Sonnet 5
22d7bc0cfd
- Eval harness's only live tool, `search_docs`, no longer needs the in-process MCP client or its dummy token — it now calls the public docs GraphQL API (`https://supabase.com/docs/api/graphql`) directly. Low risk as this is an eval-harness change only. Production assistant path (`mcp-tools.ts`) untouched. **Update:** per [@mattrossman's review](https://github.com/supabase/supabase/pull/50092#discussion_r3980396341), the eval tool's description embeds the Content API's own GraphQL schema (fetched via a `{ schema }` query and minified with `gqlmin`), mirroring how `@supabase/mcp-server-supabase`'s `docs-tools.ts`/`loadSchema` populates production's `search_docs` description. Without it, the model had no schema to work from and issued malformed queries, which caused the 218 `search_docs` errors and the -25pp Docs Faithfulness regression in the first eval run on this PR. Schema loading is required: `createSearchDocsTool()` rejects if the schema fetch fails, so preflight and the gated eval job fail loudly instead of producing untrustworthy fallback results. `createSearchDocsTool` is async because the `ai` package's `tool()` only accepts a plain string `description`, unlike the MCP SDK's async description support; both callers (`getMockTools`, `evals/preflight.ts`) await it. `gqlmin` is a direct `apps/studio` dependency and was already transitive via `@supabase/mcp-server-supabase`. ### Verification - `pnpm -C apps/studio exec -- tsc --noEmit` reaches the compiler; it reports only the pre-existing unrelated `packages/ui-patterns/src/McpUrlBuilder/components/InstructionBlocks.tsx` `StaticImageData` error. - `pnpm -C apps/studio exec -- vitest run lib/ai/tools/mock-tools.test.ts lib/ai/tools/mcp-tools.test.ts` — 21/21 passed. - `pnpm exec tsx evals/preflight.ts` — live docs API schema fetch and search_docs call passed. - `NEXT_PUBLIC_CONTENT_API_URL=http://127.0.0.1:1/graphql pnpm -C apps/studio exec -- tsx evals/preflight.ts` — failed fast as expected, proving schema/API failures gate evals. - Fresh `run-evals` pass: Docs Faithfulness 55.7% (0pp), with no systemic `search_docs` regression. Risk: eval-harness-only; schema/API outage now fails the eval job before scoring rather than allowing fallback descriptions. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added documentation search powered by the public Supabase documentation GraphQL API. * Documentation search results now include live schema information and clearer error handling for failed or invalid requests. * **Bug Fixes** * Improved evaluation tooling reliability by removing unnecessary connection-abort behavior. * Updated validation to detect missing search tools and malformed documentation responses. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
82 lines
3.0 KiB
TypeScript
82 lines
3.0 KiB
TypeScript
import { createMCPClient } from '@ai-sdk/mcp'
|
|
|
|
/**
|
|
* Default MCP server URL used when `NEXT_PUBLIC_MCP_URL` is not configured (local
|
|
* development). Mirrors `DEFAULT_MCP_URL_PLATFORM` in `ui-patterns/McpUrlBuilder`
|
|
* so the assistant resolves the same endpoint as the Connect sheet. It's
|
|
* duplicated here (rather than imported) to keep
|
|
* `ui-patterns/McpUrlBuilder/constants` — which pulls in `next/image` and image
|
|
* assets — out of this server-side bundle.
|
|
*/
|
|
const DEFAULT_MCP_URL = 'http://localhost:8080/mcp'
|
|
|
|
/**
|
|
* Identifies assistant traffic to the remote MCP server. Sent both as the MCP
|
|
* client name (logged as `client_name`) and via the `x-source-name` header
|
|
* (logged as `source_name`) by the mgmt-api McpLogger, so assistant requests are
|
|
* attributable in the MCP server's logs.
|
|
*/
|
|
const SOURCE_NAME = 'supabase-studio'
|
|
|
|
/**
|
|
* Builds the remote MCP endpoint URL for the dashboard assistant.
|
|
*
|
|
* Points at the remote MCP server configured via `NEXT_PUBLIC_MCP_URL` (e.g.
|
|
* https://mcp.supabase.com/mcp), falling back to a local default for development.
|
|
* The query parameters (`project_ref`, `read_only`) mirror `getMcpUrl` in
|
|
* `ui-patterns/McpUrlBuilder/utils/getMcpUrl` so the assistant and the Connect sheet stay in
|
|
* sync. The assistant only performs read operations, so `read_only` is always
|
|
* set.
|
|
*
|
|
* Note: the assistant only talks to the remote MCP server on the hosted platform
|
|
* (see `getTools` / `getMcpTools`), so no self-hosted branch is needed here.
|
|
*/
|
|
function getRemoteMcpUrl(projectRef: string) {
|
|
// `||` (not `??`) so an empty-string env var falls back instead of producing
|
|
// an invalid `new URL('')`.
|
|
const url = new URL(process.env.NEXT_PUBLIC_MCP_URL || DEFAULT_MCP_URL)
|
|
if (projectRef) {
|
|
url.searchParams.set('project_ref', projectRef)
|
|
}
|
|
url.searchParams.set('read_only', 'true')
|
|
|
|
return url.toString()
|
|
}
|
|
|
|
/**
|
|
* Creates an MCP client connected to the remote Supabase MCP server over HTTP.
|
|
*
|
|
* Previously the assistant instantiated the MCP server in-process and connected
|
|
* to it via an in-memory transport. It now connects to the remote MCP server so
|
|
* the dashboard assistant shares the same MCP surface as external clients.
|
|
*
|
|
* The dashboard session `accessToken` is forwarded as a bearer token. The remote
|
|
* MCP server is responsible for validating it and scoping access to the project.
|
|
*/
|
|
export async function createSupabaseMCPClient({
|
|
accessToken,
|
|
projectRef,
|
|
}: {
|
|
accessToken: string
|
|
projectRef: string
|
|
}) {
|
|
// Identifies the deployed build in the MCP server's `source_version` log field.
|
|
const sourceVersion = process.env.VERCEL_GIT_COMMIT_SHA
|
|
|
|
const client = await createMCPClient({
|
|
name: SOURCE_NAME,
|
|
transport: {
|
|
type: 'http',
|
|
url: getRemoteMcpUrl(projectRef),
|
|
headers: {
|
|
Authorization: `Bearer ${accessToken}`,
|
|
// Identify assistant traffic in the remote MCP server's logs
|
|
'x-source-name': SOURCE_NAME,
|
|
...(sourceVersion ? { 'x-source-version': sourceVersion } : {}),
|
|
},
|
|
},
|
|
})
|
|
|
|
return client
|
|
}
|