feat(studio): switch dashboard assistant to remote MCP server (#47479)

## I have read the
[CONTRIBUTING.md](<https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md>)
file.

YES

## What kind of change does this PR introduce?

Feature / refactor.

## What is the current behavior?

The dashboard assistant runs `@supabase/mcp-server-supabase` in-process
over an in-memory transport (`lib/ai/supabase-mcp.ts`).

## What is the new behavior?

The assistant connects to the **remote MCP server** over HTTP
(`@ai-sdk/mcp`), forwarding the dashboard session token as a bearer. URL
comes from `NEXT_PUBLIC_MCP_URL` with a local-dev fallback;
platform-only, and Nimbus works via the same env var.

* **Tool model unchanged:** UI-controlled `execute_sql` (with
`needsApproval`) and `deploy_edge_function` still come from Studio; the
allowlist (`TOOL_CATEGORY_MAP`) remains the gate keeping the remote's
write tools away from the assistant (`read_only` is defense-in-depth).
* **Attribution:** sends `x-source-name: supabase-studio` (+
`x-source-version`) → logged as `source_name`/`client_name`.
* **Connection lifecycle:** the HTTP client is closed via the request's
`AbortSignal` (tools execute later during streaming); `signal` is
required on `getTools`/`getMcpTools`.
* **Resilience:** a remote-MCP failure degrades to the remaining tools
instead of failing the assistant.
* **Drift protection:** relied-upon tools are typed against `keyof
typeof supabaseMcpToolSchemas`, so a package bump that renames/removes
one fails `pnpm typecheck`; a runtime check also warns if the deployed
server returns fewer tools.
* Adds unit tests for the above.

## Additional context

* Verified end-to-end against a local remote MCP server with a dashboard
token: `initialize` 200, tools listed, a tool executed, client closed
cleanly.
* The remote MCP (mgmt-api) already accepts dashboard session tokens
(GoTrue-JWT auth path) — no backend change needed. `NEXT_PUBLIC_MCP_URL`
must point at each env's `/mcp`.
* `@supabase/mcp-server-supabase` is kept — still used by the
self-hosted `/api/mcp` routes.

Closes
[AI-137](https://linear.app/supabase/issue/AI-137/switch-dashboard-assistant-to-remote-mcp)

## Rollout

* **Rollout:** merges with `USE_REMOTE_MCP` off (in-process); flip it to
`true` per environment (staging → prod → Nimbus) once each one's
prerequisites land.
* **Rollback:** unset `USE_REMOTE_MCP` and redeploy to fall back to the
in-process client — no revert needed.

## Summary by CodeRabbit

* **Bug Fixes**
* Improved AI request handling so tool loading and generation clean up
properly when a request is cancelled or the browser connection closes.
* Added safer fallback behavior when remote tool loading fails, so AI
features can continue with available tools instead of stopping entirely.
* Updated remote tool access to use the current project reference and
preserve the correct access headers.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* AI tools now connect more reliably to remote services and stop cleanly
when requests end or are canceled.
* Tool loading is more resilient, continuing with available tools if
remote access is unavailable.

* **Bug Fixes**
* Improved cleanup to prevent lingering connections during SQL
generation and policy workflows.
  * Added safer handling for remote tool changes and invalid responses.

* **Tests**
* Expanded automated coverage for remote tool setup, cancellation, and
fallback behavior.


<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pedro RodriguesandClaude Opus 4.8 authored and GitHub committed 2026-07-07 19:38:21 +01:00
1 parent 5dc054ae8f
commit c4c213ce3d
16 files changed
+888 -95

No files matched your search

+36 -2
View File
@@ -15,8 +15,42 @@ permissions:
contents: read
jobs:
# Cheap, always-on gate: verifies the eval harness can reach the MCP server
# (its one real tool, search_docs). Runs on every push/PR — no OpenAI, no full
# eval suite — so a broken MCP connection is caught early with an actionable
# message. The eval job depends on this, so evals never run against a broken
# MCP connection.
preflight:
name: Eval MCP preflight
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Install pnpm
uses: pnpm/action-setup@41ff72655975bd51cab0327fa583b6e92b6d3061 # v4.2.0
with:
run_install: false
- name: Use Node.js
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version-file: '.nvmrc'
cache: 'pnpm'
- name: Install Dependencies
run: pnpm install --frozen-lockfile
- name: Preflight — MCP connectivity
run: cd apps/studio && pnpm evals:preflight
eval:
name: Run evals
needs: preflight
if: github.event_name == 'push' || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-evals') && github.event.pull_request.head.repo.full_name == github.repository)
runs-on: ubuntu-latest
timeout-minutes: 20
@@ -43,8 +77,8 @@ jobs:
- name: Use Node.js
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version-file: ".nvmrc"
cache: "pnpm"
node-version-file: '.nvmrc'
cache: 'pnpm'
- name: Install Dependencies
run: pnpm install --frozen-lockfile
+23 -13
View File
@@ -31,20 +31,30 @@ Eval('Assistant', {
const modelResponse = await getModel({ provider: 'openai', modelEntry })
if (modelResponse.error) throw modelResponse.error
const result = await generateAssistantResponse({
...modelResponse.modelParams,
messages: [
{
id: '1',
role: 'user',
parts: [{ type: 'text', text: input.prompt }],
},
],
tools: await getMockTools(input.mockTables ? { list_tables: input.mockTables } : undefined),
})
// Owns the lifecycle of the remote MCP client opened inside getMockTools:
// aborting once generation is done closes that connection.
const toolsAbortController = new AbortController()
try {
const result = await generateAssistantResponse({
...modelResponse.modelParams,
messages: [
{
id: '1',
role: 'user',
parts: [{ type: 'text', text: input.prompt }],
},
],
tools: await getMockTools(
input.mockTables ? { list_tables: input.mockTables } : undefined,
toolsAbortController.signal
),
})
const finishReason = await result.finishReason
return { finishReason }
const finishReason = await result.finishReason
return { finishReason }
} finally {
toolsAbortController.abort()
}
},
scores: [
toolUsageScorer,
+49
View File
@@ -0,0 +1,49 @@
/**
* Eval preflight — MCP connectivity check.
*
* The assistant eval harness (`getMockTools`) mocks every tool except
* `search_docs`, which it sources from a real MCP server. If that connection is
* broken (endpoint down, bad/expired token, contract drift, missing package),
* evals fail deep inside a Braintrust run with an opaque per-case error.
*
* This preflight exercises the exact same path and fails fast with an
* actionable message, so a broken MCP connection is caught up front when the
* eval job runs (e.g. on push). Keep it in lockstep with how `getMockTools`
* obtains `search_docs` — if that switches to the remote client (see AI-897),
* switch this too.
*/
import { createInProcessSupabaseMCPClient } from '@/lib/ai/supabase-mcp'
async function runPreflight() {
let client: Awaited<ReturnType<typeof createInProcessSupabaseMCPClient>> | undefined
try {
client = await createInProcessSupabaseMCPClient({
accessToken: 'mock-access-token',
projectRef: 'mock-project-ref',
})
const tools = await client.tools()
if (!tools || !('search_docs' in tools)) {
throw new Error(
'Connected to the MCP server but `search_docs` was not returned. ' +
'The tool contract may have drifted, or the server is misconfigured.'
)
}
console.log('✅ Eval MCP preflight OK — connected and `search_docs` is available.')
} finally {
await client?.close().catch(() => {})
}
}
runPreflight().catch((error) => {
console.error(
'❌ Eval MCP preflight failed — the eval harness cannot reach the MCP server, ' +
'so evals would fail. Check NEXT_PUBLIC_MCP_URL, the access token, and the ' +
'@supabase/mcp-server-supabase dependency.'
)
console.error(error instanceof Error ? error.message : error)
process.exit(1)
})
+139
View File
@@ -0,0 +1,139 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { createSupabaseMCPClient } from './supabase-mcp'
const createMCPClientMock = vi.fn()
vi.mock('@ai-sdk/mcp', () => ({
createMCPClient: (...args: any[]) => createMCPClientMock(...args),
}))
const ACCESS_TOKEN = 'test-access-token'
const PROJECT_REF = 'abcdefghijklmnopqrst'
function getTransportConfig() {
expect(createMCPClientMock).toHaveBeenCalledTimes(1)
return createMCPClientMock.mock.calls[0][0]
}
describe('createSupabaseMCPClient', () => {
const originalMcpUrl = process.env.NEXT_PUBLIC_MCP_URL
const originalSha = process.env.VERCEL_GIT_COMMIT_SHA
beforeEach(() => {
vi.clearAllMocks()
createMCPClientMock.mockResolvedValue({ tools: vi.fn() })
delete process.env.VERCEL_GIT_COMMIT_SHA
})
afterEach(() => {
if (originalMcpUrl === undefined) delete process.env.NEXT_PUBLIC_MCP_URL
else process.env.NEXT_PUBLIC_MCP_URL = originalMcpUrl
if (originalSha === undefined) delete process.env.VERCEL_GIT_COMMIT_SHA
else process.env.VERCEL_GIT_COMMIT_SHA = originalSha
})
it('connects over the HTTP transport with the supabase-studio client name', async () => {
process.env.NEXT_PUBLIC_MCP_URL = 'https://mcp.supabase.com/mcp'
await createSupabaseMCPClient({ accessToken: ACCESS_TOKEN, projectRef: PROJECT_REF })
const config = getTransportConfig()
expect(config.name).toBe('supabase-studio')
expect(config.transport.type).toBe('http')
})
it('targets the URL from NEXT_PUBLIC_MCP_URL with project_ref and read_only', async () => {
process.env.NEXT_PUBLIC_MCP_URL = 'https://mcp.supabase.com/mcp'
await createSupabaseMCPClient({ accessToken: ACCESS_TOKEN, projectRef: PROJECT_REF })
const url = new URL(getTransportConfig().transport.url)
expect(url.origin + url.pathname).toBe('https://mcp.supabase.com/mcp')
expect(url.searchParams.get('project_ref')).toBe(PROJECT_REF)
expect(url.searchParams.get('read_only')).toBe('true')
})
it('forwards the dashboard access token as a bearer header', async () => {
process.env.NEXT_PUBLIC_MCP_URL = 'https://mcp.supabase.com/mcp'
await createSupabaseMCPClient({ accessToken: ACCESS_TOKEN, projectRef: PROJECT_REF })
expect(getTransportConfig().transport.headers.Authorization).toBe(`Bearer ${ACCESS_TOKEN}`)
})
it('identifies assistant traffic with the x-source-name header', async () => {
process.env.NEXT_PUBLIC_MCP_URL = 'https://mcp.supabase.com/mcp'
await createSupabaseMCPClient({ accessToken: ACCESS_TOKEN, projectRef: PROJECT_REF })
expect(getTransportConfig().transport.headers['x-source-name']).toBe('supabase-studio')
})
it('sends x-source-version from VERCEL_GIT_COMMIT_SHA when available', async () => {
process.env.NEXT_PUBLIC_MCP_URL = 'https://mcp.supabase.com/mcp'
process.env.VERCEL_GIT_COMMIT_SHA = 'abc1234'
await createSupabaseMCPClient({ accessToken: ACCESS_TOKEN, projectRef: PROJECT_REF })
expect(getTransportConfig().transport.headers['x-source-version']).toBe('abc1234')
})
it('omits x-source-version when VERCEL_GIT_COMMIT_SHA is not set', async () => {
process.env.NEXT_PUBLIC_MCP_URL = 'https://mcp.supabase.com/mcp'
delete process.env.VERCEL_GIT_COMMIT_SHA
await createSupabaseMCPClient({ accessToken: ACCESS_TOKEN, projectRef: PROJECT_REF })
expect(getTransportConfig().transport.headers).not.toHaveProperty('x-source-version')
})
it('never leaks the access token into the URL', async () => {
process.env.NEXT_PUBLIC_MCP_URL = 'https://mcp.supabase.com/mcp'
await createSupabaseMCPClient({ accessToken: ACCESS_TOKEN, projectRef: PROJECT_REF })
expect(getTransportConfig().transport.url).not.toContain(ACCESS_TOKEN)
})
it('always requests read-only mode', async () => {
process.env.NEXT_PUBLIC_MCP_URL = 'https://mcp.supabase.com/mcp'
await createSupabaseMCPClient({ accessToken: ACCESS_TOKEN, projectRef: PROJECT_REF })
const url = new URL(getTransportConfig().transport.url)
expect(url.searchParams.get('read_only')).toBe('true')
})
it('falls back to the default local URL when NEXT_PUBLIC_MCP_URL is unset', async () => {
delete process.env.NEXT_PUBLIC_MCP_URL
await createSupabaseMCPClient({ accessToken: ACCESS_TOKEN, projectRef: PROJECT_REF })
const url = new URL(getTransportConfig().transport.url)
expect(url.origin + url.pathname).toBe('http://localhost:8080/mcp')
expect(url.searchParams.get('project_ref')).toBe(PROJECT_REF)
})
it('falls back to the default local URL when NEXT_PUBLIC_MCP_URL is an empty string', async () => {
process.env.NEXT_PUBLIC_MCP_URL = ''
await createSupabaseMCPClient({ accessToken: ACCESS_TOKEN, projectRef: PROJECT_REF })
const url = new URL(getTransportConfig().transport.url)
expect(url.origin + url.pathname).toBe('http://localhost:8080/mcp')
})
it('returns the client created by createMCPClient', async () => {
process.env.NEXT_PUBLIC_MCP_URL = 'https://mcp.supabase.com/mcp'
const fakeClient = { tools: vi.fn() }
createMCPClientMock.mockResolvedValueOnce(fakeClient)
const client = await createSupabaseMCPClient({
accessToken: ACCESS_TOKEN,
projectRef: PROJECT_REF,
})
expect(client).toBe(fakeClient)
})
})
+104 -9
View File
@@ -1,17 +1,113 @@
import { createMCPClient } from '@ai-sdk/mcp'
import { InMemoryTransport } from '@modelcontextprotocol/sdk/inMemory.js' // .js required for esbuild ESM resolution
import { createSupabaseMcpServer } from '@supabase/mcp-server-supabase'
import { createSupabaseApiPlatform } from '@supabase/mcp-server-supabase/platform/api'
import { API_URL } from '@/lib/constants'
/**
* Default MCP server URL used when `NEXT_PUBLIC_MCP_URL` is not configured (local
* development). Mirrors `DEFAULT_MCP_URL_PLATFORM` in `ui-patterns/McpUrlBuilder`
* so the assistant resolves the same endpoint as the Connect sheet. It's
* duplicated here (rather than imported) to keep
* `ui-patterns/McpUrlBuilder/constants` — which pulls in `next/image` and image
* assets — out of this server-side bundle.
*/
const DEFAULT_MCP_URL = 'http://localhost:8080/mcp'
/**
* Identifies assistant traffic to the remote MCP server. Sent both as the MCP
* client name (logged as `client_name`) and via the `x-source-name` header
* (logged as `source_name`) by the mgmt-api McpLogger, so assistant requests are
* attributable in the MCP server's logs.
*/
const SOURCE_NAME = 'supabase-studio'
/**
* Builds the remote MCP endpoint URL for the dashboard assistant.
*
* Points at the remote MCP server configured via `NEXT_PUBLIC_MCP_URL` (e.g.
* https://mcp.supabase.com/mcp), falling back to a local default for development.
* The query parameters (`project_ref`, `read_only`) mirror `getMcpUrl` in
* `ui-patterns/McpUrlBuilder/utils/getMcpUrl` so the assistant and the Connect sheet stay in
* sync. The assistant only performs read operations, so `read_only` is always
* set.
*
* Note: the assistant only talks to the remote MCP server on the hosted platform
* (see `getTools` / `getMcpTools`), so no self-hosted branch is needed here.
*/
function getRemoteMcpUrl(projectRef: string) {
// `||` (not `??`) so an empty-string env var falls back instead of producing
// an invalid `new URL('')`.
const url = new URL(process.env.NEXT_PUBLIC_MCP_URL || DEFAULT_MCP_URL)
if (projectRef) {
url.searchParams.set('project_ref', projectRef)
}
url.searchParams.set('read_only', 'true')
return url.toString()
}
/**
* Creates an MCP client connected to the remote Supabase MCP server over HTTP.
*
* Previously the assistant instantiated the MCP server in-process and connected
* to it via an in-memory transport. It now connects to the remote MCP server so
* the dashboard assistant shares the same MCP surface as external clients.
*
* The dashboard session `accessToken` is forwarded as a bearer token. The remote
* MCP server is responsible for validating it and scoping access to the project.
*/
export async function createSupabaseMCPClient({
accessToken,
projectId,
projectRef,
}: {
accessToken: string
projectId: string
projectRef: string
}) {
// Identifies the deployed build in the MCP server's `source_version` log field.
const sourceVersion = process.env.VERCEL_GIT_COMMIT_SHA
const client = await createMCPClient({
name: SOURCE_NAME,
transport: {
type: 'http',
url: getRemoteMcpUrl(projectRef),
headers: {
Authorization: `Bearer ${accessToken}`,
// Identify assistant traffic in the remote MCP server's logs
'x-source-name': SOURCE_NAME,
...(sourceVersion ? { 'x-source-version': sourceVersion } : {}),
},
},
})
return client
}
/**
* Legacy in-process MCP client — the pre-migration behavior, kept as a fallback
* behind the `USE_REMOTE_MCP` gate (see `tools/mcp-tools.ts`).
*
* Instantiates `@supabase/mcp-server-supabase` in-process and connects to it over
* an in-memory transport. The heavy server package is imported dynamically so it
* is code-split into its own chunk and stays out of the (default, post-migration)
* remote path's bundle.
*
* TODO(AI-897): remove in process mcp — delete this once every environment has
* been flipped to the remote MCP server and has been stable. Tracked alongside
* the `USE_REMOTE_MCP` rollout.
*/
export async function createInProcessSupabaseMCPClient({
accessToken,
projectRef,
}: {
accessToken: string
projectRef: string
}) {
// Dynamic imports keep the in-process server + its transport out of the remote
// path's bundle (loaded only when this fallback is actually taken).
// `.js` is required for esbuild ESM resolution.
const { InMemoryTransport } = await import('@modelcontextprotocol/sdk/inMemory.js')
const { createSupabaseMcpServer } = await import('@supabase/mcp-server-supabase')
const { createSupabaseApiPlatform } = await import('@supabase/mcp-server-supabase/platform/api')
const { API_URL } = await import('@/lib/constants')
const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair()
// Instantiate the MCP server and connect to its transport
@@ -22,14 +118,13 @@ export async function createSupabaseMCPClient({
apiUrl,
}),
contentApiUrl: process.env.NEXT_PUBLIC_CONTENT_API_URL,
projectId,
projectId: projectRef,
readOnly: true,
})
await server.connect(serverTransport)
// Create the MCP client and connect to its transport
const client = await createMCPClient({
name: 'supabase-studio',
name: SOURCE_NAME,
transport: clientTransport,
})
+86
View File
@@ -0,0 +1,86 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { getTools } from './index'
import { getMcpTools } from './mcp-tools'
vi.mock('common', () => ({ IS_PLATFORM: true }))
vi.mock('./mcp-tools', () => ({ getMcpTools: vi.fn() }))
vi.mock('./studio-tools', () => ({ getStudioTools: vi.fn(() => ({ studio_tool: {} })) }))
vi.mock('./schema-tools', () => ({ getSchemaTools: vi.fn(() => ({ schema_tool: {} })) }))
vi.mock('./incident-tools', () => ({ getIncidentTools: vi.fn(() => ({ incident_tool: {} })) }))
vi.mock('./fallback-tools', () => ({ getFallbackTools: vi.fn(() => ({ fallback_tool: {} })) }))
// Identity filter so assertions can check the raw merged tool set
vi.mock('../tool-filter', () => ({ filterToolsByOptInLevel: vi.fn((tools) => tools) }))
const BASE_PARAMS = {
projectRef: 'abcdefghijklmnopqrst',
connectionString: 'postgresql://localhost',
authorization: 'Bearer token',
aiOptInLevel: 'schema_and_log_and_data' as const,
accessToken: 'access-token',
baseUrl: 'https://supabase.com/dashboard',
signal: new AbortController().signal,
}
describe('ai/tools getTools', () => {
beforeEach(async () => {
vi.clearAllMocks()
vi.mocked(getMcpTools).mockResolvedValue({ list_tables: {} } as any)
// Reset to platform each test; the self-hosted test overrides to false.
// Done here (not afterEach) so the spy can't leak across tests via order.
const common = await import('common')
vi.spyOn(common, 'IS_PLATFORM', 'get').mockReturnValue(true)
})
it('includes studio, MCP, schema and incident tools on platform', async () => {
const tools = await getTools(BASE_PARAMS)
expect(getMcpTools).toHaveBeenCalledWith({
accessToken: BASE_PARAMS.accessToken,
projectRef: BASE_PARAMS.projectRef,
aiOptInLevel: BASE_PARAMS.aiOptInLevel,
signal: BASE_PARAMS.signal,
})
expect(tools).toHaveProperty('studio_tool')
expect(tools).toHaveProperty('list_tables')
expect(tools).toHaveProperty('schema_tool')
expect(tools).toHaveProperty('incident_tool')
})
it('degrades gracefully to the remaining tools when remote MCP fetch fails', async () => {
const consoleSpy = vi.spyOn(console, 'error').mockImplementation(() => {})
vi.mocked(getMcpTools).mockRejectedValueOnce(new Error('remote MCP unreachable'))
const tools = await getTools(BASE_PARAMS)
// The assistant still works with the non-MCP tools instead of throwing
expect(tools).toHaveProperty('studio_tool')
expect(tools).toHaveProperty('schema_tool')
expect(tools).toHaveProperty('incident_tool')
expect(tools).not.toHaveProperty('list_tables')
expect(consoleSpy).toHaveBeenCalled()
consoleSpy.mockRestore()
})
it('does not fetch MCP tools when no access token is provided', async () => {
const tools = await getTools({ ...BASE_PARAMS, accessToken: undefined })
expect(getMcpTools).not.toHaveBeenCalled()
expect(tools).toHaveProperty('studio_tool')
expect(tools).not.toHaveProperty('list_tables')
})
it('uses fallback tools and skips MCP when self-hosted', async () => {
const common = await import('common')
vi.spyOn(common, 'IS_PLATFORM', 'get').mockReturnValue(false)
const tools = await getTools(BASE_PARAMS)
expect(getMcpTools).not.toHaveBeenCalled()
expect(tools).toHaveProperty('studio_tool')
expect(tools).toHaveProperty('fallback_tool')
expect(tools).not.toHaveProperty('list_tables')
})
})
+20 -6
View File
@@ -16,6 +16,7 @@ export const getTools = async ({
aiOptInLevel,
accessToken,
baseUrl,
signal,
}: {
projectRef: string
connectionString: string
@@ -23,6 +24,9 @@ export const getTools = async ({
aiOptInLevel: AiOptInLevel
accessToken?: string
baseUrl?: string
// Required: tools fetched from the remote MCP server hold an HTTP connection
// that is closed when this signal aborts (i.e. when the request ends).
signal: AbortSignal
}) => {
// Always include studio tools
let tools: ToolSet = getStudioTools({ projectRef, connectionString, authorization, aiOptInLevel })
@@ -39,12 +43,22 @@ export const getTools = async ({
}),
}
} else if (accessToken) {
// If platform, fetch MCP and other platform specific tools
const mcpTools = await getMcpTools({
accessToken,
projectRef,
aiOptInLevel,
})
// If platform, fetch MCP and other platform specific tools. The MCP tools
// may be fetched from the remote MCP server over the network (see
// `USE_REMOTE_MCP`), so a failure there (outage, timeout, auth) should
// degrade gracefully to the remaining tools rather than break the entire
// assistant.
let mcpTools: ToolSet = {}
try {
mcpTools = await getMcpTools({
accessToken,
projectRef,
aiOptInLevel,
signal,
})
} catch (error) {
console.error('Failed to fetch MCP tools:', error)
}
tools = {
...tools,
+172
View File
@@ -0,0 +1,172 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { createInProcessSupabaseMCPClient, createSupabaseMCPClient } from '../supabase-mcp'
import { getMcpTools } from './mcp-tools'
vi.mock('../supabase-mcp', () => ({
createSupabaseMCPClient: vi.fn(),
createInProcessSupabaseMCPClient: vi.fn(),
}))
const BASE_PARAMS = {
accessToken: 'token',
projectRef: 'abcdefghijklmnopqrst',
aiOptInLevel: 'schema_and_log_and_data' as const,
// A fresh, non-aborted signal by default; lifecycle tests override it
signal: new AbortController().signal,
}
// A realistic remote tool set: all expected read tools plus the UI-executed ones
const FULL_REMOTE_TOOLS = {
search_docs: { description: 'docs' },
list_tables: { description: 'list tables' },
list_extensions: { description: 'extensions' },
list_edge_functions: { description: 'edge functions' },
list_branches: { description: 'branches' },
get_advisors: { description: 'advisors' },
get_logs: { description: 'get logs' },
execute_sql: { description: 'execute sql' },
deploy_edge_function: { description: 'deploy' },
}
describe('ai/tools/mcp-tools getMcpTools', () => {
let close: ReturnType<typeof vi.fn>
let tools: ReturnType<typeof vi.fn>
let consoleErrorSpy: ReturnType<typeof vi.spyOn>
beforeEach(() => {
vi.clearAllMocks()
// These tests exercise the remote transport; pin the migration gate to it.
process.env.USE_REMOTE_MCP = 'true'
consoleErrorSpy = vi.spyOn(console, 'error').mockImplementation(() => {})
close = vi.fn().mockResolvedValue(undefined)
tools = vi.fn().mockResolvedValue({ ...FULL_REMOTE_TOOLS })
vi.mocked(createSupabaseMCPClient).mockResolvedValue({ tools, close } as any)
})
afterEach(() => {
delete process.env.USE_REMOTE_MCP
consoleErrorSpy.mockRestore()
})
it('returns MCP tools and strips UI-executed tools handled locally', async () => {
const result = await getMcpTools(BASE_PARAMS)
expect(result).toHaveProperty('list_tables')
expect(result).toHaveProperty('get_logs')
expect(result).not.toHaveProperty('execute_sql')
expect(result).not.toHaveProperty('deploy_edge_function')
})
it('warns when the remote server is missing an expected tool (contract drift)', async () => {
const { list_branches, ...withoutBranches } = FULL_REMOTE_TOOLS
tools.mockResolvedValueOnce(withoutBranches)
await getMcpTools(BASE_PARAMS)
expect(consoleErrorSpy).toHaveBeenCalledWith(expect.stringContaining('list_branches'))
})
it('does not warn about drift when all expected tools are present', async () => {
await getMcpTools(BASE_PARAMS)
const driftWarnings = consoleErrorSpy.mock.calls.filter(
([msg]: unknown[]) => typeof msg === 'string' && msg.includes('missing expected tools')
)
expect(driftWarnings).toHaveLength(0)
})
it('keeps the connection open until the request signal aborts', async () => {
const controller = new AbortController()
await getMcpTools({ ...BASE_PARAMS, signal: controller.signal })
// Tools execute later during streaming, so the client must stay open
expect(close).not.toHaveBeenCalled()
controller.abort()
await Promise.resolve()
expect(close).toHaveBeenCalledTimes(1)
})
it('closes the client and skips fetching tools when the signal is already aborted', async () => {
const controller = new AbortController()
controller.abort()
const result = await getMcpTools({ ...BASE_PARAMS, signal: controller.signal })
expect(close).toHaveBeenCalledTimes(1)
expect(tools).not.toHaveBeenCalled()
expect(result).toEqual({})
})
it('closes the client and rethrows when fetching tools fails, without double-closing on later abort', async () => {
tools.mockRejectedValueOnce(new Error('network unreachable'))
const controller = new AbortController()
await expect(getMcpTools({ ...BASE_PARAMS, signal: controller.signal })).rejects.toThrow(
'network unreachable'
)
expect(close).toHaveBeenCalledTimes(1)
// A subsequent abort must not close the (already closed) client again
controller.abort()
await Promise.resolve()
expect(close).toHaveBeenCalledTimes(1)
})
it('closes the client and rethrows when tool validation fails', async () => {
// A known tool name with a non-object value passes the opt-in filter but
// fails schema validation
tools.mockResolvedValueOnce({ ...FULL_REMOTE_TOOLS, list_tables: 'not-an-object' })
await expect(getMcpTools(BASE_PARAMS)).rejects.toThrow('MCP tools validation failed')
expect(close).toHaveBeenCalledTimes(1)
})
})
describe('ai/tools/mcp-tools getMcpTools transport selection', () => {
let close: ReturnType<typeof vi.fn>
let tools: ReturnType<typeof vi.fn>
let consoleErrorSpy: ReturnType<typeof vi.spyOn>
beforeEach(() => {
vi.clearAllMocks()
consoleErrorSpy = vi.spyOn(console, 'error').mockImplementation(() => {})
close = vi.fn().mockResolvedValue(undefined)
tools = vi.fn().mockResolvedValue({ ...FULL_REMOTE_TOOLS })
vi.mocked(createSupabaseMCPClient).mockResolvedValue({ tools, close } as any)
vi.mocked(createInProcessSupabaseMCPClient).mockResolvedValue({ tools, close } as any)
})
afterEach(() => {
delete process.env.USE_REMOTE_MCP
consoleErrorSpy.mockRestore()
})
it('uses the remote client when USE_REMOTE_MCP is "true"', async () => {
process.env.USE_REMOTE_MCP = 'true'
await getMcpTools(BASE_PARAMS)
expect(createSupabaseMCPClient).toHaveBeenCalledTimes(1)
expect(createInProcessSupabaseMCPClient).not.toHaveBeenCalled()
})
it('falls back to the in-process client when USE_REMOTE_MCP is unset (default)', async () => {
delete process.env.USE_REMOTE_MCP
await getMcpTools(BASE_PARAMS)
expect(createInProcessSupabaseMCPClient).toHaveBeenCalledTimes(1)
expect(createSupabaseMCPClient).not.toHaveBeenCalled()
})
it('uses the in-process client for any non-"true" value', async () => {
process.env.USE_REMOTE_MCP = 'false'
await getMcpTools(BASE_PARAMS)
expect(createInProcessSupabaseMCPClient).toHaveBeenCalledTimes(1)
expect(createSupabaseMCPClient).not.toHaveBeenCalled()
})
})
+114 -21
View File
@@ -1,42 +1,135 @@
// Type-only import (erased at build time — pulls no runtime code into this route).
import type * as SupabaseMcp from '@supabase/mcp-server-supabase'
import type { ToolSet } from 'ai'
import { createSupabaseMCPClient } from '../supabase-mcp'
import { createInProcessSupabaseMCPClient, createSupabaseMCPClient } from '../supabase-mcp'
import { filterToolsByOptInLevel, toolSetValidationSchema } from '../tool-filter'
import type { AiOptInLevel } from '@/hooks/misc/useOrgOptedIntoAi'
const UI_EXECUTED_TOOLS = ['execute_sql', 'deploy_edge_function']
/**
* Union of the tool names exposed by the pinned `@supabase/mcp-server-supabase`
* version. Studio's dependency is bumped in lockstep with the remote MCP server
* (via automated bump PRs), so typing our tool-name lists against this makes an
* upstream rename/removal a **compile-time** failure (`pnpm typecheck`) in that
* PR, instead of a silent capability loss at runtime.
*/
type SupabaseMcpToolName = keyof typeof SupabaseMcp.supabaseMcpToolSchemas
// UI-executed tools handled locally by Studio (see getStudioTools); the remote
// MCP server's versions are removed so the UI-controlled Studio versions win.
const UI_EXECUTED_TOOLS = [
'execute_sql',
'deploy_edge_function',
] as const satisfies readonly SupabaseMcpToolName[]
// Read-only tools the assistant relies on from the remote MCP server — the
// MCP-sourced subset of the allowlist in tool-filter.ts (TOOL_CATEGORY_MAP).
// `satisfies` gives the compile-time drift guard; the runtime check below also
// catches a deployed server that returns fewer tools (feature flags / version
// skew). The allowlist remains the source of truth for what is allowed.
const EXPECTED_MCP_TOOLS = [
'search_docs',
'list_tables',
'list_extensions',
'list_edge_functions',
'list_branches',
'get_advisors',
'get_logs',
] as const satisfies readonly SupabaseMcpToolName[]
export const getMcpTools = async ({
accessToken,
projectRef,
aiOptInLevel,
signal,
}: {
accessToken: string
projectRef: string
aiOptInLevel: AiOptInLevel
// Required: the remote client holds an HTTP connection that must be torn down
// when the request ends. The caller owns that lifecycle via this signal.
signal: AbortSignal
}) => {
// If platform, fetch MCP client and tools which replace old local tools
const mcpClient = await createSupabaseMCPClient({
// Connect to the MCP server and fetch its tools, which replace the old local
// tools. `USE_REMOTE_MCP` gates the transport: the remote HTTP server (target
// state) or the legacy in-process server (fallback during the migration),
// defaulting to in-process until an environment opts in. Flip it per
// environment (staging → prod → Nimbus) once each one's prerequisites are met
// (dashboard-token support on the MCP API, remote MCP enabled for Nimbus);
// unset to roll back on the next deploy. Both transports expose the same tool
// surface, so the filtering, drift detection, and lifecycle handling below are
// transport-agnostic.
//
// TODO(AI-897): remove in process mcp — once every environment has been
// flipped and is stable, delete `createInProcessSupabaseMCPClient` and this
// fallback branch.
const useRemoteMcp = process.env.USE_REMOTE_MCP === 'true'
const createClient = useRemoteMcp ? createSupabaseMCPClient : createInProcessSupabaseMCPClient
const mcpClient = await createClient({
accessToken,
projectId: projectRef,
projectRef,
})
const availableMcpTools = (await mcpClient.tools()) as ToolSet
// Filter tools based on the (potentially modified) AI opt-in level
const allowedMcpTools = filterToolsByOptInLevel(availableMcpTools, aiOptInLevel)
// Remove UI-executed tools handled locally
const filteredMcpTools: ToolSet = { ...allowedMcpTools }
UI_EXECUTED_TOOLS.forEach((toolName) => {
delete filteredMcpTools[toolName]
})
// Validate that only known tools are provided
const validation = toolSetValidationSchema.safeParse(filteredMcpTools)
if (!validation.success) {
console.error('MCP tools validation error:', validation.error)
throw new Error('Internal error: MCP tools validation failed')
// The remote client keeps an HTTP connection open. The tools' `execute`
// functions are invoked later, while the response is streaming, so the
// connection must stay open until the request ends. Close it exactly once when
// the request is done (normal completion or abort) to avoid leaking a
// connection per request.
let closed = false
const closeClient = () => {
if (closed) return
closed = true
void mcpClient.close().catch(() => {})
}
return validation.data
// The request already ended before we could fetch tools; don't bother.
if (signal.aborted) {
closeClient()
return {} as ToolSet
}
signal.addEventListener('abort', closeClient, { once: true })
try {
const availableMcpTools = (await mcpClient.tools()) as ToolSet
// Runtime drift detection: `EXPECTED_MCP_TOOLS` is compile-time-checked
// against the pinned package (see its declaration), but the *deployed* remote
// server can still return fewer tools than the pinned types — feature flags,
// killswitches, or version skew during a bump. `filterToolsByOptInLevel`
// drops missing tools silently, so warn to make that observable.
const missingExpectedTools = EXPECTED_MCP_TOOLS.filter((name) => !(name in availableMcpTools))
if (missingExpectedTools.length > 0) {
console.error(
`Remote MCP server is missing expected tools: ${missingExpectedTools.join(', ')}. ` +
'The tool contract may have drifted; the assistant will operate without them.'
)
}
// Safety gate: `filterToolsByOptInLevel` keeps only tools in the allowlist
// (tool-filter.ts TOOL_CATEGORY_MAP) and drops everything else. This — not
// the `read_only` query param — is what prevents the remote server's
// write/destructive tools (apply_migration, create_branch, ...) from reaching
// the assistant. `read_only` is defense-in-depth (those tools throw at
// runtime). Do not remove this filter on the assumption `read_only` suffices.
const allowedMcpTools = filterToolsByOptInLevel(availableMcpTools, aiOptInLevel)
// Remove UI-executed tools handled locally
const filteredMcpTools: ToolSet = { ...allowedMcpTools }
UI_EXECUTED_TOOLS.forEach((toolName) => {
delete filteredMcpTools[toolName]
})
// Validate that only known tools are provided
const validation = toolSetValidationSchema.safeParse(filteredMcpTools)
if (!validation.success) {
console.error('MCP tools validation error:', validation.error)
throw new Error('Internal error: MCP tools validation failed')
}
return validation.data
} catch (error) {
// Don't leak the connection if fetching or validating tools fails
closeClient()
throw error
}
}
@@ -0,0 +1,64 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { getMockTools } from './mock-tools'
import { createInProcessSupabaseMCPClient } from '@/lib/ai/supabase-mcp'
// The one real tool in the eval harness (search_docs) is sourced from an
// in-process MCP client. Mock that client so this test stays hermetic and
// guards the wiring, not a live connection.
vi.mock('@/lib/ai/supabase-mcp', () => ({
createInProcessSupabaseMCPClient: vi.fn(),
}))
const SEARCH_DOCS = { description: 'search the docs' }
describe('ai/tools/mock-tools getMockTools', () => {
let close: ReturnType<typeof vi.fn>
let tools: ReturnType<typeof vi.fn>
beforeEach(() => {
vi.clearAllMocks()
close = vi.fn().mockResolvedValue(undefined)
tools = vi.fn().mockResolvedValue({ search_docs: SEARCH_DOCS })
vi.mocked(createInProcessSupabaseMCPClient).mockResolvedValue({ tools, close } as any)
})
it('sources the real search_docs from the in-process MCP server alongside the deterministic mocks', async () => {
const result = await getMockTools(undefined, new AbortController().signal)
expect(createInProcessSupabaseMCPClient).toHaveBeenCalledTimes(1)
// The real tool, wired through from the MCP client
expect(result).toHaveProperty('search_docs', SEARCH_DOCS)
// A couple of the deterministic mocks, to confirm the merge
expect(result).toHaveProperty('list_tables')
expect(result).toHaveProperty('get_logs')
})
// This is the regression guard: if the eval's MCP wiring breaks (contract
// drift, or a refactor that stops sourcing search_docs — e.g. the future
// AI-897 removal of the in-process client), fail loudly in normal CI instead
// of only surfacing during an opt-in Braintrust eval run.
it('throws a clear error when the MCP server does not expose search_docs', async () => {
tools.mockResolvedValueOnce({})
await expect(getMockTools(undefined, new AbortController().signal)).rejects.toThrow(
'search_docs tool not available from MCP server'
)
})
it('closes the MCP client when the caller aborts the signal', async () => {
const controller = new AbortController()
await getMockTools(undefined, controller.signal)
// Connection stays open until generation ends (search_docs runs during it)
expect(close).not.toHaveBeenCalled()
controller.abort()
await Promise.resolve()
expect(close).toHaveBeenCalledTimes(1)
})
afterEach(() => {
vi.clearAllMocks()
})
})
+16 -5
View File
@@ -1,9 +1,9 @@
import assert from 'node:assert'
import { tool } from 'ai'
import { tool, type ToolSet } from 'ai'
import { z } from 'zod'
import { getStudioTools } from '../tools/studio-tools'
import { getMcpTools } from '@/lib/ai/tools/mcp-tools'
import { createInProcessSupabaseMCPClient } from '@/lib/ai/supabase-mcp'
const listTablesInputSchema = z.object({
schemas: z.array(z.string()).describe('The schema names to list.'),
@@ -306,14 +306,25 @@ export type MockToolOverrides = {
*
* Note: search_docs uses the real implementation
*/
export async function getMockTools(overrides?: MockToolOverrides) {
export async function getMockTools(overrides: MockToolOverrides | undefined, signal: AbortSignal) {
const mockedStudioTools = createMockedStudioTools()
const { search_docs } = await getMcpTools({
// Every tool here is a deterministic mock except `search_docs`, which uses the
// real implementation. We source it from an in-process MCP server directly
// (rather than `getMcpTools`) so the eval harness stays hermetic and decoupled
// from the assistant's transport gate (`USE_REMOTE_MCP`): the in-process server
// needs no live remote endpoint or real access token. See AI-897 for how to
// point evals at the remote MCP server instead.
const mcpClient = await createInProcessSupabaseMCPClient({
accessToken: 'mock-access-token',
projectRef: 'mock-project-ref',
aiOptInLevel: 'schema_and_log_and_data',
})
// The caller owns this signal and aborts it once generation is done, which
// closes the client opened here (search_docs executes during generation, so
// the connection must stay open until then).
signal.addEventListener('abort', () => void mcpClient.close().catch(() => {}), { once: true })
const { search_docs } = (await mcpClient.tools()) as ToolSet
assert(search_docs, 'search_docs tool not available from MCP server')
+4
View File
@@ -42,6 +42,7 @@ test('generateV4 calls the tool sanitizer', async () => {
status: vi.fn(() => mockRes),
json: vi.fn(() => mockRes),
setHeader: vi.fn(() => mockRes),
on: vi.fn(),
}
vi.mock('@/lib/ai/ai-details', () => ({
@@ -83,4 +84,7 @@ test('generateV4 calls the tool sanitizer', async () => {
await generateV4(mockReq as any, mockRes as any)
expect(sanitizeMessagePart).toHaveBeenCalled()
// The response 'close' event must be wired up so the remote MCP connection
// opened in getTools is torn down when the stream finishes or the client drops
expect(mockRes.on).toHaveBeenCalledWith('close', expect.any(Function))
})
+1
View File
@@ -33,6 +33,7 @@
"build:graphql-types": "tsx scripts/download-graphql-schema.mts && pnpm graphql-codegen --config scripts/codegen.ts",
"build:graphql-types:watch": "pnpm graphql-codegen --config scripts/codegen.ts --watch",
"evals:setup": "cp node_modules/libpg-query/wasm/libpg-query.wasm evals/libpg-query.wasm",
"evals:preflight": "tsx evals/preflight.ts",
"evals:run": "braintrust eval --no-send-logs evals/assistant.eval.ts",
"evals:upload": "braintrust eval evals/assistant.eval.ts",
"scorers:deploy": "IS_BRAINTRUST_PUSH=true braintrust push evals/scorer-online.ts"
@@ -165,6 +165,9 @@ async function handlePost(req: NextApiRequest, res: NextApiResponse, claims?: Jw
const abortController = new AbortController()
req.on('close', () => abortController.abort())
req.on('aborted', () => abortController.abort())
// Fires when the response finishes streaming or the connection drops, which
// is what tears down the remote MCP connection opened in getTools.
res.on('close', () => abortController.abort())
const tools = await getTools({
projectRef,
@@ -173,6 +176,7 @@ async function handlePost(req: NextApiRequest, res: NextApiResponse, claims?: Jw
aiOptInLevel,
accessToken,
baseUrl: getURL(),
signal: abortController.signal,
})
// Get a list of all schemas to add to context
+53 -39
View File
@@ -100,53 +100,67 @@ export async function handlePost(req: NextApiRequest, res: NextApiResponse) {
return res.status(500).json({ error: modelError.message })
}
const tools = await getTools({
projectRef,
connectionString,
authorization,
aiOptInLevel,
accessToken,
})
// Closes the remote MCP connection opened in getTools when generation is done,
// if anything below throws, or if the client disconnects mid-generation so the
// connection isn't held until generateText resolves on its own (mirrors the
// request-scoped cleanup in generate-v4.ts).
const toolsAbortController = new AbortController()
req.on('close', () => toolsAbortController.abort())
req.on('aborted', () => toolsAbortController.abort())
// Fires when the response finishes or the connection drops.
res.on('close', () => toolsAbortController.abort())
try {
const tools = await getTools({
projectRef,
connectionString,
authorization,
aiOptInLevel,
accessToken,
signal: toolsAbortController.signal,
})
const { experimental_output } = await generateText({
...modelParams,
stopWhen: stepCountIs(5),
prompt: source`
You are a Postgres RLS (Row Level Security) expert.
Determine the most appropriate policies for the "${schema}"."${tableName}" table within a Supabase project.
const { experimental_output } = await generateText({
...modelParams,
stopWhen: stepCountIs(5),
prompt: source`
You are a Postgres RLS (Row Level Security) expert.
Determine the most appropriate policies for the "${schema}"."${tableName}" table within a Supabase project.
${columns.length > 0 ? `Table columns: ${columns.join(', ')}` : 'No column metadata provided.'}
${columns.length > 0 ? `Table columns: ${columns.join(', ')}` : 'No column metadata provided.'}
${message ? `User request: ${message}` : ''}
${message ? `User request: ${message}` : ''}
RLS Guide: ${RLS_PROMPT}
RLS Guide: ${RLS_PROMPT}
Requirements:
- Use the available planning and schema tools (like "list_policies" or "list_tables") to inspect the "${schema}" schema and existing policies before generating new ones.
- Ensure policies strictly adhere to the existing schema
- Return a curated list of recommended CREATE POLICY statements as JSON.
- Each policy must include: name, sql, command (SELECT/INSERT/UPDATE/DELETE/ALL), action (PERMISSIVE/RESTRICTIVE), roles (array of role names).
- Include "definition" (USING clause expression without the USING keyword) for SELECT, UPDATE, DELETE policies.
- Include "check" (WITH CHECK clause expression without the WITH CHECK keywords) for INSERT, UPDATE policies.
- Avoid duplicating existing policies and reference the public schema and typical Supabase best practices when deciding the coverage.
- Prefer PERMISSIVE policies unless a RESTRICTIVE policy is explicitly required
`,
tools,
experimental_output: Output.object({
schema: z.object({
policies: z.array(policySchema),
Requirements:
- Use the available planning and schema tools (like "list_policies" or "list_tables") to inspect the "${schema}" schema and existing policies before generating new ones.
- Ensure policies strictly adhere to the existing schema
- Return a curated list of recommended CREATE POLICY statements as JSON.
- Each policy must include: name, sql, command (SELECT/INSERT/UPDATE/DELETE/ALL), action (PERMISSIVE/RESTRICTIVE), roles (array of role names).
- Include "definition" (USING clause expression without the USING keyword) for SELECT, UPDATE, DELETE policies.
- Include "check" (WITH CHECK clause expression without the WITH CHECK keywords) for INSERT, UPDATE policies.
- Avoid duplicating existing policies and reference the public schema and typical Supabase best practices when deciding the coverage.
- Prefer PERMISSIVE policies unless a RESTRICTIVE policy is explicitly required
`,
tools,
experimental_output: Output.object({
schema: z.object({
policies: z.array(policySchema),
}),
}),
}),
})
})
// Add table and schema to each policy from the request
const policies = (experimental_output?.policies ?? []).map((policy) => ({
...policy,
table: tableName,
schema,
}))
// Add table and schema to each policy from the request
const policies = (experimental_output?.policies ?? []).map((policy) => ({
...policy,
table: tableName,
schema,
}))
return res.json(policies)
return res.json(policies)
} finally {
toolsAbortController.abort()
}
} catch (error) {
if (error instanceof Error) {
console.error(`AI policy generation failed: ${error.message}`)
+3
View File
@@ -71,6 +71,9 @@
"OPENAI_API_KEY",
"BRAINTRUST_API_KEY",
"BRAINTRUST_PROJECT_ID",
// Gates the dashboard assistant between the remote MCP server and the
// legacy in-process one (see lib/ai/tools/mcp-tools.ts).
"USE_REMOTE_MCP",
"AUTH_JWT_SECRET",
"LOGFLARE_API_KEY",
"LOGFLARE_PUBLIC_ACCESS_TOKEN",