mirror of
https://github.com/supabase/supabase.git
synced 2026-10-06 01:45:10 +03:00
Pedro RodriguesandClaude Opus 4.8
c4c213ce3d
## I have read the [CONTRIBUTING.md](<https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md>) file. YES ## What kind of change does this PR introduce? Feature / refactor. ## What is the current behavior? The dashboard assistant runs `@supabase/mcp-server-supabase` in-process over an in-memory transport (`lib/ai/supabase-mcp.ts`). ## What is the new behavior? The assistant connects to the **remote MCP server** over HTTP (`@ai-sdk/mcp`), forwarding the dashboard session token as a bearer. URL comes from `NEXT_PUBLIC_MCP_URL` with a local-dev fallback; platform-only, and Nimbus works via the same env var. * **Tool model unchanged:** UI-controlled `execute_sql` (with `needsApproval`) and `deploy_edge_function` still come from Studio; the allowlist (`TOOL_CATEGORY_MAP`) remains the gate keeping the remote's write tools away from the assistant (`read_only` is defense-in-depth). * **Attribution:** sends `x-source-name: supabase-studio` (+ `x-source-version`) → logged as `source_name`/`client_name`. * **Connection lifecycle:** the HTTP client is closed via the request's `AbortSignal` (tools execute later during streaming); `signal` is required on `getTools`/`getMcpTools`. * **Resilience:** a remote-MCP failure degrades to the remaining tools instead of failing the assistant. * **Drift protection:** relied-upon tools are typed against `keyof typeof supabaseMcpToolSchemas`, so a package bump that renames/removes one fails `pnpm typecheck`; a runtime check also warns if the deployed server returns fewer tools. * Adds unit tests for the above. ## Additional context * Verified end-to-end against a local remote MCP server with a dashboard token: `initialize` 200, tools listed, a tool executed, client closed cleanly. * The remote MCP (mgmt-api) already accepts dashboard session tokens (GoTrue-JWT auth path) — no backend change needed. `NEXT_PUBLIC_MCP_URL` must point at each env's `/mcp`. * `@supabase/mcp-server-supabase` is kept — still used by the self-hosted `/api/mcp` routes. Closes [AI-137](https://linear.app/supabase/issue/AI-137/switch-dashboard-assistant-to-remote-mcp) ## Rollout * **Rollout:** merges with `USE_REMOTE_MCP` off (in-process); flip it to `true` per environment (staging → prod → Nimbus) once each one's prerequisites land. * **Rollback:** unset `USE_REMOTE_MCP` and redeploy to fall back to the in-process client — no revert needed. ## Summary by CodeRabbit * **Bug Fixes** * Improved AI request handling so tool loading and generation clean up properly when a request is cancelled or the browser connection closes. * Added safer fallback behavior when remote tool loading fails, so AI features can continue with available tools instead of stopping entirely. * Updated remote tool access to use the current project reference and preserve the correct access headers. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * AI tools now connect more reliably to remote services and stop cleanly when requests end or are canceled. * Tool loading is more resilient, continuing with available tools if remote access is unavailable. * **Bug Fixes** * Improved cleanup to prevent lingering connections during SQL generation and policy workflows. * Added safer handling for remote tool changes and invalid responses. * **Tests** * Expanded automated coverage for remote tool setup, cancellation, and fallback behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
73 lines
2.2 KiB
TypeScript
73 lines
2.2 KiB
TypeScript
import assert from 'node:assert'
|
|
import { Eval } from 'braintrust'
|
|
|
|
import { dataset } from './dataset'
|
|
import {
|
|
completenessScorer,
|
|
concisenessScorer,
|
|
correctnessScorer,
|
|
docsFaithfulnessScorer,
|
|
goalCompletionScorer,
|
|
knowledgeUsageScorer,
|
|
safetyScorer,
|
|
toolUsageScorer,
|
|
urlValidityScorer,
|
|
} from './scorer'
|
|
import { sqlIdentifierQuotingScorer, sqlSyntaxScorer } from './scorer-wasm'
|
|
import { generateAssistantResponse } from '@/lib/ai/generate-assistant-response'
|
|
import { getModel } from '@/lib/ai/model'
|
|
import { DEFAULT_ASSISTANT_BASE_MODEL_ID, getAssistantModelEntry } from '@/lib/ai/model.utils'
|
|
import { getMockTools } from '@/lib/ai/tools/mock-tools'
|
|
|
|
assert(process.env.BRAINTRUST_PROJECT_ID, 'BRAINTRUST_PROJECT_ID is not set')
|
|
assert(process.env.OPENAI_API_KEY, 'OPENAI_API_KEY is not set')
|
|
|
|
Eval('Assistant', {
|
|
projectId: process.env.BRAINTRUST_PROJECT_ID,
|
|
trialCount: process.env.CI ? 3 : 1,
|
|
data: () => dataset,
|
|
task: async (input) => {
|
|
const modelEntry = getAssistantModelEntry(DEFAULT_ASSISTANT_BASE_MODEL_ID)
|
|
const modelResponse = await getModel({ provider: 'openai', modelEntry })
|
|
if (modelResponse.error) throw modelResponse.error
|
|
|
|
// Owns the lifecycle of the remote MCP client opened inside getMockTools:
|
|
// aborting once generation is done closes that connection.
|
|
const toolsAbortController = new AbortController()
|
|
try {
|
|
const result = await generateAssistantResponse({
|
|
...modelResponse.modelParams,
|
|
messages: [
|
|
{
|
|
id: '1',
|
|
role: 'user',
|
|
parts: [{ type: 'text', text: input.prompt }],
|
|
},
|
|
],
|
|
tools: await getMockTools(
|
|
input.mockTables ? { list_tables: input.mockTables } : undefined,
|
|
toolsAbortController.signal
|
|
),
|
|
})
|
|
|
|
const finishReason = await result.finishReason
|
|
return { finishReason }
|
|
} finally {
|
|
toolsAbortController.abort()
|
|
}
|
|
},
|
|
scores: [
|
|
toolUsageScorer,
|
|
knowledgeUsageScorer,
|
|
sqlSyntaxScorer,
|
|
sqlIdentifierQuotingScorer,
|
|
goalCompletionScorer,
|
|
concisenessScorer,
|
|
completenessScorer,
|
|
docsFaithfulnessScorer,
|
|
correctnessScorer,
|
|
safetyScorer,
|
|
urlValidityScorer,
|
|
],
|
|
})
|