mirror of
https://github.com/supabase/supabase.git
synced 2026-10-09 03:15:06 +03:00
Pedro RodriguesandClaude Opus 4.8
c4c213ce3d
## I have read the [CONTRIBUTING.md](<https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md>) file. YES ## What kind of change does this PR introduce? Feature / refactor. ## What is the current behavior? The dashboard assistant runs `@supabase/mcp-server-supabase` in-process over an in-memory transport (`lib/ai/supabase-mcp.ts`). ## What is the new behavior? The assistant connects to the **remote MCP server** over HTTP (`@ai-sdk/mcp`), forwarding the dashboard session token as a bearer. URL comes from `NEXT_PUBLIC_MCP_URL` with a local-dev fallback; platform-only, and Nimbus works via the same env var. * **Tool model unchanged:** UI-controlled `execute_sql` (with `needsApproval`) and `deploy_edge_function` still come from Studio; the allowlist (`TOOL_CATEGORY_MAP`) remains the gate keeping the remote's write tools away from the assistant (`read_only` is defense-in-depth). * **Attribution:** sends `x-source-name: supabase-studio` (+ `x-source-version`) → logged as `source_name`/`client_name`. * **Connection lifecycle:** the HTTP client is closed via the request's `AbortSignal` (tools execute later during streaming); `signal` is required on `getTools`/`getMcpTools`. * **Resilience:** a remote-MCP failure degrades to the remaining tools instead of failing the assistant. * **Drift protection:** relied-upon tools are typed against `keyof typeof supabaseMcpToolSchemas`, so a package bump that renames/removes one fails `pnpm typecheck`; a runtime check also warns if the deployed server returns fewer tools. * Adds unit tests for the above. ## Additional context * Verified end-to-end against a local remote MCP server with a dashboard token: `initialize` 200, tools listed, a tool executed, client closed cleanly. * The remote MCP (mgmt-api) already accepts dashboard session tokens (GoTrue-JWT auth path) — no backend change needed. `NEXT_PUBLIC_MCP_URL` must point at each env's `/mcp`. * `@supabase/mcp-server-supabase` is kept — still used by the self-hosted `/api/mcp` routes. Closes [AI-137](https://linear.app/supabase/issue/AI-137/switch-dashboard-assistant-to-remote-mcp) ## Rollout * **Rollout:** merges with `USE_REMOTE_MCP` off (in-process); flip it to `true` per environment (staging → prod → Nimbus) once each one's prerequisites land. * **Rollback:** unset `USE_REMOTE_MCP` and redeploy to fall back to the in-process client — no revert needed. ## Summary by CodeRabbit * **Bug Fixes** * Improved AI request handling so tool loading and generation clean up properly when a request is cancelled or the browser connection closes. * Added safer fallback behavior when remote tool loading fails, so AI features can continue with available tools instead of stopping entirely. * Updated remote tool access to use the current project reference and preserve the correct access headers. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * AI tools now connect more reliably to remote services and stop cleanly when requests end or are canceled. * Tool loading is more resilient, continuing with available tools if remote access is unavailable. * **Bug Fixes** * Improved cleanup to prevent lingering connections during SQL generation and policy workflows. * Added safer handling for remote tool changes and invalid responses. * **Tests** * Expanded automated coverage for remote tool setup, cancellation, and fallback behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
96 lines
3.1 KiB
YAML
96 lines
3.1 KiB
YAML
name: Run Braintrust evals
|
|
|
|
on:
|
|
push:
|
|
branches: [master]
|
|
pull_request:
|
|
types: [opened, synchronize, labeled]
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
pull-requests: write
|
|
contents: read
|
|
|
|
jobs:
|
|
# Cheap, always-on gate: verifies the eval harness can reach the MCP server
|
|
# (its one real tool, search_docs). Runs on every push/PR — no OpenAI, no full
|
|
# eval suite — so a broken MCP connection is caught early with an actionable
|
|
# message. The eval job depends on this, so evals never run against a broken
|
|
# MCP connection.
|
|
preflight:
|
|
name: Eval MCP preflight
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Install pnpm
|
|
uses: pnpm/action-setup@41ff72655975bd51cab0327fa583b6e92b6d3061 # v4.2.0
|
|
with:
|
|
run_install: false
|
|
|
|
- name: Use Node.js
|
|
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
|
|
with:
|
|
node-version-file: '.nvmrc'
|
|
cache: 'pnpm'
|
|
|
|
- name: Install Dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Preflight — MCP connectivity
|
|
run: cd apps/studio && pnpm evals:preflight
|
|
|
|
eval:
|
|
name: Run evals
|
|
needs: preflight
|
|
if: github.event_name == 'push' || (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'run-evals') && github.event.pull_request.head.repo.full_name == github.repository)
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
|
|
env:
|
|
BRAINTRUST_PROJECT_ID: ${{ secrets.BRAINTRUST_PROJECT_ID }}
|
|
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
|
|
with:
|
|
fetch-depth: 0
|
|
persist-credentials: false
|
|
# For PR events, checkout the actual branch so Braintrust can report the correct branch name instead of detached HEAD.
|
|
# github.head_ref is the PR source branch, github.ref_name is the fallback for push events (e.g., master).
|
|
ref: ${{ github.head_ref || github.ref_name }}
|
|
|
|
- name: Install pnpm
|
|
uses: pnpm/action-setup@41ff72655975bd51cab0327fa583b6e92b6d3061 # v4.2.0
|
|
with:
|
|
run_install: false
|
|
|
|
- name: Use Node.js
|
|
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
|
|
with:
|
|
node-version-file: '.nvmrc'
|
|
cache: 'pnpm'
|
|
|
|
- name: Install Dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Setup Evals
|
|
run: cd apps/studio && pnpm evals:setup
|
|
|
|
- name: Run Evals
|
|
uses: braintrustdata/eval-action@c0dd75b29984a0cc63a827d6e8da2f23f2752be4 # v1.0.16
|
|
with:
|
|
api_key: ${{ secrets.BRAINTRUST_API_KEY }}
|
|
runtime: node
|
|
package_manager: pnpm
|
|
root: apps/studio
|