mirror of
https://github.com/supabase/supabase.git
synced 2026-10-06 01:45:10 +03:00
codex/fix-tanstack-e2e
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Saxon FletcherandClaude Opus 5.5
|
5e59b6047e |
chore(studio): extend Assistant response time and handle timeouts (#50892)
## Problem
- Assistant responses were capped at 120 seconds and 10 steps, which is
too short for longer reasoning or multi-step tool work.
- When the hosting platform ended a request at that limit, the
connection just dropped. The user got no explanation, and "Thinking…"
and tool rows kept spinning.
- Studio's own tools ignored the request's abort signal, so a stop,
disconnect or deadline couldn't cancel their in-flight requests.
- Aborted responses never closed their Braintrust span. Under TanStack
Start, the remote MCP client was only released on `res.on('close')`,
which the adapter never emits.
## Solution
Uses AI SDK options instead of custom stream handling:
- `maxDuration` goes to 300s and the step limit to 20. `streamText({
timeout: { totalMs } })` stops the response at 270s, leaving time to
finish the stream before the platform cutoff.
- `toUIMessageStream({ messageMetadata })` marks an aborted response
`timedOut: true`. `Chat` ignores `abort` chunks, so the client reads
this flag instead and shows a timeout alert with Retry. The flag is
saved with the message, so the alert survives a reload.
- `toUIMessageStream({ onEnd })` aborts the request whenever the stream
ends, releasing the MCP client on both runtimes. `streamText({ onAbort
})` ends the Braintrust span.
- Studio tools pass the SDK's `abortSignal` to their fetches. MCP tools
already did.
- Reasoning and server-tool rows that never finished show "Response
interrupted" instead of a spinner or "Ran X ✓".
There's no per-tool timeout. Approved SQL and migrations can
legitimately run longer, and aborting the HTTP request doesn't stop the
query in Postgres.
## Review instructions
1. Run the unit tests: `cd apps/studio && pnpm vitest run
lib/api/generate-v4.test.ts lib/ai components/ui/AIAssistantPanel`
2. To see a timeout without waiting 4.5 minutes, temporarily set
`ASSISTANT_TIMEOUT_MS` in `apps/studio/lib/ai/assistant-timeout.ts` to
`15_000` and run `pnpm dev:studio`.
3. Ask the Assistant something that needs several tool calls or long
reasoning, for example "Audit my schema for missing indexes and RLS
gaps, then write the fixes."
4. After 15 seconds, check that:
- the response stops and a "Assistant response timed out" alert appears
with Retry
- any in-progress reasoning or tool row shows "Response interrupted"
instead of spinning
- Retry starts a new response
- reloading the page still shows the alert on that chat
5. Stop a response with the Stop button before the deadline. It should
stop without the timeout alert.
6. With the default 270s, confirm that a normal response completes as
before.
## Checklist
Check all before review:
- [ ] I have read
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
- [ ] If I wrote a new docs topic or edited an existing topic, I used
the `/write-the-docs` or `/edit-the-docs` skill, which references
[WORD_LIST](https://github.com/supabase/supabase/blob/master/apps/docs/WORD_LIST.md)
and the docs
[CONTRIBUTING](https://github.com/supabase/supabase/blob/master/apps/docs/CONTRIBUTING.md)
guide
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Improvements**
* AI assistant responses can now run for up to five minutes, supporting
longer requests.
* When a response times out, the assistant displays a message suggesting
you retry or ask for a smaller change.
* Incomplete responses now show a “Response interrupted” notice, and
loading indicators stop when generation ends.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Matt Rossman
|
fd5ef806d6 |
feat(studio): enable Assistant tracing for High Compliance projects (#50759)
Assistant chats from High Compliance projects now flow to Braintrust like any other project. The constraint that required suppressing them no longer applies, see AI-1241 for the details. `isTracingAllowed` now takes only the project region to maintain EU exclusion. Traces also carry an `isHighComplianceProject` metadata field, so the project's status at the time of the trace is recorded rather than looked up later against a setting customers can toggle. To verify, see [this sample trace](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?r=afabbdcc-aa89-446e-aa52-78aaa90d44a4&v=Production&s=afabbdcc-aa89-446e-aa52-78aaa90d44a4&tvt=trace) from a High Compliance project on staging which indicates that tracing is now enabled for these projects and that it carries metadata showing the high compliance status. | High Compliance project setting | `isHighComplianceProject` metadata | |--------|--------| | <img width="1554" height="454" alt="CleanShot 2026-09-22 at 5 14 58 PM@2x" src="https://github.com/user-attachments/assets/23901c6e-0d79-44e8-a6dd-43cdedba1799" /> | <img width="1674" height="990" alt="CleanShot 2026-09-22 at 5 17 40 PM@2x" src="https://github.com/user-attachments/assets/fb2fba55-bcc1-4136-a432-b33a5c7f9ca2" /> | Closes AI-1241 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Changes** * AI project compliance information is now represented by a unified high-compliance project status. * AI response tracing is now determined by project region: tracing remains disabled for EU and unknown regions, while known non-EU regions are eligible. * AI feedback and SQL generation now use the updated compliance and regional handling. * **Tests** * Updated coverage to reflect the revised compliance and tracing behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
1966209483 |
chore(deps): upgrade vitest to v5 (#49994)
Upgrades Vitest from 4.1.4 to 5.0.0 across the monorepo, fixes the handful of things v5 turned into hard errors, and drops the `vi.clearAllMocks()` boilerplate that v5's `clearMocks` default makes redundant. **Changed:** - `vitest`, `@vitest/ui`, `@vitest/coverage-v8` 4.1.4 → 5.0.0 (catalog) - `vi.mock` calls that lived inside `beforeAll`/`beforeEach`/test bodies moved to module scope (v5 throws on nested calls). Affects the Studio and docs setup files and four Studio tests. - `detectBrowser` test restores `navigator` via `vi.unstubAllGlobals()` instead of assigning `global.navigator`, which now reaches jsdom's getter-only property. - `RowEditor.utils.test.ts` restores its `JSON.stringify` spy. It used to leak a throwing mock for the rest of the file, which v5's coverage provider now trips over. A later test in the same file had been asserting the leak's side effect (valid JSON reported as invalid) and now asserts the correct behavior. - `@testing-library/jest-dom` 6.6 → 7.0.1. Its vitest type augmentation resolves through a peer now, so it lands on each package's own `vitest` instead of whichever copy pnpm hoisted. Fixes `toBeInTheDocument` type errors in dev-tools after the reshuffle. - `@testing-library/react` 16.0.0 → 16.3.3 for the React 19 peer range. - `vite: catalog:` added to dev-tools, www, and common. Without it they resolved a newer vite than the catalog pin, which forked a second vitest instance in the lockfile. There's now one. - ai-commands custom matcher types use v5's `Matchers<R, T>` form. - 110 test files: `vi.clearAllMocks()` removed from `beforeEach`/`afterEach` hooks, along with hooks that only did that and the imports they left unused. Calls that also reset/restore mocks are untouched. Second commit, mechanical. **Added:** - `.vitest/` to the root gitignore (v5 writes JSON/JUnit/HTML reporter output there) **Removed:** - `vite-tsconfig-paths` catalog entry and deps. Vitest 5 resolves tsconfig paths itself. Release-age note: this sat in draft with a temporary `minimumReleaseAgeExclude` entry for `vitest` and `@vitest/*` while 5.0.0 was inside the workspace's 3-day `minimumReleaseAge` window. That window has closed, so the exclusion is gone and nothing bypasses the release-age gate. **Perf** (local, medians of 3 runs, same machine): | Suite | v4.1.4 | v5.0.0 | |---|---|---| | studio | 144.1s | 141.7s (-2%) | | studio `--coverage` | 156.9s | 146.4s (-7%) | | ui-patterns | 6.27s | 5.07s (-19%) | | ui `--coverage` | 3.35s | 2.14s (-36%) | | www | 0.89s | 0.47s (-47%) | Studio is dominated by jsdom environment setup per file, which v5 doesn't change. `vitest doctor` recommends keeping the current pool config: the vm pools and `isolate: false` all break tests. ## To test - `pnpm install --frozen-lockfile` succeeds with no `minimumReleaseAgeExclude` entry for vitest. - CI: Studio unit tests, ui, ui-patterns, www, docs, and typecheck/lint should all be green. The lint ratchet was checked locally: warning counts on touched Studio files are identical to master. - `pnpm test:studio` locally passes with coverage (588 files, 6240 tests). - Open a Studio test that uses `toBeInTheDocument` in your editor and confirm no type errors on jest-dom matchers, in Studio and in `packages/dev-tools`. - Known pre-existing failures unrelated to this PR: one dev-tools test (`getEventCountBadge` capped pill) fails on master too. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Tests - Improved test coverage for JSON validation and mobile navigation behavior. - Updated test setup, cleanup, environment configuration, and matcher support across application and shared package suites. - Removed obsolete coverage for alternate MCP transport selection. ## Chores - Streamlined TypeScript path resolution and Vitest reporter output handling. - Updated testing libraries and Vitest tooling across documentation, Studio, website, and shared packages. - Added Vitest reporter output to ignored files. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com> Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com> |
||
|
Ali Waseem
|
b04d14856a |
Resolve AI opt-in and tracing settings server-side (#48855)
The AI endpoints resolved organization and project settings independently and applied them together without confirming they belonged to the same pairing. Consolidates both into a single `getAIDetails` that reconciles them and falls back to the most restrictive posture when unconfirmed, and applies the HIPAA sensitivity gate to the opt-in level, which previously only existed on the client. Fixes FE-4110 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Consolidated AI access details across organization and project settings. - AI functionality now validates project ownership and disables access for mismatched or HIPAA-sensitive projects. - AI responses include plan, region, opt-in status, sensitivity, authorization, and advanced model access information. - **Bug Fixes** - Improved fail-closed behavior when project or organization data is missing or inconsistent. - Updated AI generation, feedback, rate, and policy flows to consistently apply consolidated access settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Carel de WaalandAlaister Young
|
0acc0eb8b3 |
feat: Support Form - Sync AI assistant conversation to Front (#46778)
# Sync AI assistant conversation to Front ## What & why When a user submits a support ticket, an AI assistant chat opens so they get help immediately while waiting for a human agent. This PR mirrors every turn of that chat into the Front conversation the support form already created, so the support team sees the full context and Front automations (routing, emails, CSAT) can act on it. Studio holds no Front credentials — it calls the platform endpoints (see the platform PR) to do the syncing. The assistant card is gated behind the `supportAssistantFollowUp` ConfigCat flag. ## How it works 1. **Submit** — `SupportFormV3` generates a stable `threadRef` (via the `uuid` package — `crypto.randomUUID()` is `undefined` in insecure contexts like non-localhost HTTP and would throw, silently aborting the submit) and sends it on `/platform/feedback/send`. The response returns the Front `conversationId`. Both are stored on `SubmittedSupportRequest`. 2. **Open chat** — `SupportAssistantSuccessCardContent` opens a chat seeded with `supportMetadata` (`threadRef`, `frontConversationId`, subject, category, severity, …). The first message is a `<support>…</support>` XML block. 3. **First user message** — the chat is tagged `isSupportChat = true`; the `onFinish` hook fires `syncSupportChatToFront`. 4. **Subsequent turns** — each `onFinish` slices the unsynced delta, strips the XML metadata block from the seed message, and posts to the platform messages endpoint. 5. **Escalation / resolve** — the `escalate_to_human` / `resolve_support_conversation` tools (and manual **Escalate**/**Resolve** buttons in the assistant input) flip lifecycle status via `setSupportLifecycleStatus` → `syncSupportLifecycleToFront`, which calls the escalation/resolve endpoints. Front rules act on `ai_support_status`. The assistant only resolves after the user explicitly confirms the issue is fixed. ## Key design decisions - **`threadRef` as the shared key** — one UUID travels as `threadRef` on submit and as `chatId` on every sync, so all messages thread into a single Front conversation. - **`conversationId` from the form response** — passed to all sync/lifecycle calls so the platform skips lazy derivation and PATCHes custom fields directly. - **Delta-only sync** — `lastSyncedMessageCount` tracks what's been sent; the boundary is snapshotted before the async call to avoid skipping messages that arrive mid-flight. - **Server-side de-dup** — stable `external_id` (`chatId:msg.id`) means retries don't duplicate in Front. - **Fire-and-forget** — sync failures log to Sentry, never break the chat; `isSyncing` resets on rehydration so the next `onFinish` retries the same delta. Message and lifecycle syncs use separate guards (`isSyncing` / `isLifecycleSyncing`) so an in-flight message sync can't drop an escalate/resolve. - **Lifecycle queued until the conversation exists** — if a lifecycle transition is requested before the initial message sync has returned a `frontConversationId`, it's stored as `pendingLifecycleStatus` and flushed once the id is assigned, rather than dropped. - **Tools return immediately** — the lifecycle tools return a stub to the AI SDK; the real Front call happens in `onFinish`, keeping async I/O out of the tool execute path. - **XML seed stripped before sync** — only the user's actual `<message>` is sent to Front (or dropped entirely if the form already created the conversation). ## Changes | Area | File(s) | | --- | --- | | Support form state | `SupportForm.state.ts` — `threadRef` / `frontConversationId` on `SubmittedSupportRequest` | | Support form submit | `support-ticket-send.ts` — sends `threadRef`, reads `conversationId` | | Support form UI | `SupportFormV3.tsx` — generates `threadRef`, stores `conversationId` | | AI assistant state | `ai-assistant-state.tsx` — `SupportChatMetadata`, `setSupportLifecycleStatus`, `onFinish` wiring, tool handling | | Message sync | `state/ai-chat-front-sync.ts` — delta tracking, message filtering, initial vs. incremental | | API data layer | `data/feedback/ai-chat-front-sync.ts` — typed platform-client wrappers for the three conversation endpoints | | Support tools | `lib/ai/tools/support-tools.ts` — `escalate_to_human`, `resolve_support_conversation` | | Tool integration | `lib/ai/tool-filter.ts`, `tools/index.ts`, `generate-assistant-response.ts` | | Success card | `SupportAssistantSuccessCardContent.tsx` — tags chat on first engagement | | Assistant panel UI | `AIAssistant.tsx` — Escalate/Resolve buttons, disabled input on closed chats, support placeholders | <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Support chats now include “Escalate to human” and “Resolve” actions. - Support submissions can be associated with a stable Front thread via a generated `threadRef`, preserving linkage across follow-ups. - AI assistant responses and input hints adapt when support mode is active. - **Bug Fixes** - Improved support chat state management and lifecycle handling to keep conversation metadata and message history synchronized more reliably with Front. - **Chores** - Added/updated coverage to reflect the new support-chat state and syncing behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com> |
||
|
Pedro RodriguesandClaude Opus 4.8
|
c4c213ce3d |
feat(studio): switch dashboard assistant to remote MCP server (#47479)
## I have read the [CONTRIBUTING.md](<https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md>) file. YES ## What kind of change does this PR introduce? Feature / refactor. ## What is the current behavior? The dashboard assistant runs `@supabase/mcp-server-supabase` in-process over an in-memory transport (`lib/ai/supabase-mcp.ts`). ## What is the new behavior? The assistant connects to the **remote MCP server** over HTTP (`@ai-sdk/mcp`), forwarding the dashboard session token as a bearer. URL comes from `NEXT_PUBLIC_MCP_URL` with a local-dev fallback; platform-only, and Nimbus works via the same env var. * **Tool model unchanged:** UI-controlled `execute_sql` (with `needsApproval`) and `deploy_edge_function` still come from Studio; the allowlist (`TOOL_CATEGORY_MAP`) remains the gate keeping the remote's write tools away from the assistant (`read_only` is defense-in-depth). * **Attribution:** sends `x-source-name: supabase-studio` (+ `x-source-version`) → logged as `source_name`/`client_name`. * **Connection lifecycle:** the HTTP client is closed via the request's `AbortSignal` (tools execute later during streaming); `signal` is required on `getTools`/`getMcpTools`. * **Resilience:** a remote-MCP failure degrades to the remaining tools instead of failing the assistant. * **Drift protection:** relied-upon tools are typed against `keyof typeof supabaseMcpToolSchemas`, so a package bump that renames/removes one fails `pnpm typecheck`; a runtime check also warns if the deployed server returns fewer tools. * Adds unit tests for the above. ## Additional context * Verified end-to-end against a local remote MCP server with a dashboard token: `initialize` 200, tools listed, a tool executed, client closed cleanly. * The remote MCP (mgmt-api) already accepts dashboard session tokens (GoTrue-JWT auth path) — no backend change needed. `NEXT_PUBLIC_MCP_URL` must point at each env's `/mcp`. * `@supabase/mcp-server-supabase` is kept — still used by the self-hosted `/api/mcp` routes. Closes [AI-137](https://linear.app/supabase/issue/AI-137/switch-dashboard-assistant-to-remote-mcp) ## Rollout * **Rollout:** merges with `USE_REMOTE_MCP` off (in-process); flip it to `true` per environment (staging → prod → Nimbus) once each one's prerequisites land. * **Rollback:** unset `USE_REMOTE_MCP` and redeploy to fall back to the in-process client — no revert needed. ## Summary by CodeRabbit * **Bug Fixes** * Improved AI request handling so tool loading and generation clean up properly when a request is cancelled or the browser connection closes. * Added safer fallback behavior when remote tool loading fails, so AI features can continue with available tools instead of stopping entirely. * Updated remote tool access to use the current project reference and preserve the correct access headers. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * AI tools now connect more reliably to remote services and stop cleanly when requests end or are canceled. * Tool loading is more resilient, continuing with available tools if remote access is unavailable. * **Bug Fixes** * Improved cleanup to prevent lingering connections during SQL generation and policy workflows. * Added safer handling for remote tool changes and invalid responses. * **Tests** * Expanded automated coverage for remote tool setup, cancellation, and fallback behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Joshen Lim
|
1baaded0bb |
Consolidate execute-sql-query into execute-sql-mutation (#46944)
## Context Just some clean up as I was going through stuff - `useExecuteSqlQuery` is deprecated and not used at all - As such `execute-sql-query` is technically irrelevant, the more relevant file is `execute-sql-mutation` - Hence opting to consolidate `execute-sql-query` into `execute-sql-mutation` - Also removing `ExecuteSqlError` since its just re-exporting the `ResponseError` type There's a lot of file changes but its essentially just updating the importing statements across the files |
||
|
Charis
|
39ec777d22 |
studio: SafeSql stragglers + remaining tests (6/7) (#46006)
## Summary Part 6 of 7 in the SafeSql migration stack. Picks up the few remaining files that didn't fit cleanly into earlier batches: - `components/Docs/Description.tsx` — comment statements built via `safeSql`. - `components/Integrations/IntegrationOverviewTabV2/InstallIntegrationSheet.test.tsx` — test fixture updated to `SafeSqlFragment`. - `lib/ai/tools/studio-tools.test.ts` — test fixture updated. - `lib/api/generate-v4.test.ts` — test fixture updated. Sets up PR 7, which flips the `executeSql` signature itself. ## Test plan - [x] `pnpm typecheck` passes - [x] Specific Studio unit tests run on top of the stack (`Policies.utils.test.ts`, `SidePanelEditor.utils.createTable.test.ts`, `useQueryInsightsIssues.utils.test.ts`) - [x] Dev-server smoke: Docs panel renders / accepts edits <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Updated test suites to reflect internal type definition changes. * **Refactor** * Internal code improvements to enhance type safety and consistency across the codebase. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46006) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
d143571586 |
feat(assistant): trace-level scorers + server-side tool execution with needsApproval (#45654)
## Motivation When Assistant runs a potentially destructive tool like `execute_sql`, it stops the LLM request and prompts for client-side approval and execution of the tool. After approval, a second request kicks off under a separate trace. This has made scoring and [Topics](https://www.braintrust.dev/blog/topics) classification challenging, as the generated `output` is split across stateless requests. The [span-level scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans) approach we've used thusfar (after the LLM call, we massage the result into an `output` payload that's stuck onto the root span) has been cumbersome and led to invalid scores / topics where only part of the assistant response is considered. It's also inefficient, as we're duplicating potentially large info (like the `search_docs` output) that already exists within the trace. An alternative to scoring spans is to [score traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces). Braintrust [best practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices) advise: > Use span scope for evaluating individual operations or outputs. Use trace scope for evaluating multi-turn conversations, overall workflow completion, or when your scorer needs access to the full execution context. We've also received [direct guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89) from their team to use this approach. ## Changes Migrates eval scorers from custom `AssistantEvalOutput` shape to trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with thread parsing that scores the full latest Assistant turn and passes prior conversation separately where relevant. Moves `execute_sql` and `deploy_edge_function` from client-side execution after approval to AI SDK `needsApproval` + server-side `execute()`. SQL results returned to the model are gated by AI opt-in level, so row data is only included with `schema_and_log_and_data`; otherwise the tool returns the no-data-permissions sentinel. Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within an "assistant" turn due to tool call requests/responses. For online evals, this means we should configure automations to only score traces with `metadata.isFinalStep = true` to ensure we're judging the complete generated response. Other minor kaizen changes: - Renamed `promptProviderOptions` to `systemProviderOptions` to clarify that this is associated with the "system" message and disambiguate from the root `providerOptions` - Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown` span shapes from Braintrust, to more easily access typed inputs/output on tool spans. - Bumps AI SDK floor version `^6.0.116` → `^6.0.174` - Tweaked the "Conciseness" scorer to not unfairly dock points for the new `[called tool_name]` labels in serialized assistant response ## Verification In the studio staging build, I asked Assistant to create a todos table with 3 sample todos. I manually approved the `execute_sql` call and saw Assistant generate text before & after the call. In Braintrust I verified two traces were produced (see [filtered logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})), the first with `metadata.isFinalStep = false` and the second with `metadata.isFinalStep = true`. In the Braintrust staging scorers, I ran the preview Completeness scorer on the second trace and verified it sees the complete Assistant response including markers for tool calls ([link to trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82)) <img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x" src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a" /> I also tested the `deploy_edge_function` workflow and verified it still prompts for permission and warns on deployment of existing functions. **References** - https://www.braintrust.dev/docs/evaluate/custom-code#score-traces - https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval Supercedes https://github.com/supabase/supabase/pull/45556 and https://github.com/supabase/supabase/pull/45339 Closes AI-473 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tool actions (SQL execution, edge-function deploy) now require explicit user Approve/Deny before proceeding. * **Improvements** * Assistant pauses for approval responses before sending follow-ups, giving clearer control over risky actions. * Deploy/replace flows show confirmation and clearer replace warnings. * Evaluation/scoring updated to use richer trace data for more accurate assistant performance signals. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
36ae9beb0c |
chore(ai): remove DPA signer killswitch for assistant tracing (#45134)
Removes the temporary killswitch added when Braintrust was onboarded as
a subprocessor, to satisfy the 30-day DPA notice obligation. The window
has elapsed and legal has cleared removal.
Drops the `orgIsDpaSigned` check from `isTracingAllowed`, removes the
extra `/platform/organizations/{slug}/documents/dpa-signed` network hop
from `getOrgAIDetails`, and cleans up all call sites and tests.
Closes AI-596
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Simplified AI tracing eligibility logic by removing DPA signing status
checks. Tracing authorization decisions now depend solely on region,
HIPAA addon status, and project sensitivity settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Charis
|
205cbe7d26 | chore(studio}: enforce import order, remove bare import specifiers (#44585) | ||
|
Matt Rossman
|
8aeacc6152 |
feat(assistant): disable Braintrust tracing for EU regions and DPA signers (#44504)
**Changes** - Extracted tracing conditional to an `isTracingAllowed` helper with unit tests (the function is simple but sensitive hence the extra testing precaution) - Disables Braintrust tracing for projects in EU database regions (region prefix `eu-`) to address GDPR data residency concerns - Disables Braintrust tracing for orgs whose owners have signed the previous DPA, as a stopgap during the 30-day notice period for the updated DPA that adds Braintrust as a subprocessor - Refactored `org-ai-details.ts` → `ai-details.ts`, splitting `getOrgAIDetails` into separate org and project helpers to cleanly scope the EU-region check at the project level DPA check uses the newly added `/documents/dpa-signed` endpoint from https://github.com/supabase/platform/pull/31060. This PR includes regenerated `api.d.ts` and `platform.d.ts` from running `pnpm codegen` in `packages/api-types` to get type safety on this new endpoint. Note tracing is still yet to be activated in production, this is a preparatory step. **To verify** Send a chat message and check for the `x-braintrust-span-id` response header on `POST /api/ai/sql/generate-v4` — it should be absent for DPA-signed orgs or EU-region projects, and present otherwise. <img width="3594" height="1992" alt="CleanShot 2026-04-03 at 14 28 58@2x" src="https://github.com/user-attachments/assets/4c91d7ad-2604-4531-a78e-dedf41632fa5" /> If you have access to the Braintrust dashboard, you can also verify whether logs are produced or not in the Assistant project there. Closes AI-570 Closes AI-569 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tracks organization DPA signing and detects EU-region projects * Assistant tracing now follows a combined compliance policy (HIPAA addon, DPA, project sensitivity, region) * Added helpers to fetch org and project AI details * **Documentation** * Expanded API docs with additional examples and clarified parameter descriptions * Added response schemas for subscription preview and document status * **Tests** * Added/updated tests covering DPA/region behavior and tracing policy enforcement <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
adf8b0c67c |
feat(assistant): per-endpoint reasoningEffort + model config cleanup (#43981)
We're exploring support for newer models like [gpt-5.4-nano](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/) in Assistant. This model doesn't support the `'minimal'` reasoning effort level we use for gpt-5-mini which leads to vague errors. <img width="595" height="263" alt="CleanShot 2026-03-18 at 17 13 05@2x" src="https://github.com/user-attachments/assets/cf7c2370-322d-4a8a-be55-23e680db0aa0" /> Also, we've [previously discussed](https://supabase.slack.com/archives/C0161K73J1J/p1771544464850199?thread_ts=1771493920.775699&cid=C0161K73J1J) that reasoning adds unnecessary latency to otherwise simple AI completion endpoints like `title-v2`. We want more control of reasoning level independent of model/endpoint. This PR aims to solve both problems by: - making reasoning effort configurable on a per-request basis - adding compile-time guardrails to prevent selecting an incompatible reasoning level for models - adding a `DEFAULT_COMPLETION_MODEL` with minimal reasoning that we can update with newer models that support disabling reasoning (independent of Assistant chat model reasoning) Other improvements to our model config logic: - Fixes bug in `onboarding/design.ts` and `assistant.eval.ts` where `providerOptions` was being dropped - `getModel()` now returns a bundled `modelParams` object (spread into AI SDK calls) so `providerOptions` can't be accidentally omitted (this [has happened before](https://supabase.slack.com/archives/C0161K73J1J/p1771518443534309?thread_ts=1771493920.775699&cid=C0161K73J1J)) - Introduces an `ASSISTANT_MODELS` registry as a single source of truth for assistant model config, eliminating hardcoded model IDs across the codebase - Aligns free/pro model conditional logic with `assistant.advance_model` entitlement naming conventions instead of the `isLimited` pattern - Adds `console.error` logging of Assistant stream errors so we can interpret reasoning effort compatibility errors in the future (instead of just opaque "Sorry, I'm having trouble responding right now" card) - Removes unnecessary type casts and generally making the model config logic stricter - Removes pre-existing dead code: `anthropic` provider variant in `GetModelParams` / `PROVIDERS` registry that was never implemented in `getModel()` Now if you try to select an unsupported reasoning level you get a type error: <img width="1306" height="320" alt="CleanShot 2026-03-20 at 14 37 24@2x" src="https://github.com/user-attachments/assets/a6ac234b-5ea5-4d81-8e01-ac4be34a0800" /> And if for some reason an invalid reasoning level slips through, you now get a server-side error surfacing the issue: <img width="1268" height="204" alt="CleanShot 2026-03-20 at 14 58 14@2x" src="https://github.com/user-attachments/assets/aadc1b7a-9495-475f-9741-39979bd27cd7" /> I've tested gpt-5 and gpt-5-mini are still working on the staging preview and verified the models were selected properly in Braintrust logs. Both models are available on my Pro test account, and my Free test account shows the Pro upgrade CTA. Closes AI-446 Closes AI-551 |
||
|
|
072883bcec |
feat: assistant evals (#41311)
* chore: bump `supabase` CLI * chore: stricter message types in `generate-v4.ts` * feat: tutorial eval https://www.braintrust.dev/docs/evaluation * feat: project ID for eval * refactor: `generateAssistantResponse` out of `handlePost` * refactor: generateAssistantResponse to lib/ai * feat: factuality eval with assistant response * chore: upgrade braintrust to v1.0.1 * chore: silence tsconfig warning * feat: assertion scorer * fix: aggregate tools across all steps * refactor: strict tool names, remove need for `as const` * refactor: generic tool name type in assertions * feat: transfer mocks from `feature/braintrust` * feat: LLM criteria assertion * feat: braintrust evals workflow * fix: BRAINTRUST_PROJECT_ID * feat: `sql_similar` assertion * fix: `OPENAI_API_KEY` in workflow env * feat: split AssertionScorer into separate scorers * feat: remove tutorial eval * feat: 20 minute CI timeout * feat: category in test case metadata * feat: score with gpt-5 * refactor: dataset to own file, colocate scorers * feat: "gpt-5.2-2025-12-11" for llm as a judge * feat: SQL syntax scorer with `libpg-query` * feat: `evals:setup` and `evals:run` scripts * feat: `evals:setup` in CI * feat: human readable scorer names * chore: rename to "SQL Validity" * feat: add 2 "sql_generation" test cases * feat: update requiredTools in test cases * chore: ignore Cursor MCP config * feat: "Conciseness" score * feat: "Completeness" scorer * fix: generate-v4 test mocks * feat: serialize "steps" for scorer inputs * updated node mem options for typecheck * updated runner * remove ram update as actions handle this * feat: read `BRAINTRUST_PROJECT_ID` from secrets * feat: score helpfulness, remove old scorers * feat: separate `evals:run` and `evals:upload` scripts * feat: passthrough entire classifier result * feat: use live `search_docs` impl, store docs result in metadata * feat: reduce classifier options * feat: filter workflow by `run-evals` PR label or `master` branch * chore: cleanup stubbed mock tools * fix: checkout actual branch with `ref:` * fix: capture search_docs results from all content parts * feat: simplify sql syntax score calculation * feat: use AI SDK's UI message validator * docs: justification for relative `extends` * fix: cleanup leftover validatedMessages * doc: note mock token isn't secret for snyk * fix: mock ui message to pass validation * feat: revert ignoring Cursor MCP config Using `.git/info/exclude` instead until we have an opinion on this * feat: add "tsconfig" as shared-data devDependency, revert relative path in tsconfig * refactor: tool call parsing into function * Update apps/studio/evals/assistant.eval.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * refactor: organize mock schemas and tool factories --------- Co-authored-by: Ali Waseem <waseema393@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
|
626eb30e77 |
Assistant action orientated approach (#38806)
* update onboarding * update model and fix part issue * action orientated assistant * fix tool * lock * remove unused filter * fix tests * fix again * update package * update container * fix tests * refactor(ai assistant): break out message markdown and profile picture * wip * refactor(ai assistant): break up message component * refactor: break ai assistant message down into multiple files * refactor: simplify ReportBlock state * fix: styling of draggable report block header When the drag handle is showing, it overlaps with the block header. Decrease the opacity of the header so the handle can be seen and the two can be distinguished. * fix: minor tweaks to tool ui * refactor: simplify DisplayBlockRenderer state * fix: remove double deploy button in edge function block When the confirm footer is shown, the deploy button on the top right should be hidden (not just disabled) to avoid confusion. * refactor, test: message sanitization by opt-in level Refactor the message sanitization to have more type safety and be more testable. Add tests to ensure: - Message sanitization always runs on generate-v4 - Message sanitization correctly works by opt-in level * Fix conflicts in pnpm lock * Couple of nits and refactors * Revert casing for report block snippet * adjust sanitised prompt * Fix tests --------- Co-authored-by: Charis Lam <26616127+charislam@users.noreply.github.com> Co-authored-by: Joshen Lim <joshenlimek@gmail.com> |
