## Motivation
When Assistant runs a potentially destructive tool like `execute_sql`,
it stops the LLM request and prompts for client-side approval and
execution of the tool. After approval, a second request kicks off under
a separate trace. This has made scoring and
[Topics](https://www.braintrust.dev/blog/topics) classification
challenging, as the generated `output` is split across stateless
requests. The [span-level
scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans)
approach we've used thusfar (after the LLM call, we massage the result
into an `output` payload that's stuck onto the root span) has been
cumbersome and led to invalid scores / topics where only part of the
assistant response is considered. It's also inefficient, as we're
duplicating potentially large info (like the `search_docs` output) that
already exists within the trace.
An alternative to scoring spans is to [score
traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces).
Braintrust [best
practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices)
advise:
> Use span scope for evaluating individual operations or outputs. Use
trace scope for evaluating multi-turn conversations, overall workflow
completion, or when your scorer needs access to the full execution
context.
We've also received [direct
guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89)
from their team to use this approach.
## Changes
Migrates eval scorers from custom `AssistantEvalOutput` shape to
trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with
thread parsing that scores the full latest Assistant turn and passes
prior conversation separately where relevant.
Moves `execute_sql` and `deploy_edge_function` from client-side
execution after approval to AI SDK `needsApproval` + server-side
`execute()`. SQL results returned to the model are gated by AI opt-in
level, so row data is only included with `schema_and_log_and_data`;
otherwise the tool returns the no-data-permissions sentinel.
Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within
an "assistant" turn due to tool call requests/responses. For online
evals, this means we should configure automations to only score traces
with `metadata.isFinalStep = true` to ensure we're judging the complete
generated response.
Other minor kaizen changes:
- Renamed `promptProviderOptions` to `systemProviderOptions` to clarify
that this is associated with the "system" message and disambiguate from
the root `providerOptions`
- Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown`
span shapes from Braintrust, to more easily access typed inputs/output
on tool spans.
- Bumps AI SDK floor version `^6.0.116` → `^6.0.174`
- Tweaked the "Conciseness" scorer to not unfairly dock points for the
new `[called tool_name]` labels in serialized assistant response
## Verification
In the studio staging build, I asked Assistant to create a todos table
with 3 sample todos. I manually approved the `execute_sql` call and saw
Assistant generate text before & after the call.
In Braintrust I verified two traces were produced (see [filtered
logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})),
the first with `metadata.isFinalStep = false` and the second with
`metadata.isFinalStep = true`.
In the Braintrust staging scorers, I ran the preview Completeness scorer
on the second trace and verified it sees the complete Assistant response
including markers for tool calls ([link to
trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82))
<img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x"
src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a"
/>
I also tested the `deploy_edge_function` workflow and verified it still
prompts for permission and warns on deployment of existing functions.
**References**
- https://www.braintrust.dev/docs/evaluate/custom-code#score-traces
-
https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval
Supercedes https://github.com/supabase/supabase/pull/45556 and
https://github.com/supabase/supabase/pull/45339
Closes AI-473
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tool actions (SQL execution, edge-function deploy) now require
explicit user Approve/Deny before proceeding.
* **Improvements**
* Assistant pauses for approval responses before sending follow-ups,
giving clearer control over risky actions.
* Deploy/replace flows show confirmation and clearer replace warnings.
* Evaluation/scoring updated to use richer trace data for more accurate
assistant performance signals.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
We're exploring support for newer models like
[gpt-5.4-nano](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/)
in Assistant. This model doesn't support the `'minimal'` reasoning
effort level we use for gpt-5-mini which leads to vague errors.
<img width="595" height="263" alt="CleanShot 2026-03-18 at 17 13 05@2x"
src="https://github.com/user-attachments/assets/cf7c2370-322d-4a8a-be55-23e680db0aa0"
/>
Also, we've [previously
discussed](https://supabase.slack.com/archives/C0161K73J1J/p1771544464850199?thread_ts=1771493920.775699&cid=C0161K73J1J)
that reasoning adds unnecessary latency to otherwise simple AI
completion endpoints like `title-v2`. We want more control of reasoning
level independent of model/endpoint.
This PR aims to solve both problems by:
- making reasoning effort configurable on a per-request basis
- adding compile-time guardrails to prevent selecting an incompatible
reasoning level for models
- adding a `DEFAULT_COMPLETION_MODEL` with minimal reasoning that we can
update with newer models that support disabling reasoning (independent
of Assistant chat model reasoning)
Other improvements to our model config logic:
- Fixes bug in `onboarding/design.ts` and `assistant.eval.ts` where
`providerOptions` was being dropped
- `getModel()` now returns a bundled `modelParams` object (spread into
AI SDK calls) so `providerOptions` can't be accidentally omitted (this
[has happened
before](https://supabase.slack.com/archives/C0161K73J1J/p1771518443534309?thread_ts=1771493920.775699&cid=C0161K73J1J))
- Introduces an `ASSISTANT_MODELS` registry as a single source of truth
for assistant model config, eliminating hardcoded model IDs across the
codebase
- Aligns free/pro model conditional logic with `assistant.advance_model`
entitlement naming conventions instead of the `isLimited` pattern
- Adds `console.error` logging of Assistant stream errors so we can
interpret reasoning effort compatibility errors in the future (instead
of just opaque "Sorry, I'm having trouble responding right now" card)
- Removes unnecessary type casts and generally making the model config
logic stricter
- Removes pre-existing dead code: `anthropic` provider variant in
`GetModelParams` / `PROVIDERS` registry that was never implemented in
`getModel()`
Now if you try to select an unsupported reasoning level you get a type
error:
<img width="1306" height="320" alt="CleanShot 2026-03-20 at 14 37 24@2x"
src="https://github.com/user-attachments/assets/a6ac234b-5ea5-4d81-8e01-ac4be34a0800"
/>
And if for some reason an invalid reasoning level slips through, you now
get a server-side error surfacing the issue:
<img width="1268" height="204" alt="CleanShot 2026-03-20 at 14 58 14@2x"
src="https://github.com/user-attachments/assets/aadc1b7a-9495-475f-9741-39979bd27cd7"
/>
I've tested gpt-5 and gpt-5-mini are still working on the staging
preview and verified the models were selected properly in Braintrust
logs. Both models are available on my Pro test account, and my Free test
account shows the Pro upgrade CTA.
Closes AI-446
Closes AI-551
* try a really long context window to maximize caching
* update examples
* attempt to update packages and useChat
* update endpoints
* update zod
* zod
* update to v5
* message update
* Revert "zod"
This reverts commit ec39bac6b6.
* revert zod
* zod i
* fix complete endpoints
* remove async
* change to content
* type cleanup
* Revert the package bumps to rebuild them.
* Bump zod to 2.25.76 in all packages.
* Bump openai in all packages.
* Bump ai and ai-related packages.
* Remove unneeded files.
* Fix the rest of the migration stuff.
* Prettier fixes.
* testing new gptoss models
* sonnet as pro
* back to 3.7
* revert test
* add policy list tool
* fix
* refactor
* ai sdk 5 fixes
* refactor complete endpoint
* edge function prompt
* remove example
* use limited model for completions
* remove duplicate
* Update bedrock.ts
Co-authored-by: divit <27228526+delgado3d@users.noreply.github.com>
* add default check
* only add prefix if more then one region
* temp use api
* refactor to use openai api
* test
* remove unused import
* Prettier lint
* Fix test
* revert back to bedrock
* Tiny style fix for spacing of code blocks from edit message button
* try generate text normally
* prompt adjust
* add list style to prompt
* prompt adjust
* prompt adjust emphasise display query
* update prompts
* prompt adjust
* remove package
* prompt adjust
---------
Co-authored-by: Jordi Enric <jordi.err@gmail.com>
Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
Co-authored-by: divit <27228526+delgado3d@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Clean up usage of newOrgAiOptIn and useBedrockAssistant feature flags
* Remove all OpenAI endpoints
* Fix for self-hosted
* Default isLimited to false
* Update PG meta tests
* Fix unit tests for model
* Revert pg meta tests
* Fix test
---------
Co-authored-by: Alaister Young <alaister@users.noreply.github.com>
* wip
* feat: distribute bedrock requests across regions
* fix: tests and type errors
* chore: remove AWS_BEDROCK_REGION from turbo.json
* fix: vercel oidc provider to fallback in credential chain
---------
Co-authored-by: delgado3d <27228526+delgado3d@users.noreply.github.com>
* step 1
* use mcp
* system prompt
* clean up
* space
* clean up
* add three state opt in toggle
* clean up
* todo
* hooks
* refactor opt in form and modal
* refinements
* add bedrock
* remove console
* update mcp util
* use bedrock
* remove openai sdk package
* re-add tools
* update complete endpoints
* fix: ai assistant markdown formatting (#35541)
* fix: mcp types and project ref (#35540)
* feat: more flexible aws credential provider (#35538)
* feat: more flexible aws credential provider
* fix: add AWS_REGION to turbo env vars
* change to allowed
* update complete endpoints
* add an additional permission
* refinements
* use claud 4
* legal copy changes
* update other ai functions to use bedrock
* update generate v3 copy
* remove generate sql modal
* fixes for query block
* re-add dragging to reports
* clean up
* add open ai edge function example
* use handle error from fetchers
* remove schema and lean on tools
* copy
* Assistant MCP tests (#36049)
* feat: refactor and test mcp and model logic
* fix: remove get_project tool
* fix: remove additional get_project tool references
* update copy
* Clean up, fixes, refactors
* oops
* Float errors from AI endpoionts as toasts
* Use a env var AWS_BEDROCK_PROFILE for bedrock.
* Rename the env var for AWS bedrock profile.
* feat: support custom aws bedrock env vars
* chore: add comments explaining aws credential chain
* MCP Self Hosted Check (#36185)
support self hosted
* feat: bedrock auth via vercel oidc
* Fix broken unit test
* Feeeex
* Refactor useOrgOptedIntoAi
* Remove useDisallowHipaa hook
* small system prompt change
* readd vercel packages
* fix self hosted
* increase max duration
* try more direct prompt
* max duration 90
* reduce max steps and add loading
* mono font
* backwards compat styling
* Chore/limit number of messages sent to assistant (#36388)
* Limit number of historical messages that get sent to assistant
* Update max chat history to 5
* alignment
* bump mcp server version
* Add feature flag for opt in tags (#36466)
* Add feature flag for opt in tags
* Add one more check
* security section system prompt
* rely on default link and replace image markdown
* Add custom link component to assistant message block (#36527)
* Add custom link component to assistant message block
* Update based on feedback
* Render plain text if URL is deemed unsafe
* fix mcp tools and parse data (#36593)
* Update Admonition for AI Assistant for when opt in is re-enabled (#36663)
* Update Admonition for AI Assistant for when opt in is re-enabled
* Update
* Smol fix
* Fix TS
* Tiny
---------
Co-authored-by: Greg Richardson <greg.nmr@gmail.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
Matt Rossman