## Motivation
When Assistant runs a potentially destructive tool like `execute_sql`,
it stops the LLM request and prompts for client-side approval and
execution of the tool. After approval, a second request kicks off under
a separate trace. This has made scoring and
[Topics](https://www.braintrust.dev/blog/topics) classification
challenging, as the generated `output` is split across stateless
requests. The [span-level
scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans)
approach we've used thusfar (after the LLM call, we massage the result
into an `output` payload that's stuck onto the root span) has been
cumbersome and led to invalid scores / topics where only part of the
assistant response is considered. It's also inefficient, as we're
duplicating potentially large info (like the `search_docs` output) that
already exists within the trace.
An alternative to scoring spans is to [score
traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces).
Braintrust [best
practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices)
advise:
> Use span scope for evaluating individual operations or outputs. Use
trace scope for evaluating multi-turn conversations, overall workflow
completion, or when your scorer needs access to the full execution
context.
We've also received [direct
guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89)
from their team to use this approach.
## Changes
Migrates eval scorers from custom `AssistantEvalOutput` shape to
trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with
thread parsing that scores the full latest Assistant turn and passes
prior conversation separately where relevant.
Moves `execute_sql` and `deploy_edge_function` from client-side
execution after approval to AI SDK `needsApproval` + server-side
`execute()`. SQL results returned to the model are gated by AI opt-in
level, so row data is only included with `schema_and_log_and_data`;
otherwise the tool returns the no-data-permissions sentinel.
Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within
an "assistant" turn due to tool call requests/responses. For online
evals, this means we should configure automations to only score traces
with `metadata.isFinalStep = true` to ensure we're judging the complete
generated response.
Other minor kaizen changes:
- Renamed `promptProviderOptions` to `systemProviderOptions` to clarify
that this is associated with the "system" message and disambiguate from
the root `providerOptions`
- Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown`
span shapes from Braintrust, to more easily access typed inputs/output
on tool spans.
- Bumps AI SDK floor version `^6.0.116` → `^6.0.174`
- Tweaked the "Conciseness" scorer to not unfairly dock points for the
new `[called tool_name]` labels in serialized assistant response
## Verification
In the studio staging build, I asked Assistant to create a todos table
with 3 sample todos. I manually approved the `execute_sql` call and saw
Assistant generate text before & after the call.
In Braintrust I verified two traces were produced (see [filtered
logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})),
the first with `metadata.isFinalStep = false` and the second with
`metadata.isFinalStep = true`.
In the Braintrust staging scorers, I ran the preview Completeness scorer
on the second trace and verified it sees the complete Assistant response
including markers for tool calls ([link to
trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82))
<img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x"
src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a"
/>
I also tested the `deploy_edge_function` workflow and verified it still
prompts for permission and warns on deployment of existing functions.
**References**
- https://www.braintrust.dev/docs/evaluate/custom-code#score-traces
-
https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval
Supercedes https://github.com/supabase/supabase/pull/45556 and
https://github.com/supabase/supabase/pull/45339
Closes AI-473
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tool actions (SQL execution, edge-function deploy) now require
explicit user Approve/Deny before proceeding.
* **Improvements**
* Assistant pauses for approval responses before sending follow-ups,
giving clearer control over risky actions.
* Deploy/replace flows show confirmation and clearer replace warnings.
* Evaluation/scoring updated to use richer trace data for more accurate
assistant performance signals.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.
YES
## What kind of change does this PR introduce?
Bug fix for #42574
## What is the current behavior?
If you setup a self-hosted Supabase environment and ask local AI
assistant it is unable to interpret the results for executed SQL
queries.
## What is the new behavior?
Local AI assistant can now read schemas and interpret the results for
executed SQL queries.
## Additional context
I tested it using ollama with a smaller model which does not support
reasoning mode so It would be useful if someone can test it using an
actual OpenAI API KEY.
The main issue was that `executeSql` returns a wrapped `{ result: {...}
}` on `fallback-tools.ts` making the model think there's no result.
The solution is just to unwrap it like it's done at:
`apps/studio/data/database-functions/database-functions-query.ts`:
```
export async function getDatabaseFunctions(...) {
...
const { result } = await executeSql(...)
...
return result as DatabaseFunction[]
}
```
Also added a `false` default value in `execute_sql` tool schema so zod
validation does not reject a valid LLM tool call if the model omits the
field (which happened with the smaller model)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Optimized internal schema metadata query handling in AI tools.
* **Chores**
* Enhanced SQL execution tool robustness by providing a sensible default
for query type specification when not explicitly provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Andrey A. <56412611+aantti@users.noreply.github.com>
Upgrade vite and vitest to their latest major versions across the
monorepo, along with related packages.
**Changed:**
- `vite` catalog: `^7.3.2` → `^8.0.8` (Rolldown replaces esbuild/Rollup)
- `vitest` catalog: `^3.2.0` → `^4.1.4`
- `@vitejs/plugin-react`: `^4.3.4` → `^6.0.1`
- `@vitest/coverage-v8`: `^3.2.0` → `^4.1.4`
- `@vitest/ui`: `^3.2.0` → `^4.1.4`
- `vite-tsconfig-paths`: `^4.3.2` / `^5.1.4` → `^6.1.1`
**Pinned to vite 7:**
- `apps/lite-studio` — `@react-router/dev` hasn't declared vite 8
support yet
- `blocks/vue` — Nuxt plugins (`vite-plugin-inspect`, `vite-dev-rpc`,
`vite-hot-client`, `vite-plugin-vue-tracer`) haven't declared vite 8
support yet
**Test fixes for vitest 4 breaking changes:**
- **`apps/studio/lib/api/snippets.utils.test.ts`** — Replaced
`vi.mock('fs/promises')` automock with an explicit factory. Vitest 4's
automocking doesn't create mock functions for getter-based exports on
Node built-ins, so `mockedFS.access.mockResolvedValue` etc. were
`undefined`.
- **`apps/studio/lib/api/self-hosted/functions/index.test.ts`** —
Changed `mockReturnValue` to `mockImplementation(function() { ... })`
for a constructor mock. Vitest 4 no longer allows `mockReturnValue` when
the mock is called with `new`.
- **`apps/studio/tests/pages/api/mcp/index.test.ts`** — Changed arrow
function to regular `function` in `mockImplementation` for
`StreamableHTTPServerTransport`. Arrow functions can't be constructors,
and vitest 4 now enforces this.
- **`packages/ui-patterns/vitest.setup.ts`** — Changed `ResizeObserver`
mock from arrow function to regular `function` for the same constructor
enforcement reason. This was crashing Radix popover rendering in jsdom.
## To test
- `pnpm test:studio` — all 226 test files should pass
- `pnpm --filter ui-patterns vitest run` — all 183 tests should pass
- `pnpm --filter www test -- --run` — all 19 tests should pass
- `pnpm --filter ui vitest run` — all tests should pass
- `pnpm --filter dev-tools vitest run` — all tests should pass
- `pnpm --filter ai-commands vitest run` — all tests should pass
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Standardized and updated development tooling versions and version
sources for consistent installs across the repo (Vite, Vitest,
vite-tsconfig-paths and related plugins/catalog entries).
* **Tests**
* Improved test mocks and typings (updated mock
factories/implementations and tightened spy/type assertions) to increase
test reliability and compatibility with updated tooling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com>
Moves knowledge (RLS, Edge Functions, PostgreSQL best practices,
Realtime) out of the static system prompt and into a `load_knowledge`
tool the model calls on demand, reducing prompt bloat. This is a
temporary stopgap until the [standard Supabase
agent-skills](https://github.com/supabase/agent-skills) are ready for
integration in Assistant.
- New always-available `load_knowledge` tool added to
`rendering-tools.ts`
- Updated `Message.Parts.tsx` so the "Ran load_knowledge" chip renders
in chat
- System prompt replaces the four knowledge blobs with an `## Available
Knowledge` block and is hardened to load knowledge for given topics
- New "Knowledge Usage" scorer and `requiredKnowledge` assertions check
that knowledge loads as expected in test scenarios
- Filters GraphQL error responses out of `output.docs` before
faithfulness scoring to reduce noise
See "Knowledge Usage" scoring 100% in evals with no major regressions:
https://github.com/supabase/supabase/pull/44296#issuecomment-4145760236
Sample trace showing the tool in action
([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=project_logs&object_id=5a8d02e5-b3b6-40cc-ba76-ecee286478f4&r=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1&s=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1))
<img width="2192" height="1730" alt="CleanShot 2026-03-30 at 13 53
59@2x"
src="https://github.com/user-attachments/assets/f483767c-34e0-401c-8089-5b9834fe696a"
/>
**References**
- https://ai-sdk.dev/cookbook/guides/agent-skills
Closes AI-508
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added dynamic knowledge loading capability enabling the AI assistant
to retrieve on-demand information about PostgreSQL best practices, Row
Level Security, Edge Functions, and Realtime.
* **Bug Fixes**
* Improved search results filtering to exclude error responses in tool
outputs.
* **Tests**
* Enhanced evaluation metrics with knowledge usage scoring.
* Expanded test dataset cases to validate knowledge requirement
handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Context
Shifting more dashboard queries into pg-meta so that we centralize all
manually written queries in one place
Having them in packages/pg-meta also allows us to write tests for them
## To test
Just needs a smoke test on
- Table Editor
- Fetching entities
- Viewing definition
- SQL Editor
- View ongoing queries
- Abort queries
- Integrations
- Queues
- Database
- Migrations
-Triggers (Updating)
* feat: SQL correctness scorer, override mock tables
* feat: replace "SQL Correctness" with "SQL Identifier Quoting" scorer
* fix(prompt): discourage simulating confirmation of execute_sql tool
this is already handled at the UI layer
* fix(prompt): encourage quotes on identifiers with caps
* feat: move extractIdentifiers to own file, add tests
* chore: shorten tests
* feat: extract ColumnDef column names in extractIdentifiers
* refactor: sqlIdentifierQuotingScorer with more thorough checks
* refactor: consolidate into `sql-identifier-quoting.ts`
* feat: support mocking schemas, eval test case with case sensitive schema
* fix: test cases that don't match default mock schema
* chore: format
* feat(prompt): mention special characters and reserved words
* feat: optional description in metadata, test with special characters
* feat: consolidated comprehensive test case
* fix(prompt): revert conflicting instruction
* handling and mock data support
* admonition
* only show in empty state
* rabbit
* mock without local overrides
* remove admonition
* make incident banner more prominent
* remove mock data
* move to tool
* prettier
* fix(studio): get_active_incidents tool
---------
Co-authored-by: Charis Lam <26616127+charislam@users.noreply.github.com>
* update onboarding
* update model and fix part issue
* action orientated assistant
* fix tool
* lock
* remove unused filter
* fix tests
* fix again
* update package
* update container
* fix tests
* refactor(ai assistant): break out message markdown and profile picture
* wip
* refactor(ai assistant): break up message component
* refactor: break ai assistant message down into multiple files
* refactor: simplify ReportBlock state
* fix: styling of draggable report block header
When the drag handle is showing, it overlaps with the block header.
Decrease the opacity of the header so the handle can be seen and the two
can be distinguished.
* fix: minor tweaks to tool ui
* refactor: simplify DisplayBlockRenderer state
* fix: remove double deploy button in edge function block
When the confirm footer is shown, the deploy button on the top right should be
hidden (not just disabled) to avoid confusion.
* refactor, test: message sanitization by opt-in level
Refactor the message sanitization to have more type safety and be more testable.
Add tests to ensure:
- Message sanitization always runs on generate-v4
- Message sanitization correctly works by opt-in level
* Fix conflicts in pnpm lock
* Couple of nits and refactors
* Revert casing for report block snippet
* adjust sanitised prompt
* Fix tests
---------
Co-authored-by: Charis Lam <26616127+charislam@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* try a really long context window to maximize caching
* update examples
* attempt to update packages and useChat
* update endpoints
* update zod
* zod
* update to v5
* message update
* Revert "zod"
This reverts commit ec39bac6b6.
* revert zod
* zod i
* fix complete endpoints
* remove async
* change to content
* type cleanup
* Revert the package bumps to rebuild them.
* Bump zod to 2.25.76 in all packages.
* Bump openai in all packages.
* Bump ai and ai-related packages.
* Remove unneeded files.
* Fix the rest of the migration stuff.
* Prettier fixes.
* add policy list tool
* refactor
* ai sdk 5 fixes
* refactor complete endpoint
* edge function prompt
* remove example
* slight prompt change
* Minor clean up
* More clean up
---------
Co-authored-by: Jordi Enric <jordi.err@gmail.com>
Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
Matt Rossman