mirror of
https://github.com/supabase/supabase.git
synced 2026-10-05 09:25:06 +03:00
Matt Rossman
d143571586
## Motivation When Assistant runs a potentially destructive tool like `execute_sql`, it stops the LLM request and prompts for client-side approval and execution of the tool. After approval, a second request kicks off under a separate trace. This has made scoring and [Topics](https://www.braintrust.dev/blog/topics) classification challenging, as the generated `output` is split across stateless requests. The [span-level scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans) approach we've used thusfar (after the LLM call, we massage the result into an `output` payload that's stuck onto the root span) has been cumbersome and led to invalid scores / topics where only part of the assistant response is considered. It's also inefficient, as we're duplicating potentially large info (like the `search_docs` output) that already exists within the trace. An alternative to scoring spans is to [score traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces). Braintrust [best practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices) advise: > Use span scope for evaluating individual operations or outputs. Use trace scope for evaluating multi-turn conversations, overall workflow completion, or when your scorer needs access to the full execution context. We've also received [direct guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89) from their team to use this approach. ## Changes Migrates eval scorers from custom `AssistantEvalOutput` shape to trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with thread parsing that scores the full latest Assistant turn and passes prior conversation separately where relevant. Moves `execute_sql` and `deploy_edge_function` from client-side execution after approval to AI SDK `needsApproval` + server-side `execute()`. SQL results returned to the model are gated by AI opt-in level, so row data is only included with `schema_and_log_and_data`; otherwise the tool returns the no-data-permissions sentinel. Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within an "assistant" turn due to tool call requests/responses. For online evals, this means we should configure automations to only score traces with `metadata.isFinalStep = true` to ensure we're judging the complete generated response. Other minor kaizen changes: - Renamed `promptProviderOptions` to `systemProviderOptions` to clarify that this is associated with the "system" message and disambiguate from the root `providerOptions` - Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown` span shapes from Braintrust, to more easily access typed inputs/output on tool spans. - Bumps AI SDK floor version `^6.0.116` → `^6.0.174` - Tweaked the "Conciseness" scorer to not unfairly dock points for the new `[called tool_name]` labels in serialized assistant response ## Verification In the studio staging build, I asked Assistant to create a todos table with 3 sample todos. I manually approved the `execute_sql` call and saw Assistant generate text before & after the call. In Braintrust I verified two traces were produced (see [filtered logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})), the first with `metadata.isFinalStep = false` and the second with `metadata.isFinalStep = true`. In the Braintrust staging scorers, I ran the preview Completeness scorer on the second trace and verified it sees the complete Assistant response including markers for tool calls ([link to trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82)) <img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x" src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a" /> I also tested the `deploy_edge_function` workflow and verified it still prompts for permission and warns on deployment of existing functions. **References** - https://www.braintrust.dev/docs/evaluate/custom-code#score-traces - https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval Supercedes https://github.com/supabase/supabase/pull/45556 and https://github.com/supabase/supabase/pull/45339 Closes AI-473 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tool actions (SQL execution, edge-function deploy) now require explicit user Approve/Deny before proceeding. * **Improvements** * Assistant pauses for approval responses before sending follow-ups, giving clearer control over risky actions. * Deploy/replace flows show confirmation and clearer replace warnings. * Evaluation/scoring updated to use richer trace data for more accurate assistant performance signals. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Supabase
Supabase is the Postgres development platform. We're building the features of Firebase using enterprise-grade open source tools.
- Hosted Postgres Database. Docs
- Authentication and Authorization. Docs
- Auto-generated APIs.
- Functions.
- File Storage. Docs
- AI + Vector/Embeddings Toolkit. Docs
- Dashboard
Watch "releases" of this repo to get notified of major updates.
Documentation
For full documentation, visit supabase.com/docs
To see how to Contribute, visit Getting Started
Community & Support
- Community Forum. Best for: help with building, discussion about database best practices.
- GitHub Issues. Best for: bugs and errors you encounter using Supabase.
- Email Support. Best for: problems with your database or infrastructure.
- Discord. Best for: sharing your applications and hanging out with the community.
How it works
Supabase is a combination of open source tools. We’re building the features of Firebase using enterprise-grade, open source products. If the tools and communities exist, with an MIT, Apache 2, or equivalent open license, we will use and support that tool. If the tool doesn't exist, we build and open source it ourselves. Supabase is not a 1-to-1 mapping of Firebase. Our aim is to give developers a Firebase-like developer experience using open source tools.
Architecture
Supabase is a hosted platform. You can sign up and start using Supabase without installing anything. You can also self-host and develop locally.
- Postgres is an object-relational database system with over 30 years of active development that has earned it a strong reputation for reliability, feature robustness, and performance.
- Realtime is an Elixir server that allows you to listen to PostgreSQL inserts, updates, and deletes using websockets. Realtime polls Postgres' built-in replication functionality for database changes, converts changes to JSON, then broadcasts the JSON over websockets to authorized clients.
- PostgREST is a web server that turns your PostgreSQL database directly into a RESTful API.
- GoTrue is a JWT-based authentication API that simplifies user sign-ups, logins, and session management in your applications.
- Storage a RESTful API for managing files in S3, with Postgres handling permissions.
- pg_graphql a PostgreSQL extension that exposes a GraphQL API.
- postgres-meta is a RESTful API for managing your Postgres, allowing you to fetch tables, add roles, and run queries, etc.
- Kong is a cloud-native API gateway.
Client libraries
Our approach for client libraries is modular. Each sub-library is a standalone implementation for a single external system. This is one of the ways we support existing tools.
| Language | Client | Feature-Clients (bundled in Supabase client) | ||||
|---|---|---|---|---|---|---|
| Supabase | PostgREST | GoTrue | Realtime | Storage | Functions | |
| ⚡️ Official ⚡️ | ||||||
| JavaScript (TypeScript) | supabase-js | postgrest-js | auth-js | realtime-js | storage-js | functions-js |
| Flutter | supabase-flutter | postgrest-dart | gotrue-dart | realtime-dart | storage-dart | functions-dart |
| Swift | supabase-swift | postgrest-swift | auth-swift | realtime-swift | storage-swift | functions-swift |
| Python | supabase-py | postgrest-py | gotrue-py | realtime-py | storage-py | functions-py |
| 💚 Community 💚 | ||||||
| C# | supabase-csharp | postgrest-csharp | gotrue-csharp | realtime-csharp | storage-csharp | functions-csharp |
| Go | - | postgrest-go | gotrue-go | - | storage-go | functions-go |
| Java | - | - | gotrue-java | - | storage-java | - |
| Kotlin | supabase-kt | postgrest-kt | auth-kt | realtime-kt | storage-kt | functions-kt |
| Ruby | supabase-rb | postgrest-rb | - | - | - | - |
| Rust | - | postgrest-rs | - | - | - | - |
| Godot Engine (GDScript) | supabase-gdscript | - | - | - | - | - |
Badges
[](https://supabase.com)
<a href="https://supabase.com">
<img
width="168"
height="30"
src="https://supabase.com/badge-made-with-supabase.svg"
alt="Made with Supabase"
/>
</a>
[](https://supabase.com)
<a href="https://supabase.com">
<img
width="168"
height="30"
src="https://supabase.com/badge-made-with-supabase-dark.svg"
alt="Made with Supabase"
/>
</a>
Translations
- Arabic | العربية
- Albanian / Shqip
- Bangla / বাংলা
- Bulgarian / Български
- Catalan / Català
- Croatian / Hrvatski
- Czech / čeština
- Danish / Dansk
- Dutch / Nederlands
- English
- Estonian / eesti keel
- Finnish / Suomalainen
- French / Français
- German / Deutsch
- Greek / Ελληνικά
- Gujarati / ગુજરાતી
- Hebrew / עברית
- Hindi / हिंदी
- Hungarian / Magyar
- Nepali / नेपाली
- Indonesian / Bahasa Indonesia
- Italiano / Italian
- Japanese / 日本語
- Korean / 한국어
- Lithuanian / lietuvių
- Latvian / latviski
- Malay / Bahasa Malaysia
- Norwegian (Bokmål) / Norsk (Bokmål)
- Persian / فارسی
- Polish / Polski
- Portuguese / Português
- Portuguese (Brazilian) / Português Brasileiro
- Romanian / Română
- Russian / Pусский
- Serbian / Srpski
- Sinhala / සිංහල
- Slovak / slovenský
- Slovenian / Slovenščina
- Spanish / Español
- Simplified Chinese / 简体中文
- Swedish / Svenska
- Thai / ไทย
- Traditional Chinese / 繁體中文
- Turkish / Türkçe
- Ukrainian / Українська
- Vietnamese / Tiếng Việt
- List of translations