mirror of
https://github.com/supabase/supabase.git
synced 2026-10-11 04:15:04 +03:00
feat/github-copilot-plugin
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Saxon Fletcher
|
80da153450 |
Fix for AI Assistant query and deploy confirmation (#46052)
When Assistant requests confirmation to run a query or deploy an edge function if the user doesn't skip or run and instead sends a follow-up message it errors out. This allows follow-up messages and treats them as "skips" which means adjusting confirmation message state as part of the follow-up. This also uses toModelOutput to cleanse data based on permissions. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Enhanced tool approval workflow: pending approvals are now automatically resolved as denied when submitting new messages * Improved chat input state management with better handling of approval states * Customizable loading messages for tool operations * **Bug Fixes** * Fixed chat input availability during pending tool approval states * Improved tool execution feedback during approval workflows <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46052?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
212ccf8135 |
fix(ai): contextualize cron schedule as SQL writes, score in "Tool Usage" (#45997)
When Assistant tries to schedule crons in read-only mode, it succeeds but creates the jobs under the `supabase_read_only_user`. This causes permission errors when user try to delete or unschedule them from the Cron dashboard. The root fix will be to enforce read-only transactions for that user. In the meantime, this PR steers Assistant to avoid the mistake. **Changes** - Prompts `execute_sql` to treat side-effecting function calls such as `cron.schedule()` as write queries. - Adds tool input assertions for "Tool Usage" scorer and a focused cron regression eval. - Updates eval mocks to show pg_cron extension as installed so it can call `cron.schedule()` **Verification** See [this trace](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=4a9e8c0e-83b7-4555-8502-365662c3ec8e&r=e041e69b-b70f-41d1-b88c-e8f7888c3de5&s=e041e69b-b70f-41d1-b88c-e8f7888c3de5) from Braintrust where the new eval passes "Tool Usage", correctly using `isWriteQuery` for the `cron.schedule()` Closes AI-737 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tool evaluation now validates tool inputs (including exact and substring matches) in addition to tool presence. * **Tests** * Added a test confirming cron-scheduling behavior and that SQL scheduling/enqueue calls are treated as write operations. * **Chores** * Added pg_cron to mock extension data. * Clarified description that SQL calls with side effects should be treated as writes. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/45997?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Charis
|
12869fcd9f |
studio: flip executeSql signature to SafeSqlFragment (7/7) (#46007)
## Summary Final PR in the SafeSql migration stack. Stacked on top of #46006. Tightens `executeSql`'s `sql` parameter from `string` to `SafeSqlFragment`. Any future raw-string caller is now a compile error — the SafeSql safety property becomes structural rather than convention-based. Also adapts the AI `execute_sql` tool to promote AI-generated SQL via `acceptUntrustedSql(untrustedSql(sql))` inside the `execute` callback. The tool's existing \`needsApproval: true\` gate ensures `execute` only runs after the user has explicitly approved — that approval is the gesture that promotes untrusted to safe. ## Test plan - [x] `pnpm typecheck` passes - [x] Grep for any remaining raw-string `executeSql` calls in `apps/studio` returns nothing - [x] Dev-server smoke: AI tool approval flow executes SQL |
||
|
Matt Rossman
|
d143571586 |
feat(assistant): trace-level scorers + server-side tool execution with needsApproval (#45654)
## Motivation When Assistant runs a potentially destructive tool like `execute_sql`, it stops the LLM request and prompts for client-side approval and execution of the tool. After approval, a second request kicks off under a separate trace. This has made scoring and [Topics](https://www.braintrust.dev/blog/topics) classification challenging, as the generated `output` is split across stateless requests. The [span-level scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans) approach we've used thusfar (after the LLM call, we massage the result into an `output` payload that's stuck onto the root span) has been cumbersome and led to invalid scores / topics where only part of the assistant response is considered. It's also inefficient, as we're duplicating potentially large info (like the `search_docs` output) that already exists within the trace. An alternative to scoring spans is to [score traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces). Braintrust [best practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices) advise: > Use span scope for evaluating individual operations or outputs. Use trace scope for evaluating multi-turn conversations, overall workflow completion, or when your scorer needs access to the full execution context. We've also received [direct guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89) from their team to use this approach. ## Changes Migrates eval scorers from custom `AssistantEvalOutput` shape to trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with thread parsing that scores the full latest Assistant turn and passes prior conversation separately where relevant. Moves `execute_sql` and `deploy_edge_function` from client-side execution after approval to AI SDK `needsApproval` + server-side `execute()`. SQL results returned to the model are gated by AI opt-in level, so row data is only included with `schema_and_log_and_data`; otherwise the tool returns the no-data-permissions sentinel. Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within an "assistant" turn due to tool call requests/responses. For online evals, this means we should configure automations to only score traces with `metadata.isFinalStep = true` to ensure we're judging the complete generated response. Other minor kaizen changes: - Renamed `promptProviderOptions` to `systemProviderOptions` to clarify that this is associated with the "system" message and disambiguate from the root `providerOptions` - Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown` span shapes from Braintrust, to more easily access typed inputs/output on tool spans. - Bumps AI SDK floor version `^6.0.116` → `^6.0.174` - Tweaked the "Conciseness" scorer to not unfairly dock points for the new `[called tool_name]` labels in serialized assistant response ## Verification In the studio staging build, I asked Assistant to create a todos table with 3 sample todos. I manually approved the `execute_sql` call and saw Assistant generate text before & after the call. In Braintrust I verified two traces were produced (see [filtered logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})), the first with `metadata.isFinalStep = false` and the second with `metadata.isFinalStep = true`. In the Braintrust staging scorers, I ran the preview Completeness scorer on the second trace and verified it sees the complete Assistant response including markers for tool calls ([link to trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82)) <img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x" src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a" /> I also tested the `deploy_edge_function` workflow and verified it still prompts for permission and warns on deployment of existing functions. **References** - https://www.braintrust.dev/docs/evaluate/custom-code#score-traces - https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval Supercedes https://github.com/supabase/supabase/pull/45556 and https://github.com/supabase/supabase/pull/45339 Closes AI-473 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tool actions (SQL execution, edge-function deploy) now require explicit user Approve/Deny before proceeding. * **Improvements** * Assistant pauses for approval responses before sending follow-ups, giving clearer control over risky actions. * Deploy/replace flows show confirmation and clearer replace warnings. * Evaluation/scoring updated to use richer trace data for more accurate assistant performance signals. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Manuel RubioandAndrey A.
|
667be50982 |
fix(self-hosted): local AI assistant unable to read schemas and interpret/execute SQL (#44757)
## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Bug fix for #42574 ## What is the current behavior? If you setup a self-hosted Supabase environment and ask local AI assistant it is unable to interpret the results for executed SQL queries. ## What is the new behavior? Local AI assistant can now read schemas and interpret the results for executed SQL queries. ## Additional context I tested it using ollama with a smaller model which does not support reasoning mode so It would be useful if someone can test it using an actual OpenAI API KEY. The main issue was that `executeSql` returns a wrapped `{ result: {...} }` on `fallback-tools.ts` making the model think there's no result. The solution is just to unwrap it like it's done at: `apps/studio/data/database-functions/database-functions-query.ts`: ``` export async function getDatabaseFunctions(...) { ... const { result } = await executeSql(...) ... return result as DatabaseFunction[] } ``` Also added a `false` default value in `execute_sql` tool schema so zod validation does not reject a valid LLM tool call if the model omits the field (which happened with the smaller model) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Refactor** * Optimized internal schema metadata query handling in AI tools. * **Chores** * Enhanced SQL execution tool robustness by providing a sensible default for query type specification when not explicitly provided. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Andrey A. <56412611+aantti@users.noreply.github.com> |
||
|
Charis
|
205cbe7d26 | chore(studio}: enforce import order, remove bare import specifiers (#44585) | ||
|
Matt Rossman
|
82deff37de |
feat(assistant): lazy load topic knowledge via load_knowledge tool (#44296)
Moves knowledge (RLS, Edge Functions, PostgreSQL best practices, Realtime) out of the static system prompt and into a `load_knowledge` tool the model calls on demand, reducing prompt bloat. This is a temporary stopgap until the [standard Supabase agent-skills](https://github.com/supabase/agent-skills) are ready for integration in Assistant. - New always-available `load_knowledge` tool added to `rendering-tools.ts` - Updated `Message.Parts.tsx` so the "Ran load_knowledge" chip renders in chat - System prompt replaces the four knowledge blobs with an `## Available Knowledge` block and is hardened to load knowledge for given topics - New "Knowledge Usage" scorer and `requiredKnowledge` assertions check that knowledge loads as expected in test scenarios - Filters GraphQL error responses out of `output.docs` before faithfulness scoring to reduce noise See "Knowledge Usage" scoring 100% in evals with no major regressions: https://github.com/supabase/supabase/pull/44296#issuecomment-4145760236 Sample trace showing the tool in action ([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=project_logs&object_id=5a8d02e5-b3b6-40cc-ba76-ecee286478f4&r=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1&s=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1)) <img width="2192" height="1730" alt="CleanShot 2026-03-30 at 13 53 59@2x" src="https://github.com/user-attachments/assets/f483767c-34e0-401c-8089-5b9834fe696a" /> **References** - https://ai-sdk.dev/cookbook/guides/agent-skills Closes AI-508 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added dynamic knowledge loading capability enabling the AI assistant to retrieve on-demand information about PostgreSQL best practices, Row Level Security, Edge Functions, and Realtime. * **Bug Fixes** * Improved search results filtering to exclude error responses in tool outputs. * **Tests** * Enhanced evaluation metrics with knowledge usage scoring. * Expanded test dataset cases to validate knowledge requirement handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
