mirror of
https://github.com/supabase/supabase.git
synced 2026-10-13 13:19:09 +03:00
docs/debugging-guide
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matt Rossman
|
65fab30935 |
feat(ai): judge tool inputs, add storage guidance and permissive RLS evals (#46168)
Adding broad RLS policies to public buckets can cause users to expose more than they expected, like the ability to list all profile pictures on an app. This patches Assistant with knowledge to follow our latest guidance on restrictive RLS policies for storage buckets https://github.com/supabase/supabase/pull/46172 **Changes** - Adds Storage bucket evals for public website assets and avatar access patterns to distinguish public vs private bucket use cases - Adds eval for overly permissive table policies - Adds `storage` knowledge so Assistant distinguishes public buckets, private buckets, object reads, and object listing. - Adds `includeToolCallInputs` option for scorer transcripts so LLM judges can evaluate proposed SQL/tool actions. - Bumps max step count to 10 since storage knowledge may incur another tool call (also 10 is recommended [here](https://vercel.com/academy/ai-sdk/multi-step-and-generative-ui#why-multi-step-is-required) for complex multi-tool scenarios) **References** - https://supabase.com/docs/guides/storage/buckets/fundamentals#public-buckets - https://supabase.com/docs/guides/storage/security/access-control - https://github.com/supabase/supabase/pull/46172 **Notes:** - These prompt tweaks are not meant to be exhaustive fixes, they are mainly hotfixes intended to hold us out until these cases can be addressed more deeply in skills/docs and tracked in a central evals Closes AI-676 Closes AI-756 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Storage knowledge resource for the assistant covering Supabase Storage access patterns and RLS guidance. * Added three evaluation cases: two for Storage (marketing assets, avatars) and one for RLS policy generation for user profiles. * **Improvements** * Evaluators now include tool call inputs when judging conversations. * Assistant prompts and generation enhanced with richer Storage/RLS guidance and extended streaming limits. * **Tests** * Added test ensuring tool call inputs are included in serialized thread context. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46168?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
212ccf8135 |
fix(ai): contextualize cron schedule as SQL writes, score in "Tool Usage" (#45997)
When Assistant tries to schedule crons in read-only mode, it succeeds but creates the jobs under the `supabase_read_only_user`. This causes permission errors when user try to delete or unschedule them from the Cron dashboard. The root fix will be to enforce read-only transactions for that user. In the meantime, this PR steers Assistant to avoid the mistake. **Changes** - Prompts `execute_sql` to treat side-effecting function calls such as `cron.schedule()` as write queries. - Adds tool input assertions for "Tool Usage" scorer and a focused cron regression eval. - Updates eval mocks to show pg_cron extension as installed so it can call `cron.schedule()` **Verification** See [this trace](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=4a9e8c0e-83b7-4555-8502-365662c3ec8e&r=e041e69b-b70f-41d1-b88c-e8f7888c3de5&s=e041e69b-b70f-41d1-b88c-e8f7888c3de5) from Braintrust where the new eval passes "Tool Usage", correctly using `isWriteQuery` for the `cron.schedule()` Closes AI-737 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tool evaluation now validates tool inputs (including exact and substring matches) in addition to tool presence. * **Tests** * Added a test confirming cron-scheduling behavior and that SQL scheduling/enqueue calls are treated as write operations. * **Chores** * Added pg_cron to mock extension data. * Clarified description that SQL calls with side effects should be treated as writes. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/45997?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
d143571586 |
feat(assistant): trace-level scorers + server-side tool execution with needsApproval (#45654)
## Motivation When Assistant runs a potentially destructive tool like `execute_sql`, it stops the LLM request and prompts for client-side approval and execution of the tool. After approval, a second request kicks off under a separate trace. This has made scoring and [Topics](https://www.braintrust.dev/blog/topics) classification challenging, as the generated `output` is split across stateless requests. The [span-level scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans) approach we've used thusfar (after the LLM call, we massage the result into an `output` payload that's stuck onto the root span) has been cumbersome and led to invalid scores / topics where only part of the assistant response is considered. It's also inefficient, as we're duplicating potentially large info (like the `search_docs` output) that already exists within the trace. An alternative to scoring spans is to [score traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces). Braintrust [best practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices) advise: > Use span scope for evaluating individual operations or outputs. Use trace scope for evaluating multi-turn conversations, overall workflow completion, or when your scorer needs access to the full execution context. We've also received [direct guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89) from their team to use this approach. ## Changes Migrates eval scorers from custom `AssistantEvalOutput` shape to trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with thread parsing that scores the full latest Assistant turn and passes prior conversation separately where relevant. Moves `execute_sql` and `deploy_edge_function` from client-side execution after approval to AI SDK `needsApproval` + server-side `execute()`. SQL results returned to the model are gated by AI opt-in level, so row data is only included with `schema_and_log_and_data`; otherwise the tool returns the no-data-permissions sentinel. Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within an "assistant" turn due to tool call requests/responses. For online evals, this means we should configure automations to only score traces with `metadata.isFinalStep = true` to ensure we're judging the complete generated response. Other minor kaizen changes: - Renamed `promptProviderOptions` to `systemProviderOptions` to clarify that this is associated with the "system" message and disambiguate from the root `providerOptions` - Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown` span shapes from Braintrust, to more easily access typed inputs/output on tool spans. - Bumps AI SDK floor version `^6.0.116` → `^6.0.174` - Tweaked the "Conciseness" scorer to not unfairly dock points for the new `[called tool_name]` labels in serialized assistant response ## Verification In the studio staging build, I asked Assistant to create a todos table with 3 sample todos. I manually approved the `execute_sql` call and saw Assistant generate text before & after the call. In Braintrust I verified two traces were produced (see [filtered logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})), the first with `metadata.isFinalStep = false` and the second with `metadata.isFinalStep = true`. In the Braintrust staging scorers, I ran the preview Completeness scorer on the second trace and verified it sees the complete Assistant response including markers for tool calls ([link to trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82)) <img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x" src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a" /> I also tested the `deploy_edge_function` workflow and verified it still prompts for permission and warns on deployment of existing functions. **References** - https://www.braintrust.dev/docs/evaluate/custom-code#score-traces - https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval Supercedes https://github.com/supabase/supabase/pull/45556 and https://github.com/supabase/supabase/pull/45339 Closes AI-473 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tool actions (SQL execution, edge-function deploy) now require explicit user Approve/Deny before proceeding. * **Improvements** * Assistant pauses for approval responses before sending follow-ups, giving clearer control over risky actions. * Deploy/replace flows show confirmation and clearer replace warnings. * Evaluation/scoring updated to use richer trace data for more accurate assistant performance signals. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Greg Richardson
|
538f9e3e82 |
fix: prevent AI assistant from soliciting sensitive creds (#45692)
Adds prompt guardrails and evals to prevent the AI assistant from asking users to share sensitive data (API keys, `.env` contents, etc.) and to warn when credentials are shared. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Stronger safety behavior: assistant now refuses requests to share full environment files, asks for variable names only, and directs users to secure secret-management tooling. * Immediate warning and guidance if credentials or other sensitive values are pasted in chat, without repeating exposed secrets. * **Behavior** * Clarified evaluation rules so responses more consistently follow the new safety guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Greg Richardson
|
5f8906a20e |
fix: add destructive operation guardrails to AI assistant (#45194)
Prevents the AI assistant from helping with local git/filesystem operations, and adds explicit warnings before irreversible database operations (DROP TABLE, DELETE without WHERE, etc.). Adds a `safetyScorer` and eval cases to cover these behaviours. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a Safety metric to evaluations so assistant responses are scored for safe handling of destructive or risky requests * Assistant guidance updated to refuse destructive local VCS/filesystem actions and require clear warnings for irreversible database operations * **Tests** * Added evaluation cases covering safe refusals, clear warnings, and correct handling of destructive or risky prompts * **Chores** * Enabled Safety metric in online evaluation manifests/handlers <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Charis
|
205cbe7d26 | chore(studio}: enforce import order, remove bare import specifiers (#44585) | ||
|
Matt Rossman
|
82deff37de |
feat(assistant): lazy load topic knowledge via load_knowledge tool (#44296)
Moves knowledge (RLS, Edge Functions, PostgreSQL best practices, Realtime) out of the static system prompt and into a `load_knowledge` tool the model calls on demand, reducing prompt bloat. This is a temporary stopgap until the [standard Supabase agent-skills](https://github.com/supabase/agent-skills) are ready for integration in Assistant. - New always-available `load_knowledge` tool added to `rendering-tools.ts` - Updated `Message.Parts.tsx` so the "Ran load_knowledge" chip renders in chat - System prompt replaces the four knowledge blobs with an `## Available Knowledge` block and is hardened to load knowledge for given topics - New "Knowledge Usage" scorer and `requiredKnowledge` assertions check that knowledge loads as expected in test scenarios - Filters GraphQL error responses out of `output.docs` before faithfulness scoring to reduce noise See "Knowledge Usage" scoring 100% in evals with no major regressions: https://github.com/supabase/supabase/pull/44296#issuecomment-4145760236 Sample trace showing the tool in action ([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=project_logs&object_id=5a8d02e5-b3b6-40cc-ba76-ecee286478f4&r=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1&s=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1)) <img width="2192" height="1730" alt="CleanShot 2026-03-30 at 13 53 59@2x" src="https://github.com/user-attachments/assets/f483767c-34e0-401c-8089-5b9834fe696a" /> **References** - https://ai-sdk.dev/cookbook/guides/agent-skills Closes AI-508 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added dynamic knowledge loading capability enabling the AI assistant to retrieve on-demand information about PostgreSQL best practices, Row Level Security, Edge Functions, and Realtime. * **Bug Fixes** * Improved search results filtering to exclude error responses in tool outputs. * **Tests** * Enhanced evaluation metrics with knowledge usage scoring. * Expanded test dataset cases to validate knowledge requirement handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
e4a9b6882c |
fix(assistant): use extractTextOnly in conciseness scorer (#43612)
`concisenessScorer` was passing full serialized text + tool calls JSON to the LLM judge (SQL queries, GraphQL payloads, etc.). Switches to `extractTextOnly` so the judge only evaluates text the user actually sees. Prerequisite for https://github.com/supabase/supabase/pull/43613 to set a fair conciseness baseline score. Ref AI-402 |
||
|
Matt Rossman
|
517171b246 |
feat(assistant): online evals support and CI workflows (#43194)
Lays groundwork for online evals on Assistant chat logs. https://www.braintrust.dev/docs/observe/score-online ### Changes - New workflows: - `braintrust-scorers-deploy.yml` keeps prod scorers in sync on push to `master` - `braintrust-preview-scorers-deploy.yml` deploys preview scorers to the staging project for PRs labeled `preview-scorers`, posting a comment with scorer links ([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222)) - `braintrust-preview-scorers-cleanup.yml` deletes preview scorers when the PR is closed ([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000749847)) - Adds `evals/scorer-online.ts` entry point invoked with `pnpm scorers:deploy`, registering scorers for online evals in the Braintrust "Assistant" project - Refactors scorer code to separate online-compatible scorers (`scorer-online.ts`) from WASM-dependent ones (`scorer-wasm.ts`) - "URL Validity" scorer now only checks Supabase domains to prevent requests to untrusted origins - Span `input` is now shaped `{ prompt: string }` instead of plain `string` for compatibility with offline eval scorers - Env vars `BRAINTRUST_STAGING_PROJECT_ID` and `BRAINTRUST_PROJECT_ID` configured in GitHub repo settings - `generateAssistantResponse` now uses `startSpan` + `withCurrent` instead of `traced()` to manually manage the root span lifecycle — this ensures `onFinish` logs output to the span _before_ `span.end()` is called, which is when Braintrust triggers scoring automations ### Online Scorers We share scoring logic across offline and online evals, but some of our scorers aren't transferrable to an "online" setting due to runtime challenges or ground truth requirements. **Supported** - Goal Completion - Conciseness - Completeness - Docs Faithfulness - URL Validity **Unsupported** - Correctness (requires ground truth output) - Tool Usage (requires ground truth requiredTools) - SQL Syntax (uses libpg-query WASM) - SQL Identifier Quoting (uses libpg-query WASM) ### How to use these scorers Going forward if you want to add/edit online eval scorers, add the `preview-scorers` label to a PR. This deploys scorers to the [Assistant (Staging Scorers)](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)?v=Overview) project in Braintrust with branch-specific slugs, and comments on the PR ([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222)). From the Braintrust dashboard you can "Test" the scorer with traces from any project. <img width="1866" height="528" alt="CleanShot 2026-03-05 at 15 15 00@2x" src="https://github.com/user-attachments/assets/4f15cebc-3f2d-4e8a-9ee2-fe8ef7bf4199" /> Once merged, scorers are deployed to the primary [Assistant](https://www.braintrust.dev/app/supabase.io/p/Assistant) project, and preview scorers are deleted from the staging project. Down the road, scorers on the Assistant project will run automatically on a sample of production traces. Closes AI-437 |
||
|
Matt Rossman
|
4b8bab4d14 |
feat(assistant): score URL validity and fix support ticket URL guidance (#42227)
**Logic changes** - Adds function in `helpers.ts` to extract URLs from text via regex - I also considering using a library like [linkify-it](https://www.npmjs.com/package/linkify-it) for this but figured it's not worth the extra dep - Adds associated tests in `helpers.test.ts` - Adds "URL Validity" scorer which performs a HEAD request for links in Assistant response text and determins what portion of links have `.ok` responses - Adds eval case to check correctness of support ticket URL answers **Prompt changes** - Informs Assistant of https://supabase.com/dashboard/support/new being the URL to create support tickets - Encourages Assistant to "self-debug" issues before directing users to create support tickets See [Eval Report](https://github.com/supabase/supabase/pull/42227#issuecomment-3807772871) and [Correctness](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=1ad0f9b0-5adb-436c-9812-a87aac62c036&r=1ef13459-a98c-4904-925e-6d81276cebb2&s=dbe5c607-a560-462b-8745-41d430744431) analysis for new support ticket test case. Resolves AI-384 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added URL validity scoring to evaluations and helper utilities for extracting/cleaning URLs. * Added evaluation cases for support-ticket URL handling and OAuth callback guidance. * **Documentation** * Updated assistant guidance to prefer self-resolution, include support-ticket direction, clarified data-recovery search steps, and added template-URL notation. * **Tests** * Expanded URL extraction and related utility tests to cover many formats and edge cases. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
eb259f1364 |
feat(assistant): score and improve SQL identifier quoting (#42122)
* feat: SQL correctness scorer, override mock tables * feat: replace "SQL Correctness" with "SQL Identifier Quoting" scorer * fix(prompt): discourage simulating confirmation of execute_sql tool this is already handled at the UI layer * fix(prompt): encourage quotes on identifiers with caps * feat: move extractIdentifiers to own file, add tests * chore: shorten tests * feat: extract ColumnDef column names in extractIdentifiers * refactor: sqlIdentifierQuotingScorer with more thorough checks * refactor: consolidate into `sql-identifier-quoting.ts` * feat: support mocking schemas, eval test case with case sensitive schema * fix: test cases that don't match default mock schema * chore: format * feat(prompt): mention special characters and reserved words * feat: optional description in metadata, test with special characters * feat: consolidated comprehensive test case * fix(prompt): revert conflicting instruction |
||
|
Matt Rossman
|
4553f09bb5 |
feat(assistant): hallucination scorers + corrective measures for storage versioning answers (#41655)
* feat: "Docs Faithfulness" scorer * feat: test case for storage object restoration * feat: "Factuality" scorer * feat: "Factuality" -> "Correctness" * feat: update Storage recovery test case * feat: finishReason in task output * feat: encourage parallel tool calls + docs search, discourage superfluous context gathering * prompt tuning (tool selection strategy) * add data recovery section in chat prompt * test: S3 versioning support correctness * refactor: derive stepsSerialized/textOnly from shared steps data * fix: input in correctness scorer |
||
|
|
072883bcec |
feat: assistant evals (#41311)
* chore: bump `supabase` CLI * chore: stricter message types in `generate-v4.ts` * feat: tutorial eval https://www.braintrust.dev/docs/evaluation * feat: project ID for eval * refactor: `generateAssistantResponse` out of `handlePost` * refactor: generateAssistantResponse to lib/ai * feat: factuality eval with assistant response * chore: upgrade braintrust to v1.0.1 * chore: silence tsconfig warning * feat: assertion scorer * fix: aggregate tools across all steps * refactor: strict tool names, remove need for `as const` * refactor: generic tool name type in assertions * feat: transfer mocks from `feature/braintrust` * feat: LLM criteria assertion * feat: braintrust evals workflow * fix: BRAINTRUST_PROJECT_ID * feat: `sql_similar` assertion * fix: `OPENAI_API_KEY` in workflow env * feat: split AssertionScorer into separate scorers * feat: remove tutorial eval * feat: 20 minute CI timeout * feat: category in test case metadata * feat: score with gpt-5 * refactor: dataset to own file, colocate scorers * feat: "gpt-5.2-2025-12-11" for llm as a judge * feat: SQL syntax scorer with `libpg-query` * feat: `evals:setup` and `evals:run` scripts * feat: `evals:setup` in CI * feat: human readable scorer names * chore: rename to "SQL Validity" * feat: add 2 "sql_generation" test cases * feat: update requiredTools in test cases * chore: ignore Cursor MCP config * feat: "Conciseness" score * feat: "Completeness" scorer * fix: generate-v4 test mocks * feat: serialize "steps" for scorer inputs * updated node mem options for typecheck * updated runner * remove ram update as actions handle this * feat: read `BRAINTRUST_PROJECT_ID` from secrets * feat: score helpfulness, remove old scorers * feat: separate `evals:run` and `evals:upload` scripts * feat: passthrough entire classifier result * feat: use live `search_docs` impl, store docs result in metadata * feat: reduce classifier options * feat: filter workflow by `run-evals` PR label or `master` branch * chore: cleanup stubbed mock tools * fix: checkout actual branch with `ref:` * fix: capture search_docs results from all content parts * feat: simplify sql syntax score calculation * feat: use AI SDK's UI message validator * docs: justification for relative `extends` * fix: cleanup leftover validatedMessages * doc: note mock token isn't secret for snyk * fix: mock ui message to pass validation * feat: revert ignoring Cursor MCP config Using `.git/info/exclude` instead until we have an opinion on this * feat: add "tsconfig" as shared-data devDependency, revert relative path in tsconfig * refactor: tool call parsing into function * Update apps/studio/evals/assistant.eval.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * refactor: organize mock schemas and tool factories --------- Co-authored-by: Ali Waseem <waseema393@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
