Matt Rossman
0d8cd9c315
## Motivation When an Assistant tool needs a higher opt-in level than the org has, the user only sees the model say it has no access with (inconsistent) written instructions how to fix, instead of Assistant proactively facilitating the fix. <img width="400" alt="CleanShot 2026-10-07 at 3 44 29 PM@2x" src="https://github.com/user-attachments/assets/ba95568a-5db4-4d8c-b4ff-64855f2b87ec" /> The majority of recent labelled [Assistant Issues](https://www.braintrust.dev/app/supabase.io/p/Assistant/topics) in Braintrust are tools or query results blocked by the org opt-in level. We've considered making Assistant disabled entirely when data opt-in is off to eliminate the most common footgun (AI-405), but that's an intrusive change which could break legitimate use cases. ## Changes https://github.com/user-attachments/assets/206a2438-630e-4368-9cec-2b04b66c9cd4 A new `update_opt_in_level` tool renders an inline card which prompts admins to review the opt-in level and highlights the proposed change (non-admins see a message telling them to contact their admin to change the setting). Saving approves the call and the turn resumes at the new level. Results appear in a Braintrust tool span based on https://github.com/supabase/supabase/pull/45654. I added 5 evals that simulate the user answering the opt-in card to verify that the Assistant asks when blocked, continues after the user accepts, and doesn't invent data after a skip. Tool Usage and Correctness are at 100% in the [sample run](https://www.braintrust.dev/app/supabase.io/p/Dev%20(mattrossman%2FAssistant)/experiments/mattrossman%2Fai-158-prompt-users-to-update-settings-instead-of-showing-no-tool-1791483991). <details> <summary>📸 Screenshots</summary> | Admin, pending | Modal, Current and Proposed | | -- | -- | | <img width="100%" alt="CleanShot 2026-10-07 at 3 23 37 PM@2x" src="https://github.com/user-attachments/assets/2fc8ba11-0530-4028-a056-40ca02afd3ef" /> | <img width="1184" height="1754" alt="CleanShot 2026-10-07 at 3 32 37 PM@2x" src="https://github.com/user-attachments/assets/73639633-89ac-4d4c-b869-d9301735d5b9" /> | | **After saving, turn continues** | **Non-admin** | | <img width="100%" alt="CleanShot 2026-10-07 at 3 27 14 PM@2x" src="https://github.com/user-attachments/assets/a91c6760-6dfb-4f51-9ef8-4bdbe1b22495" /> | <img width="100%" alt="CleanShot 2026-10-07 at 3 15 33 PM@2x" src="https://github.com/user-attachments/assets/ecb9aa87-26dc-4b69-a086-cb77a28bb860" /> | </details> ## Verification To test in staging, set your org's Assistant opt-in level to Disabled, ask "What tables do I have?", and review the card. Try selecting the proposed (or different) opt-in level and saving to continue. [This trace](https://www.braintrust.dev/app/supabase.io/p/Dev%20(mattrossman%2FAssistant)/experiments/mattrossman%2Fai-158-prompt-users-to-update-settings-instead-of-showing-no-tool-1791483991?r=90066cd9-4da3-4817-9650-c369d2b42cea) is a sample from the accept eval case after saving Schema Only, note the `update_opt_in_level` span. ## Safety considerations I added a banner to the modal to make it more obvious that this setting impacts the whole org, not just the current chat or project: <img width="592" height="78" alt="CleanShot 2026-10-08 at 2 04 44 PM@2x" src="https://github.com/user-attachments/assets/6b140ed6-65fb-4bc0-b5cc-9c82fe2a67aa" /> I leave the current opt-in value selected by default in the form so the user has to consciously select the proposed value instead of mindlessly clicking save without understanding implications. `execute_sql` and `run_notebook` outputs are now stamped with the level they ran under, and history is sanitized by the least permissive of the stamped vs current opt-in levels. That way raising the level mid-chat doesn't unexpectedly expose earlier query rows to the model. Closes AI-158
Studio Assistant Evals
We use Braintrust to evaluate Assistant behaviors against a tracked dataset (offline evals) and against live traces (online evals).
Offline Evals
Add offline eval test cases to dataset.ts. If needed, add new scorers (see below) for the specific dimension you wish to test. Expect to update and run offline evals when adding new Assistant behaviors
You may wish to run offline evals when:
- You updated the eval suite with a new test case or scorer
- You changed Assistant's behavior and want to check for improvements/regressions
Running Offline Evals in CI
Add the run-evals label on a PR to the repo and Braintrust's GitHub Action will run evals and post a summary comment (example).
You can find detailed results in the "Experiments" tab of the "Assistant" project on Braintrust.
Running Offline Evals in Local Dev
Within apps/studio
# To set up WASM files
pnpm evals:setup
# Run all evals and upload results to Braintrust
pnpm evals:upload
# Run all evals without uploading results
pnpm evals:run
# Run an upload single test case
pnpm braintrust eval evals/assistant.eval.ts --filter "input.prompt=How many projects"
Upload results when you want to inspect Experiments or Logs in the Braintrust dashboard or API. You can use developer tools like Braintrust MCP or bt CLI to analyze results with an agent.
Scorers
Scorers look at a thread or task output and assign a score deterministically or via LLM-as-a-judge. Optionally they can consider expected values.
Define scorers in scorer.ts and include them in assistant.eval.ts to run them in offline evals.
Updating Online Scorers
Online scorers run as serverless functions on Braintrust infrastructure. They're deployed from the scorer-online.ts script. Since these scoring against production traces, they can't rely on ground truth expected values. Structure scoring logic and LLM prompts accordingly. Not every scorer needs to be an online scorer.
To opt-in to online scoring, add the scorer to scorer-online-manifest.json and add a corresponding handler in scorer-online.ts
Testing & Deploying Online Scorers
Add the preview-scorers label to a PR to deploy branch-prefixed scorers to the "Assistant (Staging Scorers)" Braintrust project (example). From that project dashboard, you can manually test the scorer against a trace from any project.
After merge to master, preview scorers automatically clean up and deploy to the production in the "Assistant" Braintrust project. Update the "Online Scoring" automation in the Logs page to include the new scorer function.