mirror of
https://github.com/supabase/supabase.git
synced 2026-10-10 11:55:05 +03:00
Matt Rossman
0d8cd9c315
## Motivation When an Assistant tool needs a higher opt-in level than the org has, the user only sees the model say it has no access with (inconsistent) written instructions how to fix, instead of Assistant proactively facilitating the fix. <img width="400" alt="CleanShot 2026-10-07 at 3 44 29 PM@2x" src="https://github.com/user-attachments/assets/ba95568a-5db4-4d8c-b4ff-64855f2b87ec" /> The majority of recent labelled [Assistant Issues](https://www.braintrust.dev/app/supabase.io/p/Assistant/topics) in Braintrust are tools or query results blocked by the org opt-in level. We've considered making Assistant disabled entirely when data opt-in is off to eliminate the most common footgun (AI-405), but that's an intrusive change which could break legitimate use cases. ## Changes https://github.com/user-attachments/assets/206a2438-630e-4368-9cec-2b04b66c9cd4 A new `update_opt_in_level` tool renders an inline card which prompts admins to review the opt-in level and highlights the proposed change (non-admins see a message telling them to contact their admin to change the setting). Saving approves the call and the turn resumes at the new level. Results appear in a Braintrust tool span based on https://github.com/supabase/supabase/pull/45654. I added 5 evals that simulate the user answering the opt-in card to verify that the Assistant asks when blocked, continues after the user accepts, and doesn't invent data after a skip. Tool Usage and Correctness are at 100% in the [sample run](https://www.braintrust.dev/app/supabase.io/p/Dev%20(mattrossman%2FAssistant)/experiments/mattrossman%2Fai-158-prompt-users-to-update-settings-instead-of-showing-no-tool-1791483991). <details> <summary>📸 Screenshots</summary> | Admin, pending | Modal, Current and Proposed | | -- | -- | | <img width="100%" alt="CleanShot 2026-10-07 at 3 23 37 PM@2x" src="https://github.com/user-attachments/assets/2fc8ba11-0530-4028-a056-40ca02afd3ef" /> | <img width="1184" height="1754" alt="CleanShot 2026-10-07 at 3 32 37 PM@2x" src="https://github.com/user-attachments/assets/73639633-89ac-4d4c-b869-d9301735d5b9" /> | | **After saving, turn continues** | **Non-admin** | | <img width="100%" alt="CleanShot 2026-10-07 at 3 27 14 PM@2x" src="https://github.com/user-attachments/assets/a91c6760-6dfb-4f51-9ef8-4bdbe1b22495" /> | <img width="100%" alt="CleanShot 2026-10-07 at 3 15 33 PM@2x" src="https://github.com/user-attachments/assets/ecb9aa87-26dc-4b69-a086-cb77a28bb860" /> | </details> ## Verification To test in staging, set your org's Assistant opt-in level to Disabled, ask "What tables do I have?", and review the card. Try selecting the proposed (or different) opt-in level and saving to continue. [This trace](https://www.braintrust.dev/app/supabase.io/p/Dev%20(mattrossman%2FAssistant)/experiments/mattrossman%2Fai-158-prompt-users-to-update-settings-instead-of-showing-no-tool-1791483991?r=90066cd9-4da3-4817-9650-c369d2b42cea) is a sample from the accept eval case after saving Schema Only, note the `update_opt_in_level` span. ## Safety considerations I added a banner to the modal to make it more obvious that this setting impacts the whole org, not just the current chat or project: <img width="592" height="78" alt="CleanShot 2026-10-08 at 2 04 44 PM@2x" src="https://github.com/user-attachments/assets/6b140ed6-65fb-4bc0-b5cc-9c82fe2a67aa" /> I leave the current opt-in value selected by default in the form so the user has to consciously select the proposed value instead of mindlessly clicking save without understanding implications. `execute_sql` and `run_notebook` outputs are now stamped with the level they ran under, and history is sanitized by the least permissive of the stamped vs current opt-in levels. That way raising the level mid-chat doesn't unexpectedly expose earlier query rows to the model. Closes AI-158