Files
supabase/apps/studio/hooks/forms/useAIOptInForm.ts
T
Matt Rossman 0d8cd9c315 feat(studio): prompt for a higher AI opt-in level instead of "no tool access" (#51411)
## Motivation

When an Assistant tool needs a higher opt-in level than the org has, the
user only sees the model say it has no access with (inconsistent)
written instructions how to fix, instead of Assistant proactively
facilitating the fix.

<img width="400" alt="CleanShot 2026-10-07 at 3 44 29 PM@2x"
src="https://github.com/user-attachments/assets/ba95568a-5db4-4d8c-b4ff-64855f2b87ec"
/>

The majority of recent labelled [Assistant
Issues](https://www.braintrust.dev/app/supabase.io/p/Assistant/topics)
in Braintrust are tools or query results blocked by the org opt-in
level. We've considered making Assistant disabled entirely when data
opt-in is off to eliminate the most common footgun (AI-405), but that's
an intrusive change which could break legitimate use cases.

## Changes



https://github.com/user-attachments/assets/206a2438-630e-4368-9cec-2b04b66c9cd4



A new `update_opt_in_level` tool renders an inline card which prompts
admins to review the opt-in level and highlights the proposed change
(non-admins see a message telling them to contact their admin to change
the setting). Saving approves the call and the turn resumes at the new
level. Results appear in a Braintrust tool span based on
https://github.com/supabase/supabase/pull/45654.

I added 5 evals that simulate the user answering the opt-in card to
verify that the Assistant asks when blocked, continues after the user
accepts, and doesn't invent data after a skip. Tool Usage and
Correctness are at 100% in the [sample
run](https://www.braintrust.dev/app/supabase.io/p/Dev%20(mattrossman%2FAssistant)/experiments/mattrossman%2Fai-158-prompt-users-to-update-settings-instead-of-showing-no-tool-1791483991).

<details>
<summary>📸 Screenshots</summary>

| Admin, pending | Modal, Current and Proposed |
| -- | -- |
| <img width="100%" alt="CleanShot 2026-10-07 at 3 23 37 PM@2x"
src="https://github.com/user-attachments/assets/2fc8ba11-0530-4028-a056-40ca02afd3ef"
/> | <img width="1184" height="1754" alt="CleanShot 2026-10-07 at 3 32
37 PM@2x"
src="https://github.com/user-attachments/assets/73639633-89ac-4d4c-b869-d9301735d5b9"
/> |
| **After saving, turn continues** | **Non-admin** |
| <img width="100%" alt="CleanShot 2026-10-07 at 3 27 14 PM@2x"
src="https://github.com/user-attachments/assets/a91c6760-6dfb-4f51-9ef8-4bdbe1b22495"
/> | <img width="100%" alt="CleanShot 2026-10-07 at 3 15 33 PM@2x"
src="https://github.com/user-attachments/assets/ecb9aa87-26dc-4b69-a086-cb77a28bb860"
/> |

</details>

## Verification

To test in staging, set your org's Assistant opt-in level to Disabled,
ask "What tables do I have?", and review the card. Try selecting the
proposed (or different) opt-in level and saving to continue.

[This
trace](https://www.braintrust.dev/app/supabase.io/p/Dev%20(mattrossman%2FAssistant)/experiments/mattrossman%2Fai-158-prompt-users-to-update-settings-instead-of-showing-no-tool-1791483991?r=90066cd9-4da3-4817-9650-c369d2b42cea)
is a sample from the accept eval case after saving Schema Only, note the
`update_opt_in_level` span.

## Safety considerations

I added a banner to the modal to make it more obvious that this setting
impacts the whole org, not just the current chat or project:

<img width="592" height="78" alt="CleanShot 2026-10-08 at 2 04 44 PM@2x"
src="https://github.com/user-attachments/assets/6b140ed6-65fb-4bc0-b5cc-9c82fe2a67aa"
/>


I leave the current opt-in value selected by default in the form so the
user has to consciously select the proposed value instead of mindlessly
clicking save without understanding implications.

`execute_sql` and `run_notebook` outputs are now stamped with the level
they ran under, and history is sanitized by the least permissive of the
stamped vs current opt-in levels. That way raising the level mid-chat
doesn't unexpectedly expose earlier query rows to the model.

Closes AI-158
2026-10-09 10:39:23 -04:00

112 lines
3.9 KiB
TypeScript

import { zodResolver } from '@hookform/resolvers/zod'
import { PermissionAction } from '@supabase/shared-types/out/constants'
import { useQueryClient } from '@tanstack/react-query'
import { LOCAL_STORAGE_KEYS } from 'common'
import { useForm } from 'react-hook-form'
import { toast } from 'sonner'
import * as z from 'zod'
import { useOrganizationUpdateMutation } from '@/data/organizations/organization-update-mutation'
import { invalidateOrganizationsQuery } from '@/data/organizations/organizations-query'
import { useAsyncCheckPermissions } from '@/hooks/misc/useCheckPermissions'
import { useLocalStorageQuery } from '@/hooks/misc/useLocalStorage'
import { getAiOptInLevel, type AiOptInLevel } from '@/hooks/misc/useOrgOptedIntoAi'
import { useSelectedOrganizationQuery } from '@/hooks/misc/useSelectedOrganization'
import { OPT_IN_TAGS } from '@/lib/constants'
import type { ResponseError } from '@/types'
// Shared schema definition
export const AIOptInSchema = z.object({
aiOptInLevel: z.enum(['disabled', 'schema', 'schema_and_log', 'schema_and_log_and_data'], {
required_error: 'AI Opt-in level selection is required',
}),
})
export type AIOptInFormValues = z.infer<typeof AIOptInSchema>
/**
* Hook to manage the AI Opt-In form state and submission logic.
* Optionally takes an onSuccess callback (e.g., to close a modal).
*/
export const useAIOptInForm = (onSuccessCallback?: (level: AiOptInLevel) => void) => {
const queryClient = useQueryClient()
const { data: selectedOrganization } = useSelectedOrganizationQuery()
const { can: canUpdateOrganization } = useAsyncCheckPermissions(
PermissionAction.UPDATE,
'organizations'
)
const [, setUpdatedOptInSinceMCP] = useLocalStorageQuery(
LOCAL_STORAGE_KEYS.AI_ASSISTANT_MCP_OPT_IN,
false
)
const { mutate: updateOrganization, isPending: isUpdating } = useOrganizationUpdateMutation()
const form = useForm<AIOptInFormValues>({
resolver: zodResolver(AIOptInSchema),
defaultValues: {
aiOptInLevel: getAiOptInLevel(selectedOrganization?.opt_in_tags),
},
})
const onSubmit = async (values: AIOptInFormValues) => {
if (!canUpdateOrganization) {
return toast.error('You do not have the required permissions to update this organization')
}
if (!selectedOrganization?.slug) {
console.error('Organization slug is required')
return toast.error('Failed to update settings: Organization not found.')
}
const existingOptInTags = selectedOrganization?.opt_in_tags ?? []
let updatedOptInTags = existingOptInTags.filter(
(tag: string) =>
tag !== OPT_IN_TAGS.AI_SQL &&
tag !== (OPT_IN_TAGS.AI_DATA ?? 'AI_DATA') &&
tag !== (OPT_IN_TAGS.AI_LOG ?? 'AI_LOG')
)
if (
values.aiOptInLevel === 'schema' ||
values.aiOptInLevel === 'schema_and_log' ||
values.aiOptInLevel === 'schema_and_log_and_data'
) {
updatedOptInTags.push(OPT_IN_TAGS.AI_SQL)
}
if (
values.aiOptInLevel === 'schema_and_log' ||
values.aiOptInLevel === 'schema_and_log_and_data'
) {
updatedOptInTags.push(OPT_IN_TAGS.AI_LOG)
}
if (values.aiOptInLevel === 'schema_and_log_and_data') {
updatedOptInTags.push(OPT_IN_TAGS.AI_DATA)
}
updatedOptInTags = [...new Set(updatedOptInTags)]
updateOrganization(
{ slug: selectedOrganization.slug, opt_in_tags: updatedOptInTags },
{
onSuccess: () => {
invalidateOrganizationsQuery(queryClient)
toast.success('Successfully updated AI opt-in settings')
setUpdatedOptInSinceMCP(true)
onSuccessCallback?.(values.aiOptInLevel)
},
onError: (error: ResponseError) => {
toast.error(`Failed to update settings: ${error.message}`)
},
}
)
}
return {
form,
onSubmit,
isUpdating,
currentOptInLevel: getAiOptInLevel(selectedOrganization?.opt_in_tags),
}
}