mirror of
https://github.com/supabase/supabase.git
synced 2026-10-11 04:15:04 +03:00
## Motivation When Assistant runs a potentially destructive tool like `execute_sql`, it stops the LLM request and prompts for client-side approval and execution of the tool. After approval, a second request kicks off under a separate trace. This has made scoring and [Topics](https://www.braintrust.dev/blog/topics) classification challenging, as the generated `output` is split across stateless requests. The [span-level scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans) approach we've used thusfar (after the LLM call, we massage the result into an `output` payload that's stuck onto the root span) has been cumbersome and led to invalid scores / topics where only part of the assistant response is considered. It's also inefficient, as we're duplicating potentially large info (like the `search_docs` output) that already exists within the trace. An alternative to scoring spans is to [score traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces). Braintrust [best practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices) advise: > Use span scope for evaluating individual operations or outputs. Use trace scope for evaluating multi-turn conversations, overall workflow completion, or when your scorer needs access to the full execution context. We've also received [direct guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89) from their team to use this approach. ## Changes Migrates eval scorers from custom `AssistantEvalOutput` shape to trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with thread parsing that scores the full latest Assistant turn and passes prior conversation separately where relevant. Moves `execute_sql` and `deploy_edge_function` from client-side execution after approval to AI SDK `needsApproval` + server-side `execute()`. SQL results returned to the model are gated by AI opt-in level, so row data is only included with `schema_and_log_and_data`; otherwise the tool returns the no-data-permissions sentinel. Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within an "assistant" turn due to tool call requests/responses. For online evals, this means we should configure automations to only score traces with `metadata.isFinalStep = true` to ensure we're judging the complete generated response. Other minor kaizen changes: - Renamed `promptProviderOptions` to `systemProviderOptions` to clarify that this is associated with the "system" message and disambiguate from the root `providerOptions` - Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown` span shapes from Braintrust, to more easily access typed inputs/output on tool spans. - Bumps AI SDK floor version `^6.0.116` → `^6.0.174` - Tweaked the "Conciseness" scorer to not unfairly dock points for the new `[called tool_name]` labels in serialized assistant response ## Verification In the studio staging build, I asked Assistant to create a todos table with 3 sample todos. I manually approved the `execute_sql` call and saw Assistant generate text before & after the call. In Braintrust I verified two traces were produced (see [filtered logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})), the first with `metadata.isFinalStep = false` and the second with `metadata.isFinalStep = true`. In the Braintrust staging scorers, I ran the preview Completeness scorer on the second trace and verified it sees the complete Assistant response including markers for tool calls ([link to trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82)) <img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x" src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a" /> I also tested the `deploy_edge_function` workflow and verified it still prompts for permission and warns on deployment of existing functions. **References** - https://www.braintrust.dev/docs/evaluate/custom-code#score-traces - https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval Supercedes https://github.com/supabase/supabase/pull/45556 and https://github.com/supabase/supabase/pull/45339 Closes AI-473 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tool actions (SQL execution, edge-function deploy) now require explicit user Approve/Deny before proceeding. * **Improvements** * Assistant pauses for approval responses before sending follow-ups, giving clearer control over risky actions. * Deploy/replace flows show confirmation and clearer replace warnings. * Evaluation/scoring updated to use richer trace data for more accurate assistant performance signals. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
224 lines
7.1 KiB
TypeScript
224 lines
7.1 KiB
TypeScript
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
|
|
|
import { getStudioTools } from './studio-tools'
|
|
import { executeSql } from '@/data/sql/execute-sql-query'
|
|
import { NO_DATA_PERMISSIONS } from '@/lib/ai/tools/tool-sanitizer'
|
|
|
|
vi.mock('@/data/sql/execute-sql-query', () => ({
|
|
executeSql: vi.fn(),
|
|
}))
|
|
|
|
describe('ai/tools/studio-tools', () => {
|
|
beforeEach(() => {
|
|
vi.mocked(executeSql).mockReset()
|
|
})
|
|
|
|
describe('getStudioTools', () => {
|
|
it('should return an object with tool definitions', () => {
|
|
const tools = getStudioTools()
|
|
|
|
expect(tools).toBeDefined()
|
|
expect(typeof tools).toBe('object')
|
|
})
|
|
|
|
it('should include execute_sql tool', () => {
|
|
const tools = getStudioTools()
|
|
|
|
expect(tools.execute_sql).toBeDefined()
|
|
expect(tools.execute_sql.description).toContain('execute a SQL statement')
|
|
})
|
|
|
|
it('should include deploy_edge_function tool', () => {
|
|
const tools = getStudioTools()
|
|
|
|
expect(tools.deploy_edge_function).toBeDefined()
|
|
expect(tools.deploy_edge_function.description).toContain('deploy a Supabase Edge Function')
|
|
})
|
|
|
|
it('should include rename_chat tool', () => {
|
|
const tools = getStudioTools()
|
|
|
|
expect(tools.rename_chat).toBeDefined()
|
|
expect(tools.rename_chat.description).toContain('Rename the current chat session')
|
|
})
|
|
|
|
it('should have exactly 4 tools', () => {
|
|
const tools = getStudioTools()
|
|
const toolNames = Object.keys(tools)
|
|
|
|
expect(toolNames).toHaveLength(4)
|
|
expect(toolNames).toContain('load_knowledge')
|
|
expect(toolNames).toContain('execute_sql')
|
|
expect(toolNames).toContain('deploy_edge_function')
|
|
expect(toolNames).toContain('rename_chat')
|
|
})
|
|
|
|
it('should have execute_sql with correct input schema fields', () => {
|
|
const tools = getStudioTools()
|
|
const executeSqlTool = tools.execute_sql
|
|
|
|
// Check that the tool has an input schema
|
|
expect(executeSqlTool.inputSchema).toBeDefined()
|
|
|
|
// Verify the schema exists and is a Zod object
|
|
const schema = executeSqlTool.inputSchema
|
|
expect(schema).toBeDefined()
|
|
expect((schema as any)._def.typeName).toBe('ZodObject')
|
|
})
|
|
|
|
it('should have deploy_edge_function with input schema', () => {
|
|
const tools = getStudioTools()
|
|
const deployTool = tools.deploy_edge_function
|
|
|
|
expect(deployTool.inputSchema).toBeDefined()
|
|
|
|
// Verify the schema exists and is a Zod object
|
|
expect(deployTool.inputSchema).toBeDefined()
|
|
expect((deployTool.inputSchema as any)._def.typeName).toBe('ZodObject')
|
|
})
|
|
|
|
it('should have rename_chat with execute function', async () => {
|
|
const tools = getStudioTools()
|
|
const renameTool = tools.rename_chat
|
|
|
|
expect(renameTool.execute).toBeDefined()
|
|
expect(typeof renameTool.execute).toBe('function')
|
|
|
|
// Test the execute function
|
|
if (!renameTool.execute) throw new Error('execute is undefined')
|
|
const result = await renameTool.execute(
|
|
{ newName: 'Test Chat' },
|
|
{ toolCallId: 'test', messages: [] }
|
|
)
|
|
expect(result).toEqual({ status: 'Chat request sent to client' })
|
|
})
|
|
|
|
it('should validate execute_sql input schema correctly', () => {
|
|
const tools = getStudioTools()
|
|
const schema = tools.execute_sql.inputSchema
|
|
|
|
// Check if schema is a Zod schema with safeParse
|
|
if ('safeParse' in schema) {
|
|
// Valid input
|
|
const validInput = {
|
|
sql: 'SELECT * FROM users',
|
|
label: 'Get users',
|
|
chartConfig: { view: 'table' as const },
|
|
isWriteQuery: false,
|
|
}
|
|
expect(schema.safeParse(validInput).success).toBe(true)
|
|
|
|
// Valid chart config
|
|
const validChartInput = {
|
|
sql: 'SELECT count(*) FROM users',
|
|
label: 'User count',
|
|
chartConfig: { view: 'chart' as const, xAxis: 'date', yAxis: 'count' },
|
|
isWriteQuery: false,
|
|
}
|
|
expect(schema.safeParse(validChartInput).success).toBe(true)
|
|
|
|
// Missing required field
|
|
const invalidInput = {
|
|
sql: 'SELECT * FROM users',
|
|
// missing label, chartConfig, isWriteQuery
|
|
}
|
|
expect(schema.safeParse(invalidInput).success).toBe(false)
|
|
} else {
|
|
// Skip test if schema doesn't have safeParse
|
|
expect(schema).toBeDefined()
|
|
}
|
|
})
|
|
|
|
it('should require approval for read and write SQL queries', () => {
|
|
const tools = getStudioTools()
|
|
|
|
expect(tools.execute_sql.needsApproval).toBe(true)
|
|
})
|
|
|
|
it('should sanitize execute_sql output without data opt-in', async () => {
|
|
vi.mocked(executeSql).mockResolvedValue({ result: [{ email: 'test@example.com' }] })
|
|
|
|
const tools = getStudioTools({
|
|
projectRef: 'test-project',
|
|
connectionString: 'encrypted-connection-string',
|
|
aiOptInLevel: 'schema',
|
|
})
|
|
|
|
if (!tools.execute_sql.execute) throw new Error('execute is undefined')
|
|
const result = await tools.execute_sql.execute(
|
|
{
|
|
sql: 'SELECT email FROM users',
|
|
label: 'Get emails',
|
|
chartConfig: { view: 'table' },
|
|
isWriteQuery: false,
|
|
},
|
|
{ toolCallId: 'test', messages: [] }
|
|
)
|
|
|
|
expect(executeSql).toHaveBeenCalledWith(
|
|
{
|
|
projectRef: 'test-project',
|
|
connectionString: 'encrypted-connection-string',
|
|
sql: 'SELECT email FROM users',
|
|
},
|
|
undefined,
|
|
undefined
|
|
)
|
|
expect(result).toBe(NO_DATA_PERMISSIONS)
|
|
})
|
|
|
|
it('should return execute_sql rows with data opt-in', async () => {
|
|
const rows = [{ email: 'test@example.com' }]
|
|
vi.mocked(executeSql).mockResolvedValue({ result: rows })
|
|
|
|
const tools = getStudioTools({
|
|
projectRef: 'test-project',
|
|
connectionString: 'encrypted-connection-string',
|
|
aiOptInLevel: 'schema_and_log_and_data',
|
|
})
|
|
|
|
if (!tools.execute_sql.execute) throw new Error('execute is undefined')
|
|
const result = await tools.execute_sql.execute(
|
|
{
|
|
sql: 'SELECT email FROM users',
|
|
label: 'Get emails',
|
|
chartConfig: { view: 'table' },
|
|
isWriteQuery: false,
|
|
},
|
|
{ toolCallId: 'test', messages: [] }
|
|
)
|
|
|
|
expect(executeSql).toHaveBeenCalledWith(
|
|
{
|
|
projectRef: 'test-project',
|
|
connectionString: 'encrypted-connection-string',
|
|
sql: 'SELECT email FROM users',
|
|
},
|
|
undefined,
|
|
undefined
|
|
)
|
|
expect(result).toEqual(rows)
|
|
})
|
|
|
|
it('should validate rename_chat input schema correctly', () => {
|
|
const tools = getStudioTools()
|
|
const schema = tools.rename_chat.inputSchema
|
|
|
|
// Check if schema is a Zod schema with safeParse
|
|
if ('safeParse' in schema) {
|
|
// Valid input
|
|
expect(schema.safeParse({ newName: 'My Chat' }).success).toBe(true)
|
|
|
|
// Invalid input - missing newName
|
|
expect(schema.safeParse({}).success).toBe(false)
|
|
|
|
// Invalid input - wrong type
|
|
expect(schema.safeParse({ newName: 123 }).success).toBe(false)
|
|
} else {
|
|
// Skip test if schema doesn't have safeParse
|
|
expect(schema).toBeDefined()
|
|
}
|
|
})
|
|
})
|
|
})
|