mirror of
https://github.com/supabase/supabase.git
synced 2026-10-05 09:25:06 +03:00
test(studio): add eval cases for list_databases-driven notebook creation (#49398)
## Summary - Adds three eval cases exercising the behavior this stack wires up: a happy path where the model calls `list_databases` before targeting a named read replica, a guard against fabricating an identifier when the user names a region/replica `list_databases` doesn't actually return, and a guard against targeting a non-primary database when the user never asked for one. Part 6/6 (final) of the stack for FE-4225 (expose valid database identifiers to the notebook AI agent). Stacked on #49334. ## Test plan - [x] Ran all three cases against the real model; inspected transcripts directly - [x] Re-verified reworded `correctAnswer` text against real outputs via the correctness evaluator - [x] `pnpm --filter studio exec tsc --noEmit` passes - [x] `prettier --check` passes <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added evaluation coverage for selecting the correct database replica when creating notebooks. * Verified primary-database defaults when no database is specified. * Added checks to prevent fabricated database identifiers when a requested replica is unavailable. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
1 parent
27ff49c145
commit
c172195269
1 file changed
+89
@@ -584,6 +584,95 @@ export const dataset: AssistantEvalCase[] = [
|
||||
'Exercises absolute_time_range on a log cell, and guards against miscategorizing a logs query as a database_cell or writing legacy BigQuery-style SQL instead of ClickHouse',
|
||||
},
|
||||
},
|
||||
{
|
||||
input: {
|
||||
prompt:
|
||||
"Create a notebook called 'Replica read check' with a query that counts rows in the customers table, and make sure it runs against my read replica, not the primary database.",
|
||||
mockTables: {
|
||||
public: [
|
||||
{
|
||||
name: 'customers',
|
||||
rls_enabled: true,
|
||||
columns: [
|
||||
{ name: 'id', data_type: 'uuid' },
|
||||
{ name: 'created_at', data_type: 'timestamp with time zone' },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
expected: {
|
||||
requiredTools: [
|
||||
'list_databases',
|
||||
{ name: 'create_notebook', input: { name: { equals: 'Replica read check' } } },
|
||||
],
|
||||
correctAnswer:
|
||||
"Calls list_databases before creating the notebook, then creates a notebook via create_notebook named 'Replica read check' with a database_cell that counts rows in customers and sets database_identifier to 'mock-project-ref-replica-1' — the non-primary database the mock list_databases fixture returns — not 'mock-project-ref' (the primary) and not some other fabricated string.",
|
||||
},
|
||||
metadata: {
|
||||
category: ['general_help'],
|
||||
description:
|
||||
'Exercises calling list_databases before setting a database_cell to target a non-primary database',
|
||||
},
|
||||
},
|
||||
{
|
||||
input: {
|
||||
prompt:
|
||||
"Create a notebook called 'Customer signups overview' with a query that shows the 20 most recently created customers.",
|
||||
mockTables: {
|
||||
public: [
|
||||
{
|
||||
name: 'customers',
|
||||
rls_enabled: true,
|
||||
columns: [
|
||||
{ name: 'id', data_type: 'uuid' },
|
||||
{ name: 'created_at', data_type: 'timestamp with time zone' },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
expected: {
|
||||
requiredTools: [
|
||||
{ name: 'create_notebook', input: { name: { equals: 'Customer signups overview' } } },
|
||||
],
|
||||
correctAnswer:
|
||||
"Creates a notebook via create_notebook named 'Customer signups overview' with a database_cell selecting the 20 most recent customers. Since the user never named a specific database or replica, the cell either omits database_identifier or sets it to 'mock-project-ref' (the primary) — it must not set it to 'mock-project-ref-replica-1' or any other non-primary/fabricated value.",
|
||||
},
|
||||
metadata: {
|
||||
category: ['general_help'],
|
||||
description:
|
||||
'Guards against targeting a non-primary database when the user never asked for a specific one — the cell must omit database_identifier or target the primary, never a replica',
|
||||
},
|
||||
},
|
||||
{
|
||||
input: {
|
||||
prompt:
|
||||
"Create a notebook called 'EU replica check' with a query that counts rows in the customers table, targeting my EU read replica.",
|
||||
mockTables: {
|
||||
public: [
|
||||
{
|
||||
name: 'customers',
|
||||
rls_enabled: true,
|
||||
columns: [
|
||||
{ name: 'id', data_type: 'uuid' },
|
||||
{ name: 'created_at', data_type: 'timestamp with time zone' },
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
expected: {
|
||||
requiredTools: ['list_databases'],
|
||||
correctAnswer:
|
||||
"Calls list_databases and finds no EU-region replica among the real results. Either creates the notebook against a database_identifier it actually found (while noting it isn't in the EU) or asks the user to confirm before proceeding — it does not invent an identifier that merely sounds like an EU replica.",
|
||||
},
|
||||
metadata: {
|
||||
category: ['general_help'],
|
||||
description:
|
||||
'Guards against fabricating a database_identifier when the user names a region/replica that list_databases does not actually return',
|
||||
},
|
||||
},
|
||||
{
|
||||
input: {
|
||||
prompt:
|
||||
|
||||
Reference in new issue
Block a user