Matt Rossman
a127f2cbbc
test(assistant): add eval case for execute_sql usage on default "Generate sample data" prompt ( #42219 )
...
test: add eval case for execute_sql on sample data generation
2026-01-28 10:02:22 -05:00
Matt Rossman
eb259f1364
feat(assistant): score and improve SQL identifier quoting ( #42122 )
...
* feat: SQL correctness scorer, override mock tables
* feat: replace "SQL Correctness" with "SQL Identifier Quoting" scorer
* fix(prompt): discourage simulating confirmation of execute_sql tool
this is already handled at the UI layer
* fix(prompt): encourage quotes on identifiers with caps
* feat: move extractIdentifiers to own file, add tests
* chore: shorten tests
* feat: extract ColumnDef column names in extractIdentifiers
* refactor: sqlIdentifierQuotingScorer with more thorough checks
* refactor: consolidate into `sql-identifier-quoting.ts`
* feat: support mocking schemas, eval test case with case sensitive schema
* fix: test cases that don't match default mock schema
* chore: format
* feat(prompt): mention special characters and reserved words
* feat: optional description in metadata, test with special characters
* feat: consolidated comprehensive test case
* fix(prompt): revert conflicting instruction
2026-01-27 09:28:56 -05:00
Matt Rossman
4553f09bb5
feat(assistant): hallucination scorers + corrective measures for storage versioning answers ( #41655 )
...
* feat: "Docs Faithfulness" scorer
* feat: test case for storage object restoration
* feat: "Factuality" scorer
* feat: "Factuality" -> "Correctness"
* feat: update Storage recovery test case
* feat: finishReason in task output
* feat: encourage parallel tool calls + docs search, discourage superfluous context gathering
* prompt tuning (tool selection strategy)
* add data recovery section in chat prompt
* test: S3 versioning support correctness
* refactor: derive stepsSerialized/textOnly from shared steps data
* fix: input in correctness scorer
2026-01-15 14:22:46 +07:00
072883bcec
feat: assistant evals ( #41311 )
...
* chore: bump `supabase` CLI
* chore: stricter message types in `generate-v4.ts`
* feat: tutorial eval
https://www.braintrust.dev/docs/evaluation
* feat: project ID for eval
* refactor: `generateAssistantResponse` out of `handlePost`
* refactor: generateAssistantResponse to lib/ai
* feat: factuality eval with assistant response
* chore: upgrade braintrust to v1.0.1
* chore: silence tsconfig warning
* feat: assertion scorer
* fix: aggregate tools across all steps
* refactor: strict tool names, remove need for `as const`
* refactor: generic tool name type in assertions
* feat: transfer mocks from `feature/braintrust`
* feat: LLM criteria assertion
* feat: braintrust evals workflow
* fix: BRAINTRUST_PROJECT_ID
* feat: `sql_similar` assertion
* fix: `OPENAI_API_KEY` in workflow env
* feat: split AssertionScorer into separate scorers
* feat: remove tutorial eval
* feat: 20 minute CI timeout
* feat: category in test case metadata
* feat: score with gpt-5
* refactor: dataset to own file, colocate scorers
* feat: "gpt-5.2-2025-12-11" for llm as a judge
* feat: SQL syntax scorer with `libpg-query`
* feat: `evals:setup` and `evals:run` scripts
* feat: `evals:setup` in CI
* feat: human readable scorer names
* chore: rename to "SQL Validity"
* feat: add 2 "sql_generation" test cases
* feat: update requiredTools in test cases
* chore: ignore Cursor MCP config
* feat: "Conciseness" score
* feat: "Completeness" scorer
* fix: generate-v4 test mocks
* feat: serialize "steps" for scorer inputs
* updated node mem options for typecheck
* updated runner
* remove ram update as actions handle this
* feat: read `BRAINTRUST_PROJECT_ID` from secrets
* feat: score helpfulness, remove old scorers
* feat: separate `evals:run` and `evals:upload` scripts
* feat: passthrough entire classifier result
* feat: use live `search_docs` impl, store docs result in metadata
* feat: reduce classifier options
* feat: filter workflow by `run-evals` PR label or `master` branch
* chore: cleanup stubbed mock tools
* fix: checkout actual branch with `ref:`
* fix: capture search_docs results from all content parts
* feat: simplify sql syntax score calculation
* feat: use AI SDK's UI message validator
* docs: justification for relative `extends`
* fix: cleanup leftover validatedMessages
* doc: note mock token isn't secret for snyk
* fix: mock ui message to pass validation
* feat: revert ignoring Cursor MCP config
Using `.git/info/exclude` instead until we have an opinion on this
* feat: add "tsconfig" as shared-data devDependency, revert relative path in tsconfig
* refactor: tool call parsing into function
* Update apps/studio/evals/assistant.eval.ts
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* refactor: organize mock schemas and tool factories
---------
Co-authored-by: Ali Waseem <waseema393@gmail.com >
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-12-22 23:45:48 -05:00
Matt Rossman