2 Commits

Include renames

Author	SHA1	Message	Date
3 people	072883bcec	feat: assistant evals (#41311) ... * chore: bump `supabase` CLI * chore: stricter message types in `generate-v4.ts` * feat: tutorial eval https://www.braintrust.dev/docs/evaluation * feat: project ID for eval * refactor: `generateAssistantResponse` out of `handlePost` * refactor: generateAssistantResponse to lib/ai * feat: factuality eval with assistant response * chore: upgrade braintrust to v1.0.1 * chore: silence tsconfig warning * feat: assertion scorer * fix: aggregate tools across all steps * refactor: strict tool names, remove need for `as const` * refactor: generic tool name type in assertions * feat: transfer mocks from `feature/braintrust` * feat: LLM criteria assertion * feat: braintrust evals workflow * fix: BRAINTRUST_PROJECT_ID * feat: `sql_similar` assertion * fix: `OPENAI_API_KEY` in workflow env * feat: split AssertionScorer into separate scorers * feat: remove tutorial eval * feat: 20 minute CI timeout * feat: category in test case metadata * feat: score with gpt-5 * refactor: dataset to own file, colocate scorers * feat: "gpt-5.2-2025-12-11" for llm as a judge * feat: SQL syntax scorer with `libpg-query` * feat: `evals:setup` and `evals:run` scripts * feat: `evals:setup` in CI * feat: human readable scorer names * chore: rename to "SQL Validity" * feat: add 2 "sql_generation" test cases * feat: update requiredTools in test cases * chore: ignore Cursor MCP config * feat: "Conciseness" score * feat: "Completeness" scorer * fix: generate-v4 test mocks * feat: serialize "steps" for scorer inputs * updated node mem options for typecheck * updated runner * remove ram update as actions handle this * feat: read `BRAINTRUST_PROJECT_ID` from secrets * feat: score helpfulness, remove old scorers * feat: separate `evals:run` and `evals:upload` scripts * feat: passthrough entire classifier result * feat: use live `search_docs` impl, store docs result in metadata * feat: reduce classifier options * feat: filter workflow by `run-evals` PR label or `master` branch * chore: cleanup stubbed mock tools * fix: checkout actual branch with `ref:` * fix: capture search_docs results from all content parts * feat: simplify sql syntax score calculation * feat: use AI SDK's UI message validator * docs: justification for relative `extends` * fix: cleanup leftover validatedMessages * doc: note mock token isn't secret for snyk * fix: mock ui message to pass validation * feat: revert ignoring Cursor MCP config Using `.git/info/exclude` instead until we have an opinion on this * feat: add "tsconfig" as shared-data devDependency, revert relative path in tsconfig * refactor: tool call parsing into function * Update apps/studio/evals/assistant.eval.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * refactor: organize mock schemas and tool factories --------- Co-authored-by: Ali Waseem <waseema393@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>	2025-12-22 23:45:48 -05:00
3 people	626eb30e77	Assistant action orientated approach (#38806) ... * update onboarding * update model and fix part issue * action orientated assistant * fix tool * lock * remove unused filter * fix tests * fix again * update package * update container * fix tests * refactor(ai assistant): break out message markdown and profile picture * wip * refactor(ai assistant): break up message component * refactor: break ai assistant message down into multiple files * refactor: simplify ReportBlock state * fix: styling of draggable report block header When the drag handle is showing, it overlaps with the block header. Decrease the opacity of the header so the handle can be seen and the two can be distinguished. * fix: minor tweaks to tool ui * refactor: simplify DisplayBlockRenderer state * fix: remove double deploy button in edge function block When the confirm footer is shown, the deploy button on the top right should be hidden (not just disabled) to avoid confusion. * refactor, test: message sanitization by opt-in level Refactor the message sanitization to have more type safety and be more testable. Add tests to ensure: - Message sanitization always runs on generate-v4 - Message sanitization correctly works by opt-in level * Fix conflicts in pnpm lock * Couple of nits and refactors * Revert casing for report block snippet * adjust sanitised prompt * Fix tests --------- Co-authored-by: Charis Lam <26616127+charislam@users.noreply.github.com> Co-authored-by: Joshen Lim <joshenlimek@gmail.com>	2025-09-29 03:57:36 +00:00