Charis
8c409e2df5
## Problem Scorers previously derived the assistant's final answer via Braintrust's `trace.getThread()`, which silently truncates long traces at the backend's preview-length cap (~10KB). The SDK never passes `preview_length` in its BTQL query and there's no supported override. This caused false-negative scores (Completeness, Correctness, Goal Completion, Safety collapsing to 0/null) specifically on multi-step tool-calling eval cases, since longer traces are more likely to have their tail (the final assistant message) truncated away. ## Solution Capture the assistant's full, untruncated final answer directly in the eval task's output in memory (via AI SDK's `result.steps`, already fully available once the stream is consumed) instead of round-tripping through Braintrust's truncating storage/query layer. Scorers now read `output.transcript` instead of calling `trace.getThread()`. ## Changes - **New**: `apps/studio/evals/transcript.ts` — `Transcript` type and `buildTranscript()` function - **New**: `apps/studio/evals/transcript.test.ts` — unit tests (5 passing) - **Modified**: `apps/studio/evals/assistant.eval.ts` — captures `result.steps` and returns transcript - **Modified**: `apps/studio/evals/scorer.ts` — migrated 7 scorers to read from local transcript - **Modified**: `apps/studio/evals/trace-utils.ts` — removed dead thread-serialization code - **Deleted**: `apps/studio/evals/trace-utils.test.ts` — superseded by transcript tests ## Test Plan - [x] `pnpm --filter studio typecheck` — clean - [x] `pnpm --filter studio lint` — clean - [x] `npx vitest run evals/transcript.test.ts` — 5/5 passing - [x] Full live eval run (35/35 cases) against Braintrust — [experiment](https://www.braintrust.dev/app/supabase.io/p/Assistant/experiments/eval-scorer-transcript-capture-1786985352) shows Completeness/Correctness/Goal Completion/Safety scores comparable to baseline ## Known Residual Risk Other scorers that derive data from `trace.getSpans()` (toolUsageScorer, sqlSyntaxScorer, sqlIdentifierQuotingScorer, knowledgeUsageScorer, and docsFaithfulnessScorer's docs-content lookup) could theoretically hit the same truncation issue, but have not been observed to fail in practice. This is not addressed in this PR. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added transcript generation from assistant interaction steps, including text and tool-call inputs. * Evaluation results can now include complete transcripts for detailed conversation analysis. * Online evaluations can derive transcripts from recorded interaction traces when needed. * **Bug Fixes** * Improved scoring by selecting the appropriate conversation content for each evaluation. * Ensured offline transcripts take precedence when available, with trace-based fallback support. * **Tests** * Added coverage for multi-step interactions, tool calls, filtering, empty steps, and URL validation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Studio Assistant Evals
We use Braintrust to evaluate Assistant behaviors against a tracked dataset (offline evals) and against live traces (online evals).
Offline Evals
Add offline eval test cases to dataset.ts. If needed, add new scorers (see below) for the specific dimension you wish to test. Expect to update and run offline evals when adding new Assistant behaviors
You may wish to run offline evals when:
- You updated the eval suite with a new test case or scorer
- You changed Assistant's behavior and want to check for improvements/regressions
Running Offline Evals in CI
Add the run-evals label on a PR to the repo and Braintrust's GitHub Action will run evals and post a summary comment (example).
You can find detailed results in the "Experiments" tab of the "Assistant" project on Braintrust.
Running Offline Evals in Local Dev
Within apps/studio
# To set up WASM files
pnpm evals:setup
# Run all evals and upload results to Braintrust
pnpm evals:upload
# Run all evals without uploading results
pnpm evals:run
# Run an upload single test case
pnpm braintrust eval evals/assistant.eval.ts --filter "input.prompt=How many projects"
Upload results when you want to inspect Experiments or Logs in the Braintrust dashboard or API. You can use developer tools like Braintrust MCP or bt CLI to analyze results with an agent.
Scorers
Scorers look at a thread or task output and assign a score deterministically or via LLM-as-a-judge. Optionally they can consider expected values.
Define scorers in scorer.ts and include them in assistant.eval.ts to run them in offline evals.
Updating Online Scorers
Online scorers run as serverless functions on Braintrust infrastructure. They're deployed from the scorer-online.ts script. Since these scoring against production traces, they can't rely on ground truth expected values. Structure scoring logic and LLM prompts accordingly. Not every scorer needs to be an online scorer.
To opt-in to online scoring, add the scorer to scorer-online-manifest.json and add a corresponding handler in scorer-online.ts
Testing & Deploying Online Scorers
Add the preview-scorers label to a PR to deploy branch-prefixed scorers to the "Assistant (Staging Scorers)" Braintrust project (example). From that project dashboard, you can manually test the scorer against a trace from any project.
After merge to master, preview scorers automatically clean up and deploy to the production in the "Assistant" Braintrust project. Update the "Online Scoring" automation in the Logs page to include the new scorer function.