mirror of
https://github.com/supabase/supabase.git
synced 2026-10-11 04:15:04 +03:00
Matt Rossman
517171b246
Lays groundwork for online evals on Assistant chat logs. https://www.braintrust.dev/docs/observe/score-online ### Changes - New workflows: - `braintrust-scorers-deploy.yml` keeps prod scorers in sync on push to `master` - `braintrust-preview-scorers-deploy.yml` deploys preview scorers to the staging project for PRs labeled `preview-scorers`, posting a comment with scorer links ([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222)) - `braintrust-preview-scorers-cleanup.yml` deletes preview scorers when the PR is closed ([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000749847)) - Adds `evals/scorer-online.ts` entry point invoked with `pnpm scorers:deploy`, registering scorers for online evals in the Braintrust "Assistant" project - Refactors scorer code to separate online-compatible scorers (`scorer-online.ts`) from WASM-dependent ones (`scorer-wasm.ts`) - "URL Validity" scorer now only checks Supabase domains to prevent requests to untrusted origins - Span `input` is now shaped `{ prompt: string }` instead of plain `string` for compatibility with offline eval scorers - Env vars `BRAINTRUST_STAGING_PROJECT_ID` and `BRAINTRUST_PROJECT_ID` configured in GitHub repo settings - `generateAssistantResponse` now uses `startSpan` + `withCurrent` instead of `traced()` to manually manage the root span lifecycle — this ensures `onFinish` logs output to the span _before_ `span.end()` is called, which is when Braintrust triggers scoring automations ### Online Scorers We share scoring logic across offline and online evals, but some of our scorers aren't transferrable to an "online" setting due to runtime challenges or ground truth requirements. **Supported** - Goal Completion - Conciseness - Completeness - Docs Faithfulness - URL Validity **Unsupported** - Correctness (requires ground truth output) - Tool Usage (requires ground truth requiredTools) - SQL Syntax (uses libpg-query WASM) - SQL Identifier Quoting (uses libpg-query WASM) ### How to use these scorers Going forward if you want to add/edit online eval scorers, add the `preview-scorers` label to a PR. This deploys scorers to the [Assistant (Staging Scorers)](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)?v=Overview) project in Braintrust with branch-specific slugs, and comments on the PR ([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222)). From the Braintrust dashboard you can "Test" the scorer with traces from any project. <img width="1866" height="528" alt="CleanShot 2026-03-05 at 15 15 00@2x" src="https://github.com/user-attachments/assets/4f15cebc-3f2d-4e8a-9ee2-fe8ef7bf4199" /> Once merged, scorers are deployed to the primary [Assistant](https://www.braintrust.dev/app/supabase.io/p/Assistant) project, and preview scorers are deleted from the staging project. Down the road, scorers on the Assistant project will run automatically on a sample of production traces. Closes AI-437
Supabase Studio
A dashboard for managing your self-hosted Supabase project, and used on our hosted platform. Built with:
What's included
Studio is designed to work with existing deployments - either the local hosted, docker setup, or our CLI. It is not intended for managing the deployment and administration of projects - that's out of scope.
As such, the features exposed on Studio for existing deployments are limited to those which manage your database:
- Table & SQL editors
- Saved queries are unavailable
- Database management
- Policies, roles, extensions, replication
- API documentation
Managing Project Settings
Project settings are managed outside of the Dashboard. If you use docker compose, you should manage the settings in your docker-compose file. If you're deploying Supabase to your own cloud, you should store your secrets and env vars in a vault or secrets manager.
How to contribute?
- Branch from
masterand name your branches with the following structure{type}/{branch_name}- Type:
chore | fix | feature - The branch name is arbitrary — just make sure it summarizes the work.
- Type:
- When you send a PR to
master, it will automatically tag members of the frontend team for review. - Review the contributing checklists to help test your feature before sending a PR.
- The Dashboard is under active development. You should run
git pullfrequently to make sure you're up to date.
Developer Quickstart
Note
Supabase internal use: To develop on Studio locally with the backend services, see the instructions in the internal
infrastructurerepo.
# You'll need to be on Node v20
# in /studio
## For external contributors
pnpm install # install dependencies
pnpm run dev # start dev server
## For internal contributors
## First clone the private supabase/platform repo and follow instructions for setting up mise
mise studio # Run from supabase/platform alongside `mise infra`
## For all
pnpm run test # run tests
pnpm run test -- --watch # run tests in watch mode
Running within a self-hosted environment
Follow the self-hosting guide to get started.
cd ..
cd docker
docker compose -f docker-compose.yml -f ./dev/docker-compose.dev.yml up
Once you've got that set up, update .env in the studio folder with the corresponding values.
POSTGRES_PASSWORD=
SUPABASE_ANON_KEY=
SUPABASE_SERVICE_KEY=
Then run the following commands to install dependencies and start the dashboard.
npm install
npm run dev
If you would like to configure different defaults for "Default Organization" and "Default Project", you will need to update the .env in the studio folder with the corresponding values.
DEFAULT_ORGANIZATION_NAME=
DEFAULT_PROJECT_NAME=