This PR fixes some prettier issues:
- Bump and unify all prettier versions to 3.7.3 across teh whole repo
- Bump the SQL prettier plugin
- When running `test:prettier`, check `mdx` files also
- Run the new prettier format on all files
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.
YES
## What kind of change does this PR introduce?
GitHub action permissions have changed, so because of that We need to
apply permissions on the parent
## Problem
Since the upgrade to NextJS 16.1.6, many e2e tests run fails with
timeout, probably when building or starting the studio
## Solution
- [x] Add dedicated steps for supabase setup and NextJS build to ease
debugging
- [x] Fix flakyness
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.
YES
## What kind of change does this PR introduce?
Our runners is running out of memory and CPU when running the build.
Just trying to bump up the blacksmith runner so they match the build
machine in Vercel.
## Problem
The docker build for studio fails because pnpm patches are not included
## Solution
- bump pnpm version in Dockefile to match the one used by the repo
- bump turbo version in Dockerfile to use one that handles pnpm patches
- add a script to test the build locally
- add a github action to validate the build on each PR
Lays groundwork for online evals on Assistant chat logs.
https://www.braintrust.dev/docs/observe/score-online
### Changes
- New workflows:
- `braintrust-scorers-deploy.yml` keeps prod scorers in sync on push to
`master`
- `braintrust-preview-scorers-deploy.yml` deploys preview scorers to the
staging project for PRs labeled `preview-scorers`, posting a comment
with scorer links
([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222))
- `braintrust-preview-scorers-cleanup.yml` deletes preview scorers when
the PR is closed
([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000749847))
- Adds `evals/scorer-online.ts` entry point invoked with `pnpm
scorers:deploy`, registering scorers for online evals in the Braintrust
"Assistant" project
- Refactors scorer code to separate online-compatible scorers
(`scorer-online.ts`) from WASM-dependent ones (`scorer-wasm.ts`)
- "URL Validity" scorer now only checks Supabase domains to prevent
requests to untrusted origins
- Span `input` is now shaped `{ prompt: string }` instead of plain
`string` for compatibility with offline eval scorers
- Env vars `BRAINTRUST_STAGING_PROJECT_ID` and `BRAINTRUST_PROJECT_ID`
configured in GitHub repo settings
- `generateAssistantResponse` now uses `startSpan` + `withCurrent`
instead of `traced()` to manually manage the root span lifecycle — this
ensures `onFinish` logs output to the span _before_ `span.end()` is
called, which is when Braintrust triggers scoring automations
### Online Scorers
We share scoring logic across offline and online evals, but some of our
scorers aren't transferrable to an "online" setting due to runtime
challenges or ground truth requirements.
**Supported**
- Goal Completion
- Conciseness
- Completeness
- Docs Faithfulness
- URL Validity
**Unsupported**
- Correctness (requires ground truth output)
- Tool Usage (requires ground truth requiredTools)
- SQL Syntax (uses libpg-query WASM)
- SQL Identifier Quoting (uses libpg-query WASM)
### How to use these scorers
Going forward if you want to add/edit online eval scorers, add the
`preview-scorers` label to a PR. This deploys scorers to the [Assistant
(Staging
Scorers)](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)?v=Overview)
project in Braintrust with branch-specific slugs, and comments on the PR
([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222)).
From the Braintrust dashboard you can "Test" the scorer with traces from
any project.
<img width="1866" height="528" alt="CleanShot 2026-03-05 at 15 15 00@2x"
src="https://github.com/user-attachments/assets/4f15cebc-3f2d-4e8a-9ee2-fe8ef7bf4199"
/>
Once merged, scorers are deployed to the primary
[Assistant](https://www.braintrust.dev/app/supabase.io/p/Assistant)
project, and preview scorers are deleted from the staging project. Down
the road, scorers on the Assistant project will run automatically on a
sample of production traces.
Closes AI-437
## Problem
e2e tests are still flaky and not as fast as they could be
## Solution
- [x] Reset supabase instance: this makes the database visualiser tests
faster as there are less tables to screenshot
- [x] Improve the single file setup utilities so that they never block
local tests by cleaning them up before starting
- [x] Disable animations while running the tests (less time waiting for
animations to complete
- [x] Add utility functions that help reproducing flaky tests locally
## Problem
On Safari, it's impossible to resize a column.
## Solution
This is actually a bug in `react-data-grid` that has been fixed in more
recent versions but we can't use them.
Patch the package
## How to test
- Open the any table in the _Table editor_
- Resize a column
This pull request refactors the GitHub Actions workflow for notifying
about stale Dashboard PRs by replacing custom JavaScript scripts and the
`actions/github-script` action with new TypeScript scripts that
communicate via standard input/output. This simplifies the workflow,
improves maintainability, and adds better error handling, especially for
API rate limits. The Slack notification script is also rewritten in
TypeScript and now reads PR data from stdin, making the workflow steps
more composable.
This pull request introduces a testing setup for the Next.js app using
Vitest. The main changes include the addition of a test configuration,
test scripts, and a sample test for the Next.js config. It also adds
relevant dependencies to support Vitest and path resolution, and updates
the lock file accordingly.
Testing infrastructure:
* Added a sample test file `next.config.test.ts` using Vitest to verify
that specific headers are present in the Next.js configuration.
* Created a Vitest configuration file `vitest.config.ts` with TypeScript
path resolution support via the `vite-tsconfig-paths` plugin.
Scripts and dependencies:
* Added `test` and `test:watch` scripts to `package.json` for running
and watching tests, and included `vitest` and `vite-tsconfig-paths` as
dev dependencies.
GITHUB_TOKEN is no longer allowed to create PRs, so use the generated
app token instead.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved authentication for automated sync workflows by switching to
an app-generated token, enhancing reliability of automated
documentation/troubleshooting syncs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
We're using the default GITHUB_TOKEN to create the ratchet baseline
decrease PR. By default, this does not auto-run the PR checks to prevent
infinite loops.
Switching to a GitHub App token instead so the PR checks will auto-run.
* trying arm runners for linting plus MORE MEM
* updated arm check s
* remove unneeded options
* smaller x86 runner to check performance
* Update .github/workflows/typecheck.yml
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* trying arm one more time
* remove arm its just way to slow
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* updated commands and expose ai key locally
* added tests for AI assistant
* added OPEN_API_KEY for e2e test suite
* updated log drain options
* updated README
* feat: test workflow with GitHub step summary
* fix: run test workflow on commit
* feat: tweak summary
* fix: format
* feat: format as table
* fix: wording
* fix: format
* feat: remote test workflow, add summary to `publish_image.yml`
* fix: wording
* added PR reminder action
* remove unneeded comment
* updated to trigger force
* updated to skip PRs that forks for now
* updated secret
* updated script
* updated env to pass correct values
* updated formatting
* [debug] just print the first block
* updated to remove dependabot
* updated to remove draft PRs
* updated to ignore merge conflicts
* removed merge conflict check
* updated time zone for PR reminders
* removed dashboard pr to push branch
* chore: add eslint ratchet script
* chore: add eslint ratchet action
* refactor(ratchet script): convert to typescript
* ci(ratchet script): add --decrease-baselines flag
Allows us to decrease the baselines on schedule as we fix warnings
* ci(ratchet): add action to decrease baseline if possible every week
* chore(eslint): fix exhaustive-deps error
* docs(internal): improve docs for eslint ratchet script
* chore(ratchet): add new ratchet rules
Add:
- import/no-anonymous-default-export
- @tanstack/query/exhaustive-deps
- @tanstack/query/no-deprecated-options
Not adding `no-restricted-exports` even though we have many violations
because we first need to reconfigure it (if possible) to ignore those
files where Next.js requires a default export.
* a new hope
* run tests in ci against cli mode
* summary
* try vercel action to run e2e against studio self hosted preview
* believe
* debug
* gh pages artifact
* test
* rm pages step
* fix automation bypass missing
* continue on error
* only install necessary deps for CI
* fix bypass
* remove
* fail job if test fails
* disable customer query if is_platform false
* vercel check
* fix var name, make comment update instead
* check bypass on runtime
* add env var
* fix tests going to project ref instead of default
* fix
* better dates in comment
* Update E2E test workflow to include flaky test detection and improve summary output
* fix
* fix dumb mistake
