Commit Graph
297 Commits
Author SHA1 Message Date
Katerina Skroumpelou 4b2e9a206c fix(docs): scope js-libs-update vitest run and make tests self-contained (#46737)
The `Update JS Client Libraries Docs` workflow's "Refresh
reference-content snapshot" step fails because `npx vitest run --update
scripts/build-reference-content.test.ts` is treated as a filter, not a
restriction, under Vitest 4, so all docs tests
run. `CodeSample.test.ts` then fails with ENOENT on
`apps/docs/examples/_internal/fixtures/*` because the workflow bypasses
the `pretest` npm hook that runs `codegen:examples`.
  
This is a regression of #44943 (commit 28ef62cf31), reintroduced by
#46163 when the snapshot step was renamed and rewritten back to the
file-path form.
  
Fixes:
- Workflow: use `--dir scripts` so only the snapshot test runs.
- Tests: add a vitest globalSetup that mirrors `codegen:examples`, so
`npx vitest`, `pnpm test:local`, and IDE single-test runs no longer
depend on the `pretest` lifecycle hook being triggered first.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated test infrastructure configuration and initialization process
for improved test suite management.
* Refined test execution workflow to efficiently manage test discovery
and execution across project directories.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-08 15:33:20 +03:00
Andrey A. f5c49e89c5 chore(self-hosted): expand smoke tests (#46722) 2026-06-08 12:23:36 +02:00
f20cd22dc3 docs: improve SDK automation build step on docs (#46163)
# Second try of making a new better process for SDK automation

Instead of building a new pipeline. We will take the lessons learned
form round 1, plus the good design and improvement on DX quality for
drop-in file as a single step required from SDK team and produce almost
identical set of files as used right now to render using the current
pipeline.



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* New reference-content pipeline producing per-library reference
artifacts and integrating into prebuilds, search ingestion, and
rendering (type-aware examples).

* **Documentation**
* Added comprehensive JavaScript SDK v2 reference content and partials
(Auth MFA, passkeys, admin, TypeScript support, filters, modifiers,
Installing, Initializing, Buckets, etc.).

* **Tests & CI**
* Added regression snapshot test and updated workflows to refresh
reference snapshots and ensure spec downloads.

* **Chores**
* Updated ignore rules, build scripts, Makefile targets, and package
lifecycle hooks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Katerina Skroumpelou <mandarini@users.noreply.github.com>
Co-authored-by: Katerina Skroumpelou <sk.katherine@gmail.com>
2026-06-03 11:46:02 +03:00
Charis 6e3ffb2b0d fix(ci): authenticate git push in decrease-baselines workflow (#46596)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Bug fix (CI workflow).

## What is the current behavior?

The weekly `Decrease studio lint ratchet baselines` workflow fails on
`git push` with:

```
fatal: could not read Username for 'https://github.com': No such device or address
Error: Process completed with exit code 128.
```

`actions/checkout` runs with `persist-credentials: false`, so no auth is
stored for git. The job generates a GitHub App token but never wires it
into git, so the push has no credentials.

## What is the new behavior?

Push to an explicit token URL using the already-available `GH_TOKEN`,
matching the pattern in `.github/workflows/autofix_linters.yml`. The
force-push to `bot/decrease-eslint-ratchet-baselines` now authenticates
correctly.

## Additional context

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated GitHub Actions workflow configuration to improve automated
process reliability.

**Note:** This release contains no user-facing changes. The updates are
internal infrastructure improvements.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-02 17:06:56 -04:00
Etienne Stalmans 22388ee557 fix: create pr need contents-write (#46579)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES
## What kind of change does this PR introduce?

Bug fix - CI

## What is the current behavior?

action fails:
https://github.com/supabase/supabase/actions/runs/26818877603/job/79068294605
2026-06-02 12:29:39 +00:00
Charis f95fc2bf3b chore(ci): block merge while api-deploy-required label is present (#46482)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

CI / tooling — GitHub Actions workflow update.

## What is the current behavior?

PRs that modify `packages/api-types/types/**` can be merged before the
corresponding API has shipped to production, breaking the Studio
frontend when it calls endpoints that do not exist yet. The
`api-deploy-required` label is enforced only by convention and code
review, which is easy to miss.

## What is the new behavior?

- `.github/labeler.yml`: auto-applies `api-deploy-required` to any PR
that touches `packages/api-types/types/**`.
- `.github/workflows/label_prs.yml`: drops the `apps/docs/**/*` path
filter so the labeler runs on all PRs, and posts a one-time comment when
`api-deploy-required` is newly added (uses the labeler's `new-labels`
output so re-pushes do not re-comment).
- `.github/workflows/validate-pr.yml`: adds a step that fails the
`Validate pull request` check while the `api-deploy-required` label is
present, mirroring the existing `do-not-merge` pattern. The author
removes the label after confirming the API is live to unblock merge.

Reviewer: please confirm `Validate pull request` is configured as a
required check on `master` in branch protection — that step is what
enforces the block.

## Additional context

Resolves FE-3479

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Enhanced pull request automation with improved labeling rules for
API-related changes.
* Added validation that blocks pull request merging until API deployment
is confirmed for changes affecting API types.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46482?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-28 14:37:06 -04:00
Etienne StalmansandAli Waseem c9cc6cd835 chore: cleanup gh actions (#46454)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Chore, CI hardening


## Additional context

Hardens all GitHub actions to recommendations of
[zizmor](https://docs.zizmor.sh/audits/)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Disabled persistence of checkout credentials across many CI workflows
to reduce credential exposure.
* Upgraded GitHub App token tooling and tightened generated token
permissions for automation.
* Added cooldown/rate-limiting to dependency update automation to reduce
update churn.
* Adjusted workflow-level permissions, required secret inputs for
workflow callers, and refactored some job step logic.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46454?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Ali Waseem <waseema393@gmail.com>
2026-05-28 17:31:04 +02:00
Charis cc0b2d3d21 chore(studio): remove require-safe-sql-fragment ESLint rule (#46079)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Refactor / chore.

## What is the current behavior?

A custom type-aware ESLint rule (`studio/require-safe-sql-fragment`)
enforces that the `sql` argument to `executeSql` is a `SafeSqlFragment`.
It runs in a separate `eslint.type-checks.config.cjs` and a dedicated CI
ratchet step, and pulls in `@typescript-eslint/utils` as a direct dev
dependency.

## What is the new behavior?

`SafeSqlFragment` enforcement is now handled entirely by TypeScript
compilation. The ESLint rule, its dedicated config, the ratchet
baselines for it, the CI step, and the `@typescript-eslint/utils` direct
dev dependency have all been removed.

## Additional context

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Simplified development linting workflow by removing type-aware ESLint
checks and associated rule files.
* Cleaned up ESLint configuration and dependencies in the studio
application.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46079?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-19 20:30:23 +00:00
Stephen Morgan 1e28e6ac01 fix: run format step in separate low-privilege job (#46090)
Out of an abundance of caution, running the format step in a strictly
read-only context.
Added explicit validations on pushes instead of relying on an implicit
lack of pull_request_target

Resolves PRODSEC-118

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Reworked automated autofix workflow into a two-step format-and-apply
flow, reduced top-level permissions, and moved from an auto-commit
action to a manual apply/commit/push process for more controlled
automated fixes.
  * No user-facing features or functionality changed.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46090?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-19 17:54:12 +12:00
Gildas Garcia 1e59ef9a06 chore: balance e2e shards load (#45661)
## Problem

We run the e2e tests across two shards to reduce their duration.
However, one takes longer than the other.

## Solution

Use a currently hidden Playwright feature to balance their load

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Upgraded Playwright test dependency to a newer version for refreshed
test tooling and fixes.
* Adjusted end-to-end test shard weighting to rebalance distribution
across parallel runs.

* **Tests**
* Stabilized several end-to-end editor interactions to make E2E flows
more reliable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-07 06:53:09 -06:00
Gildas Garcia f809d1c99a chore: fix e2e flaky Supabase CLI start (#45637)
## Problem

The Supabase CLI sometimes fails to start because the postgres port is
already used:
> failed to start docker container: Error response from daemon: failed
to set up container networking: driver failed programming external
connectivity on endpoint supabase_db_supabase
(10587f26e7287c8086fd27db12be7aac4afe1f3fc521c9d7a8dfe109993c995d):
failed to bind host port for 0.0.0.0:54322:172.18.0.2:5432/tcp: address
already in use

## Solution

None yet, debugging

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Enhanced end-to-end test workflow with added pre-start and failure
diagnostics for better observability and troubleshooting of test runs.
* Updated the e2e setup CLI script to skip certain local services by
default and clarify script formatting for maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-06 16:02:30 +02:00
Inder Singh d3bce8fc29 chore(self-hosted): use quiet pull for compose in smoke test (#45216) 2026-04-24 10:38:35 +00:00
Katerina Skroumpelou 28ef62cf31 fix(docs): scope vitest run to target dir in js-libs-update workflow (#44943)
Vitest 4 no longer restricts test execution when a file path is passed
as a positional filter, causing all docs tests to run and fail on
unrelated tests. Use --dir to scope to the target directory. PR #44833
made the update which broke the behaviour.
2026-04-16 14:07:16 +00:00
Katerina Skroumpelou d5c8f2cd3f chore: @supabase/ssr version management (#44705)
* Add `@supabase/ssr` to catalog
* Use `catalog:` version of `@supabase/ssr` across repo
* Add workflow to update `@supabase/ssr`
* Switch runners for package and docs updates to `ubuntu-latest` (free,
lighter)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Centralized `@supabase/ssr` versioning across the workspace via the
package catalog.
* Updated CI workflow runners to a more consistent execution
environment.
* Added an automated workflow to streamline updating the `@supabase/ssr`
package and refreshing dependency lockfiles.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-15 13:55:19 +03:00
Charis bf46092290 ci: add safesql ratchet (#44678)
We are currently migrating to the safeSql utility for all SQL arguments
of executeSql. During the migration, executeSql will continue to accept
plain strings for backwards compatibility. Adding a custom ESLint rule
so we can ratchet this and prevent new calls of executeSql with plain
strings.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added SQL safety validation throughout the application to enforce
secure query construction and prevent SQL-related vulnerabilities
* Introduced type-aware linting to identify and catch type-related
issues during development and continuous integration processes

* **Chores**
* Enhanced continuous integration pipeline with improved code quality
enforcement

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-09 15:32:33 -04:00
Andrey A. 0950b99bf0 chore: add a nofile limit for supavisor in smoke test (#44582) 2026-04-07 10:46:28 +02:00
Ali Waseem db281bd1d1 feat: alert in Slack when master breaks (#44456)
## Summary
- Adds a new GitHub Actions workflow that sends a Slack alert when
Studio E2E tests or unit tests fail on master
- Uses `workflow_run` trigger so no changes needed to existing CI
workflows
- Reuses the existing `SLACK_DASHBOARD_WEBHOOK_URL` secret

Closes FE-2883

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Implemented automated Slack notifications to alert when master branch
build failures occur, including build run details, commit information,
and commit author.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-02 07:16:06 -06:00
Katerina Skroumpelou 75e1bbb4a6 ci: add release notes on update js libs pr (#44241)
Include release notes in the automated PR that updates supabase-js
across the repo.

* Looks at what version is installed
* It includes all changelogs between installed version and
to-be-installed version
2026-03-27 15:07:43 +02:00
Etienne Stalmans 55852a0eb1 chore: pin all actions (#42068)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

chore


## Additional context

We will require all actions to be pinned going forward


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated CI/CD workflows to use pinned action versions for improved
reliability and consistency.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-03-26 08:01:12 +01:00
Katerina Skroumpelou 23fa582695 ci: fix mgmt api docs generator (#44121)
The [`docs-mgmt-api-update`
workflow](https://github.com/supabase/supabase/blob/master/.github/workflows/docs-mgmt-api-update.yml#L21)
uses `sparse-checkout`, only pulling `apps/docs` and patches. After PR
#42987, the Makefile's `redocly` commands were changed to run via `pnpm
exec` `redocly` from `packages/generator`, but that directory was never
checked out, causing the can't `cd` error. Adding `packages/generator`
to `sparse-checkout` makes the existing Makefile dependency explicit in
the workflow.
2026-03-24 12:00:32 +02:00
Andrey A. c3701936b7 fix(self-hosted): increase timeouts for supavisor (#43905) 2026-03-18 10:47:03 +00:00
Ivan Vasilov d07e78a616 chore: Run prettier on examples folder (#43851)
This PR runs `prettier` on the `examples` folder.

Depends on https://github.com/supabase/supabase/pull/43849.
2026-03-18 10:51:10 +01:00
Ivan VasilovandJoshen Lim 9fa96977be chore: Minor prettier fixes (#43849)
This PR fixes some prettier issues:
- Bump and unify all prettier versions to 3.7.3 across teh whole repo
- Bump the SQL prettier plugin
- When running `test:prettier`, check `mdx` files also
- Run the new prettier format on all files

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2026-03-17 11:17:42 +01:00
Ali Waseem 4d967740f7 fix permissions for docker image publish (#43823)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

GitHub action permissions have changed, so because of that We need to
apply permissions on the parent
2026-03-16 15:51:55 +00:00
Gildas Garcia 43e793f5ad chore: fix e2e flakyness (#43739)
## Problem

Since the upgrade to NextJS 16.1.6, many e2e tests run fails with
timeout, probably when building or starting the studio

## Solution

- [x] Add dedicated steps for supabase setup and NextJS build to ease
debugging
- [x] Fix flakyness
2026-03-16 15:51:20 +01:00
Ali Waseem 3172fc47fe fix: bump up runner due to resources (#43722)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Our runners is running out of memory and CPU when running the build.
Just trying to bump up the blacksmith runner so they match the build
machine in Vercel.
2026-03-12 12:00:45 -06:00
Gildas Garcia 2773179a5b fix: studio docker build fails because of package patches (#43713)
## Problem

The docker build for studio fails because pnpm patches are not included

## Solution

- bump pnpm version in Dockefile to match the one used by the repo
- bump turbo version in Dockerfile to use one that handles pnpm patches
- add a script to test the build locally
- add a github action to validate the build on each PR
2026-03-12 16:24:07 +01:00
Katerina SkroumpelouandIvan Vasilov 77e5180c5c chore: format files (#43668)
Format files. Run `pnpm format` at root.

---------

Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
2026-03-11 18:40:01 +00:00
Etienne Stalmans 92bdbb2f57 chore: explicit permissions on actions (#43526)
enforces minimal set of permissions, the same as other actions in the
repo
2026-03-10 14:47:26 +01:00
Matt Rossman 517171b246 feat(assistant): online evals support and CI workflows (#43194)
Lays groundwork for online evals on Assistant chat logs.

https://www.braintrust.dev/docs/observe/score-online

### Changes

- New workflows:
- `braintrust-scorers-deploy.yml` keeps prod scorers in sync on push to
`master`
- `braintrust-preview-scorers-deploy.yml` deploys preview scorers to the
staging project for PRs labeled `preview-scorers`, posting a comment
with scorer links
([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222))
- `braintrust-preview-scorers-cleanup.yml` deletes preview scorers when
the PR is closed
([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000749847))
- Adds `evals/scorer-online.ts` entry point invoked with `pnpm
scorers:deploy`, registering scorers for online evals in the Braintrust
"Assistant" project
- Refactors scorer code to separate online-compatible scorers
(`scorer-online.ts`) from WASM-dependent ones (`scorer-wasm.ts`)
- "URL Validity" scorer now only checks Supabase domains to prevent
requests to untrusted origins
- Span `input` is now shaped `{ prompt: string }` instead of plain
`string` for compatibility with offline eval scorers
- Env vars `BRAINTRUST_STAGING_PROJECT_ID` and `BRAINTRUST_PROJECT_ID`
configured in GitHub repo settings
- `generateAssistantResponse` now uses `startSpan` + `withCurrent`
instead of `traced()` to manually manage the root span lifecycle — this
ensures `onFinish` logs output to the span _before_ `span.end()` is
called, which is when Braintrust triggers scoring automations

### Online Scorers

We share scoring logic across offline and online evals, but some of our
scorers aren't transferrable to an "online" setting due to runtime
challenges or ground truth requirements.

**Supported**
- Goal Completion
- Conciseness
- Completeness
- Docs Faithfulness
- URL Validity

**Unsupported**
- Correctness (requires ground truth output)
- Tool Usage (requires ground truth requiredTools)
- SQL Syntax (uses libpg-query WASM)
- SQL Identifier Quoting (uses libpg-query WASM)
 
### How to use these scorers

Going forward if you want to add/edit online eval scorers, add the
`preview-scorers` label to a PR. This deploys scorers to the [Assistant
(Staging
Scorers)](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)?v=Overview)
project in Braintrust with branch-specific slugs, and comments on the PR
([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222)).
From the Braintrust dashboard you can "Test" the scorer with traces from
any project.

<img width="1866" height="528" alt="CleanShot 2026-03-05 at 15 15 00@2x"
src="https://github.com/user-attachments/assets/4f15cebc-3f2d-4e8a-9ee2-fe8ef7bf4199"
/>

Once merged, scorers are deployed to the primary
[Assistant](https://www.braintrust.dev/app/supabase.io/p/Assistant)
project, and preview scorers are deleted from the staging project. Down
the road, scorers on the Assistant project will run automatically on a
sample of production traces.

Closes AI-437
2026-03-09 13:05:26 -04:00
Gildas Garcia 13bfc4502a chore: e2e general improvements (#43547)
## Problem

e2e tests are still flaky and not as fast as they could be

## Solution

- [x] Reset supabase instance: this makes the database visualiser tests
faster as there are less tables to screenshot
- [x] Improve the single file setup utilities so that they never block
local tests by cleaning them up before starting
- [x] Disable animations while running the tests (less time waiting for
animations to complete
- [x] Add utility functions that help reproducing flaky tests locally
2026-03-09 15:38:06 +01:00
Gildas Garcia 77004016f4 fix: cannot resize table editor columns on safari (#43450)
## Problem

On Safari, it's impossible to resize a column.

## Solution 

This is actually a bug in `react-data-grid` that has been fixed in more
recent versions but we can't use them.
Patch the package

## How to test

- Open the any table in the _Table editor_
- Resize a column
2026-03-06 08:01:22 +01:00
Gildas GarciaandCharis 82c5ffb19f Configure sharding for Studio e2e tests (#43211)
## Problem

The e2e test suite for Studio takes around 20min. Feedback loop is too
long

## Solution

Try enabling [playwright
sharding](https://playwright.dev/docs/test-sharding#merge-reports-cli).
Only 2 shards for now

## Results

Before:
<img width="867" height="113" alt="image"
src="https://github.com/user-attachments/assets/aef01026-0eaf-4bd0-9a56-c0f5123dd2cc"
/>
<img width="845" height="115" alt="image"
src="https://github.com/user-attachments/assets/46fcf523-8128-4cf4-83cb-3f4e414337d8"
/>

After:
<img width="716" height="105" alt="image"
src="https://github.com/user-attachments/assets/ada0ffd7-3a9f-4ca0-9b2e-98b21df3188b"
/>

## Next steps

In future dedicated PRs, improve the tests themselves.

---------

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>
2026-03-02 16:43:47 +01:00
Joshen Lim 1b5e472340 Attempt to fix coveralls upload step (#43285) 2026-03-02 15:34:57 +00:00
Jordi Enric a9a6357227 fix: tests workflow blocking non-studio/ui prs (#43185) 2026-02-25 17:03:44 +01:00
Jordi EnricandJoshen Lim 26777710d2 remove fail-on-error (#43175)
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2026-02-25 22:45:33 +08:00
Charis 9923496486 chore: update prose linter (#43147) 2026-02-25 09:39:21 -05:00
Jordi Enric 03351c784a chore: split tests running from coveralls upload (#43171)
this is so we can easily tell when tests failed vs when coveralls failed
in the PR checks / github action breakdown.

<img width="1510" height="316" alt="CleanShot 2026-02-25 at 11 28 00@2x"
src="https://github.com/user-attachments/assets/ed119654-0341-4554-a2a1-e95f4c7a0995"
/>
2026-02-25 20:53:19 +08:00
Ivan Vasilov 03660838ef chore: Refactor the script for posting PRs for review on Slack (#43022)
This pull request refactors the GitHub Actions workflow for notifying
about stale Dashboard PRs by replacing custom JavaScript scripts and the
`actions/github-script` action with new TypeScript scripts that
communicate via standard input/output. This simplifies the workflow,
improves maintainability, and adds better error handling, especially for
API rate limits. The Slack notification script is also rewritten in
TypeScript and now reads PR data from stdin, making the workflow steps
more composable.
2026-02-20 15:32:12 +01:00
Ivan Vasilov a41d7138a5 feat: add vitest configuration and tests for X-Robots-Tag headers (#42873)
This pull request introduces a testing setup for the Next.js app using
Vitest. The main changes include the addition of a test configuration,
test scripts, and a sample test for the Next.js config. It also adds
relevant dependencies to support Vitest and path resolution, and updates
the lock file accordingly.

Testing infrastructure:

* Added a sample test file `next.config.test.ts` using Vitest to verify
that specific headers are present in the Next.js configuration.
* Created a Vitest configuration file `vitest.config.ts` with TypeScript
path resolution support via the `vite-tsconfig-paths` plugin.

Scripts and dependencies:

* Added `test` and `test:watch` scripts to `package.json` for running
and watching tests, and included `vitest` and `vite-tsconfig-paths` as
dev dependencies.
2026-02-17 07:15:55 -07:00
Charis aa9a10e65d fix(docs): sync troubleshooting upstream workflow (#42463)
GITHUB_TOKEN is no longer allowed to create PRs, so use the generated
app token instead.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Improved authentication for automated sync workflows by switching to
an app-generated token, enhancing reliability of automated
documentation/troubleshooting syncs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-05 10:20:44 +11:00
Chris Chinchilla 91072db7b5 docs: Add documentation label to troubleshooting sync workflow (#41939)
Add documentation label to troubleshooting sync workflow
2026-01-16 15:24:25 +07:00
Charis 30673a3644 ci(studio): use app token for ratchet decrease workflow (#41799)
We're using the default GITHUB_TOKEN to create the ratchet baseline
decrease PR. By default, this does not auto-run the PR checks to prevent
infinite loops.

Switching to a GitHub App token instead so the PR checks will auto-run.
2026-01-08 12:43:26 -05:00
Ivan Vasilov b7bac67075 fix: Use head_commit_sha in authorize-vercel-deploys.yml action (#41795)
* Use head commit sha in the authorize-vercel-deploys.yml.

* Add comments.
2026-01-08 16:33:14 +01:00
Ivan Vasilov d59711effd fix: Fix the trigger workflow name for the approve-vercel-deploys action (#41791)
Fix the wrong name of the GH workflow.
2026-01-08 14:27:30 +01:00
Ivan Vasilov 6f24342b55 fix: Change the authorize-vercel-builds to be triggered by another action (#41789)
Refactor the authorize-vercel-builds to run triggered by another action in order to access secrets.
2026-01-08 14:15:38 +01:00
Ivan Vasilov cdadd74c9b feat: Add GH workflow for authorizing Vercel deploys (#41717)
* Add a script and call it from a GH action.

* Minor updates.

* Minor fixes.
2026-01-08 12:11:05 +01:00
Katerina Skroumpelou 8c367d0c86 ci: use github app generated token (#41767)
* ci: use github app generated token

* ci: pin action to commit sha
2026-01-07 13:27:04 +00:00
072883bcec feat: assistant evals (#41311)
* chore: bump `supabase` CLI

* chore: stricter message types in `generate-v4.ts`

* feat: tutorial eval

https://www.braintrust.dev/docs/evaluation

* feat: project ID for eval

* refactor: `generateAssistantResponse` out of `handlePost`

* refactor: generateAssistantResponse to lib/ai

* feat: factuality eval with assistant response

* chore: upgrade braintrust to v1.0.1

* chore: silence tsconfig warning

* feat: assertion scorer

* fix: aggregate tools across all steps

* refactor: strict tool names, remove need for `as const`

* refactor: generic tool name type in assertions

* feat: transfer mocks from `feature/braintrust`

* feat: LLM criteria assertion

* feat: braintrust evals workflow

* fix: BRAINTRUST_PROJECT_ID

* feat: `sql_similar` assertion

* fix: `OPENAI_API_KEY` in workflow env

* feat: split AssertionScorer into separate scorers

* feat: remove tutorial eval

* feat: 20 minute CI timeout

* feat: category in test case metadata

* feat: score with gpt-5

* refactor: dataset to own file, colocate scorers

* feat: "gpt-5.2-2025-12-11" for llm as a judge

* feat: SQL syntax scorer with `libpg-query`

* feat: `evals:setup` and `evals:run` scripts

* feat: `evals:setup` in CI

* feat: human readable scorer names

* chore: rename to "SQL Validity"

* feat: add 2 "sql_generation" test cases

* feat: update requiredTools in test cases

* chore: ignore Cursor MCP config

* feat: "Conciseness" score

* feat: "Completeness" scorer

* fix: generate-v4 test mocks

* feat: serialize "steps" for scorer inputs

* updated node mem options for typecheck

* updated runner

* remove ram update as actions handle this

* feat: read `BRAINTRUST_PROJECT_ID` from secrets

* feat: score helpfulness, remove old scorers

* feat: separate `evals:run` and `evals:upload` scripts

* feat: passthrough entire classifier result

* feat: use live `search_docs` impl, store docs result in metadata

* feat: reduce classifier options

* feat: filter workflow by `run-evals` PR label or `master` branch

* chore: cleanup stubbed mock tools

* fix: checkout actual branch with `ref:`

* fix: capture search_docs results from all content parts

* feat: simplify sql syntax score calculation

* feat: use AI SDK's UI message validator

* docs: justification for relative `extends`

* fix: cleanup leftover validatedMessages

* doc: note mock token isn't secret for snyk

* fix: mock ui message to pass validation

* feat: revert ignoring Cursor MCP config

Using `.git/info/exclude` instead until we have an opinion on this

* feat: add "tsconfig" as shared-data devDependency, revert relative path in tsconfig

* refactor: tool call parsing into function

* Update apps/studio/evals/assistant.eval.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* refactor: organize mock schemas and tool factories

---------

Co-authored-by: Ali Waseem <waseema393@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-12-22 23:45:48 -05:00
Ali Waseemandcoderabbitai[bot] 3d178fcad2 chore: trying arm runners for linting plus MORE MEM (#41472)
* trying arm runners for linting plus MORE MEM

* updated arm check s

* remove unneeded options

* smaller x86 runner to check performance

* Update .github/workflows/typecheck.yml

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* trying arm one more time

* remove arm its just way to slow

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-12-18 21:09:18 +00:00