Commit Graph
28 Commits
Author SHA1 Message Date
072883bcec feat: assistant evals (#41311)
* chore: bump `supabase` CLI

* chore: stricter message types in `generate-v4.ts`

* feat: tutorial eval

https://www.braintrust.dev/docs/evaluation

* feat: project ID for eval

* refactor: `generateAssistantResponse` out of `handlePost`

* refactor: generateAssistantResponse to lib/ai

* feat: factuality eval with assistant response

* chore: upgrade braintrust to v1.0.1

* chore: silence tsconfig warning

* feat: assertion scorer

* fix: aggregate tools across all steps

* refactor: strict tool names, remove need for `as const`

* refactor: generic tool name type in assertions

* feat: transfer mocks from `feature/braintrust`

* feat: LLM criteria assertion

* feat: braintrust evals workflow

* fix: BRAINTRUST_PROJECT_ID

* feat: `sql_similar` assertion

* fix: `OPENAI_API_KEY` in workflow env

* feat: split AssertionScorer into separate scorers

* feat: remove tutorial eval

* feat: 20 minute CI timeout

* feat: category in test case metadata

* feat: score with gpt-5

* refactor: dataset to own file, colocate scorers

* feat: "gpt-5.2-2025-12-11" for llm as a judge

* feat: SQL syntax scorer with `libpg-query`

* feat: `evals:setup` and `evals:run` scripts

* feat: `evals:setup` in CI

* feat: human readable scorer names

* chore: rename to "SQL Validity"

* feat: add 2 "sql_generation" test cases

* feat: update requiredTools in test cases

* chore: ignore Cursor MCP config

* feat: "Conciseness" score

* feat: "Completeness" scorer

* fix: generate-v4 test mocks

* feat: serialize "steps" for scorer inputs

* updated node mem options for typecheck

* updated runner

* remove ram update as actions handle this

* feat: read `BRAINTRUST_PROJECT_ID` from secrets

* feat: score helpfulness, remove old scorers

* feat: separate `evals:run` and `evals:upload` scripts

* feat: passthrough entire classifier result

* feat: use live `search_docs` impl, store docs result in metadata

* feat: reduce classifier options

* feat: filter workflow by `run-evals` PR label or `master` branch

* chore: cleanup stubbed mock tools

* fix: checkout actual branch with `ref:`

* fix: capture search_docs results from all content parts

* feat: simplify sql syntax score calculation

* feat: use AI SDK's UI message validator

* docs: justification for relative `extends`

* fix: cleanup leftover validatedMessages

* doc: note mock token isn't secret for snyk

* fix: mock ui message to pass validation

* feat: revert ignoring Cursor MCP config

Using `.git/info/exclude` instead until we have an opinion on this

* feat: add "tsconfig" as shared-data devDependency, revert relative path in tsconfig

* refactor: tool call parsing into function

* Update apps/studio/evals/assistant.eval.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* refactor: organize mock schemas and tool factories

---------

Co-authored-by: Ali Waseem <waseema393@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-12-22 23:45:48 -05:00
Saxon FletcherandCharis 7e6796125c AI model selection (#39472)
* expose model and model selection

* tooltip

* simplify

* Update ai-assistant-state.tsx

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

* Update ModelSelector.tsx

---------

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>
2025-10-17 11:17:43 +10:00
Greg Richardson d94f1f9b3a fix(ai): increase request body size limit (#39330)
* fix(ai): increase request body size limit

* fix: remove incorrect config type
2025-10-07 11:31:54 -06:00
54ad0b7495 Chore/ai realtime (#39145)
* update onboarding

* update model and fix part issue

* action orientated assistant

* fix tool

* lock

* remove unused filter

* fix tests

* fix again

* update package

* update container

* fix tests

* ai realtime

* ai realtime

* refactor(ai assistant): break out message markdown and profile picture

* wip

* refactor(ai assistant): break up message component

* refactor: break ai assistant message down into multiple files

* add limitations prompt

* limitations prompt

* link prompt

* refactor: simplify ReportBlock state

* fix: styling of draggable report block header

When the drag handle is showing, it overlaps with the block header.
Decrease the opacity of the header so the handle can be seen and the two
can be distinguished.

* fix: minor tweaks to tool ui

* refactor: simplify DisplayBlockRenderer state

* fix: remove double deploy button in edge function block

When the confirm footer is shown, the deploy button on the top right should be
hidden (not just disabled) to avoid confusion.

* refactor, test: message sanitization by opt-in level

Refactor the message sanitization to have more type safety and be more testable.
Add tests to ensure:

- Message sanitization always runs on generate-v4
- Message sanitization correctly works by opt-in level

* Fix conflicts in pnpm lock

* Couple of nits and refactors

* Revert casing for report block snippet

* adjust sanitised prompt

* Fix tests

* empty states

* prompt otpimise

* refine prompt

* prompt optimizer

* remove realtime for now

* Update apps/studio/lib/ai/prompts.ts

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

* Update apps/studio/lib/ai/prompts.ts

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

* updates

* feature flag

* use flag and additional check

* remove sort

* messages copy

---------

Co-authored-by: Charis Lam <26616127+charislam@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2025-10-06 10:31:40 +10:00
1973615a42 Prompt GPT-5 Optimizations (#39031)
* update onboarding

* update model and fix part issue

* action orientated assistant

* fix tool

* lock

* remove unused filter

* fix tests

* fix again

* update package

* update container

* fix tests

* refactor(ai assistant): break out message markdown and profile picture

* wip

* refactor(ai assistant): break up message component

* refactor: break ai assistant message down into multiple files

* add limitations prompt

* limitations prompt

* link prompt

* refactor: simplify ReportBlock state

* fix: styling of draggable report block header

When the drag handle is showing, it overlaps with the block header.
Decrease the opacity of the header so the handle can be seen and the two
can be distinguished.

* fix: minor tweaks to tool ui

* refactor: simplify DisplayBlockRenderer state

* fix: remove double deploy button in edge function block

When the confirm footer is shown, the deploy button on the top right should be
hidden (not just disabled) to avoid confusion.

* refactor, test: message sanitization by opt-in level

Refactor the message sanitization to have more type safety and be more testable.
Add tests to ensure:

- Message sanitization always runs on generate-v4
- Message sanitization correctly works by opt-in level

* Fix conflicts in pnpm lock

* Couple of nits and refactors

* Revert casing for report block snippet

* adjust sanitised prompt

* Fix tests

* prompt optimizer

* remove realtime for now

* Update apps/studio/lib/ai/prompts.ts

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

* Update apps/studio/lib/ai/prompts.ts

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

---------

Co-authored-by: Charis Lam <26616127+charislam@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2025-09-29 14:07:53 +10:00
626eb30e77 Assistant action orientated approach (#38806)
* update onboarding

* update model and fix part issue

* action orientated assistant

* fix tool

* lock

* remove unused filter

* fix tests

* fix again

* update package

* update container

* fix tests

* refactor(ai assistant): break out message markdown and profile picture

* wip

* refactor(ai assistant): break up message component

* refactor: break ai assistant message down into multiple files

* refactor: simplify ReportBlock state

* fix: styling of draggable report block header

When the drag handle is showing, it overlaps with the block header.
Decrease the opacity of the header so the handle can be seen and the two
can be distinguished.

* fix: minor tweaks to tool ui

* refactor: simplify DisplayBlockRenderer state

* fix: remove double deploy button in edge function block

When the confirm footer is shown, the deploy button on the top right should be
hidden (not just disabled) to avoid confusion.

* refactor, test: message sanitization by opt-in level

Refactor the message sanitization to have more type safety and be more testable.
Add tests to ensure:

- Message sanitization always runs on generate-v4
- Message sanitization correctly works by opt-in level

* Fix conflicts in pnpm lock

* Couple of nits and refactors

* Revert casing for report block snippet

* adjust sanitised prompt

* Fix tests

---------

Co-authored-by: Charis Lam <26616127+charislam@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2025-09-29 03:57:36 +00:00
Greg Richardson 9820707d71 feat: local mcp server (#38797)
* feat: local mcp server

* feat(local-mcp): implement migrations

* fix: remove unsupported mcp args

* feat(local-mcp): tests

* fix(local-mcp): packages to adhere to minimumReleaseAge

* fix(mcp): import path for createSupabaseApiPlatform

* fix(local-mcp): move tests out of pages/api dir

* refactor: self-hosted execute sql logic

* fix: deps
2025-09-23 12:00:08 -06:00
Saxon Fletcher d276aca6cb Chore/ai flow model (#38805)
* update onboarding

* update model and fix part issue

* lock

* remove unused filter

* fix tests

* fix again

* update package

* fix merge issue
2025-09-23 12:46:20 +10:00
Saxon FletcherandJoshen Lim bf004b8a1d Choose provider and model (#38267)
* choose provider and model

* fix test

* Fix lint

* Nit clean up

* add provider options and remove anthropic

* remove package

* use chat model and update prompt

* prompt rendering

* default to pro

* fix test

* low

* remove log

* remove redundant part.text

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2025-09-09 10:05:39 +00:00
cddc472705 Test GPT-OSS (#37710)
* try a really long context window to maximize caching

* update examples

* attempt to update packages and useChat

* update endpoints

* update zod

* zod

* update to v5

* message update

* Revert "zod"

This reverts commit ec39bac6b6.

* revert zod

* zod i

* fix complete endpoints

* remove async

* change to content

* type cleanup

* Revert the package bumps to rebuild them.

* Bump zod to 2.25.76 in all packages.

* Bump openai in all packages.

* Bump ai and ai-related packages.

* Remove unneeded files.

* Fix the rest of the migration stuff.

* Prettier fixes.

* testing new gptoss models

* sonnet as pro

* back to 3.7

* revert test

* add policy list tool

* fix

* refactor

* ai sdk 5 fixes

* refactor complete endpoint

* edge function prompt

* remove example

* use limited model for completions

* remove duplicate

* Update bedrock.ts

Co-authored-by: divit <27228526+delgado3d@users.noreply.github.com>

* add default check

* only add prefix if more then one region

* temp use api

* refactor to use openai api

* test

* remove unused import

* Prettier lint

* Fix test

* revert back to bedrock

* Tiny style fix for spacing of code blocks from edit message button

* try generate text normally

* prompt adjust

* add list style to prompt

* prompt adjust

* prompt adjust emphasise display query

* update prompts

* prompt adjust

* remove package

* prompt adjust

---------

Co-authored-by: Jordi Enric <jordi.err@gmail.com>
Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
Co-authored-by: divit <27228526+delgado3d@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2025-08-27 11:34:44 +07:00
Saxon Fletcher adfc138450 Remove error display temporarily (#37804) 2025-08-09 14:56:05 +10:00
TaishiandJoshen Lim 81e1acc2d8 Add ability to stop AI Assistant responses mid-stream (#37628)
* get `stop()` from useChat and use it

* Implement abort signal in `generate-v4` API

* Enhance `onStop` functionality in AIAssistant to store partial message when aborted

* Update dependencies in AIAssistant to include `updateMessage` in effect hook (lint warning fix)

* change onStop prop to be required in AssistantChatForm

* add comment for saving partial AI responses

* remove redundant text size class

* Make onStop prop optional in AssistantChatForm and conditionally render stop button

* Add loading spinner to stop button in AssistantChatForm

* simplify opacity handling for submit button

* Add tooltips

* Improve UI error handler for assistant panel

* Small tweak

* Filter out any tool calls that are streaming in generate-v4, happens when aborting the assistant while a tool is being called

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2025-08-08 17:47:46 +07:00
d60aceb562 Prompt and tool refactoring (#37500)
* try a really long context window to maximize caching

* update examples

* attempt to update packages and useChat

* update endpoints

* update zod

* zod

* update to v5

* message update

* Revert "zod"

This reverts commit ec39bac6b6.

* revert zod

* zod i

* fix complete endpoints

* remove async

* change to content

* type cleanup

* Revert the package bumps to rebuild them.

* Bump zod to 2.25.76 in all packages.

* Bump openai in all packages.

* Bump ai and ai-related packages.

* Remove unneeded files.

* Fix the rest of the migration stuff.

* Prettier fixes.

* add policy list tool

* refactor

* ai sdk 5 fixes

* refactor complete endpoint

* edge function prompt

* remove example

* slight prompt change

* Minor clean up

* More clean up

---------

Co-authored-by: Jordi Enric <jordi.err@gmail.com>
Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2025-08-08 15:25:57 +07:00
d52e2c252b Update AI SDK 5 (#37656)
* attempt to update packages and useChat

* update endpoints

* update zod

* zod

* update to v5

* message update

* Revert "zod"

This reverts commit ec39bac6b6.

* revert zod

* zod i

* fix complete endpoints

* remove async

* change to content

* type cleanup

* Revert the package bumps to rebuild them.

* Bump zod to 2.25.76 in all packages.

* Bump openai in all packages.

* Bump ai and ai-related packages.

* Remove unneeded files.

* Fix the rest of the migration stuff.

* Prettier fixes.

* zod

* v4 again

---------

Co-authored-by: Jordi Enric <jordi.err@gmail.com>
Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
2025-08-08 14:15:39 +07:00
Joshen LimandAlaister Young e70242f822 Chore/clean up assistant mcp feature flags (#36772)
* Clean up usage of newOrgAiOptIn and useBedrockAssistant feature flags

* Remove all OpenAI endpoints

* Fix for self-hosted

* Default isLimited to false

* Update PG meta tests

* Fix unit tests for model

* Revert pg meta tests

* Fix test

---------

Co-authored-by: Alaister Young <alaister@users.noreply.github.com>
2025-07-22 16:05:16 +08:00
Saxon Fletcher 8375880f9d remove dynamic chat name from prompt for caching (#37281) 2025-07-18 00:43:25 +00:00
Greg Richardson 2afd0c92ab fix(ai): temporarily remove log tools (#37274)
* fix(ai): temporarily remove log tools

* fix: temp remove log tools from tool filter logic

* fix: temp remove log tools from tests

* fix: remove references to get_logs in system prompt
2025-07-17 21:27:54 +00:00
Saxon FletcherandGreg Richardson 09c1c95d7e Upgrades pro users and above (#37183)
* feat: bedrock prompt caching

* fix: prompt cache point not sending to bedrock

* upgrade paid orgs to higher model

---------

Co-authored-by: Greg Richardson <greg.nmr@gmail.com>
2025-07-16 14:58:33 +10:00
Greg Richardson 283826cbd9 feat: bedrock prompt caching (#37140)
* feat: bedrock prompt caching

* fix: prompt cache point not sending to bedrock
2025-07-15 20:08:31 -05:00
Saxon Fletcher fd670667ab Context reduction (#37032)
* context saving

* fix model name

* add results comment

* fix errors
2025-07-11 16:03:37 +10:00
Saxon Fletcher e3125e59c2 Limit tools based on extensions (#37006)
* limit tools based on extensions

* if failed, fall back to disabled
2025-07-10 15:54:19 +10:00
Saxon Fletcher 0419d45b6f Add search docs, advisors, and custom logging tools (#36852)
* ui refinements

* rename chat

* copy

* prose message styles

* add icon back

* fix message save

* simplify empty state

* update suggestions

* pass through props

* button styles

* onboarding icons

* name button

* remove results

* current chat name

* use type

* add advisor and search

* fix down arrow

* logging tools

* more general filter tools

* fixes

* pageparam fix

* move search docs
2025-07-10 09:43:38 +10:00
Saxon Fletcher 216b235877 Review branch diff with AI (#36915) 2025-07-10 09:11:34 +10:00
Saxon Fletcher 94c45c2863 AI UI refine and renaming (#36819)
* ui refinements

* rename chat

* copy

* prose message styles

* add icon back

* fix message save

* simplify empty state

* update suggestions

* pass through props

* button styles

* onboarding icons

* name button

* remove results

* current chat name

* use type

* fix down arrow

* re-add chat name
2025-07-08 12:09:11 +10:00
Terry SuttonandIvan Vasilov 4d15119c29 opt in tags (#36885)
* Validate the aiOptInLevel on the server

* Fix another issue when the project and mismatched org.

* Fix a type error.

---------

Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
2025-07-04 14:17:42 -02:30
Greg Richardsonanddelgado3d ea91bf4739 Distributes bedrock requests evenly across regions (#36837)
* wip

* feat: distribute bedrock requests across regions

* fix: tests and type errors

* chore: remove AWS_BEDROCK_REGION from turbo.json

* fix: vercel oidc provider to fallback in credential chain

---------

Co-authored-by: delgado3d <27228526+delgado3d@users.noreply.github.com>
2025-07-02 17:35:37 +00:00
Alaister Young f1b67c3f24 chore: generate-v4 expose error message (#36702) 2025-06-26 20:07:12 +08:00
Joshen LimandSaxon Fletcher 256dbb7f38 Feature flag MCP changes (#36689)
* Feature flag assistant endpoint + opt in UI

* Feature flag bedrock stuff for the other endpoints like title, cron, complete

* add edge function complete v2

* revert to old complete

* Revert hardcode

* fix chart colour

---------

Co-authored-by: Saxon Fletcher <saxonafletcher@gmail.com>
2025-06-26 18:41:31 +08:00