dima/supabase
1
0 Fork 0
mirror of https://github.com/supabase/supabase.git synced 2026-10-08 19:05:06 +03:00
Code Issues Packages Projects Releases Wiki Activity
37,804 Commits 1,791 Branches 62 Tags
fix/api-keys-array-coerce
Commit Graph
87 Commits
Author SHA1 Message Date
Charis 21511042a3 feat(studio): assistant logs context and reports guard (#48514) 
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Feature — final PR (9/9) of the SQL editor logs-source stack.

**Base branch:** `charislam/sql-editor-inline-ai-clickhouse-dialect` (PR
8). Nothing here is user-visible: entry points stay behind
`sqlEditorLogsSource` + `otelLegacyLogs`, and flag rollout happens after
the whole stack merges.

## What is the current behavior?

- The Assistant has no idea a SQL editor snippet targets the logs
backend. Ask it about a logs snippet and it answers in Postgres, because
the attached query is fenced as ` ```sql ` and nothing tells the model
otherwise.
- Because the `sql` fence is what `MessageMarkdown` treats as runnable
Postgres, an attached ClickHouse query is rendered with a
Run-against-Postgres affordance and branded with `untrustedSql`.
- "Debug with Assistant" on a failed logs query produces a dialect-less
prompt, so both the in-app assistant and the copyable version get
debugged as Postgres.
- A report referencing a `log_sql` snippet runs its ClickHouse SQL
against the user's Postgres database and surfaces the resulting error.

## What is the new behavior?

**Assistant panel.** The "Current Query" chip records which backend the
attached query targets. That reaches the model two ways: each attachment
is fenced with its own dialect (` ```clickhouse ` vs ` ```sql `), and a
`containsLogsSnippets` flag rides on the user message as AI SDK
`metadata`. The server reads the flag off the conversation and prepends
the ClickHouse dialect rules plus the logs schema reference as a
non-cached context message.

Two design points worth calling out in review:

- The flag lives on the **message**, not the request body, so Retry and
the tool-approval continuation reproduce the context a message was
originally asked in — neither of those passes a per-call body.
- It's derived from **what's actually attached**, so detaching the chip
drops the claim rather than leaving the two able to disagree.

The `clickhouse` fence also keeps a logs query out of
`MessageMarkdown`'s `sql` branch, so it's no longer offered as runnable
Postgres or branded with `untrustedSql` — a boundary this stack's
distinct brands exist to prevent crossing.

**Debug flow.** `buildDebugChatArgs` attaches its query with a source
for the same reason, and names the dialect in the prompt text so the
copyable version stands on its own outside the app.

**Reports.** A report only stores a snippet id, so whether it queries
the logs backend is only knowable once the content loads. `ReportBlock`
guards on the fetched type and renders a `LogsSnippetReportBlock`
placeholder instead of executing. Double-guarded: no `sql` for a logs
snippet (so it's out of the query key and `queryFn` short-circuits even
on an explicit `refetch`) and `enabled` excludes it.

**Incidental cleanups.** `buildAssistantContextMessages` extracted out
of `generate-assistant-response`; a schema-access sentinel that was
duplicated as a string literal across two files (and compared against)
replaced with one exported constant; `SqlSnippet` deduplicated to a
single declaration; `resolveSnippetSource` / `isLogsSource` shared
instead of re-implemented per surface.

**Tests.** 4 new/extended suites. Notable cases pinned: a message with
no metadata must validate (`safeValidateUIMessages` applies
`metadataSchema` to *every* message, so a required schema would 400
every existing conversation); only *user* messages count, so a model
reply can't talk the server into a different dialect; a mixed-attachment
message is flagged without overclaiming a single source; and
`ReportBlock` registers no pg-meta mock for the logs cases, so an
unhandled request failing the test *is* the assertion that logs SQL
never reaches Postgres.

Verified: `pnpm typecheck`, `lint:ratchet` (no regression), Prettier,
and the full Studio suite (459 files / 4969 tests).

## Additional context

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Added support for recognizing log snippets in reports, with clear
guidance to open them in the SQL editor or remove them.
- AI Assistant now understands log snippets and provides
ClickHouse-specific context, formatting, and troubleshooting guidance.
- Snippets retain their source information when shared with the AI
Assistant.

- **Bug Fixes**
- Prevented unsupported log snippets from being executed as regular
database queries.
  - Improved source detection when opening snippets directly from links.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-04 09:02:40 -04:00
Charis 8b38e0d1ed feat(studio): ClickHouse dialect for logs snippet AI + rewrite to ClickHouse (#48501) 
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Feature, plus a refactor of the shared logs-rewrite flow.

PR 8 of the SQL editor query-source series. Stacked on #48457 — review
that one first, and merge this after it.

## What is the current behavior?

A `log_sql` snippet runs against the ClickHouse-backed analytics
endpoint, but the SQL editor's AI still writes Postgres: inline edits
get Postgres system prompts, and the result is run through
`sql-formatter`, which mangles ClickHouse backticks and `log_attributes`
map lookups.

Legacy Logs Explorer saved queries open in the editor as `log_sql`
snippets. Those are BigQuery dialect and error against the ClickHouse
endpoint the editor runs them on, with no in-editor way out — only the
Logs Explorer offered a rewrite.

The completion route was also asymmetric. It assembled a
schema/code/instruction message for Postgres but forwarded `prompt`
verbatim for ClickHouse, so a client wanting ClickHouse had to
hand-build the equivalent string.

## What is the new behavior?

**Inline AI speaks ClickHouse for logs snippets.** `sqlSourceToDialect`
maps a snippet's source to `postgres`/`clickhouse` and
`buildCompletionRequestBody` threads it through. For ClickHouse,
`useSqlEditorAi` strips code fences from the response and skips
`formatSql`. Execution and dialect both follow the snippet type, so a
snippet's valid dialect never flips.

**Rewrite to ClickHouse in the editor.** A banner offers the rewrite for
a logs snippet whose text trips `looksLikeLegacyLogsQuery`, and proposes
the result through the editor's existing AI diff view rather than
replacing the snippet, so it's accepted or discarded like any other AI
edit. Gated on `otelLegacyLogs`: on a non-migrated org the BigQuery text
is still correct, so rewriting it would break a working query.

The offer is a state machine (`offered` / `rewriting` / `failed` /
`noRewriteNeeded` / `dismissed`) with a declarative table of valid
transitions, so the states are mutually exclusive by construction and
dismissal is terminal. A failure keeps its message and offers a retry; a
response identical to the input is reported rather than opening an empty
diff.

**One place assembles completion prompts.** The route now uses a single
template for both dialects, branching only the schema section and — for
`intent: 'rewrite'` — the instruction. `lib/ai/clickhouse-logs.ts` is
the single home for ClickHouse-logs prompt content, replacing two
independently maintained descriptions of the same table. Clients carry
no prompt text.

**The rewrite flow is shared with the Logs Explorer.** Both surfaces
previously hand-rolled the same sequence and had drifted: only one
detected a no-op rewrite, they sourced `log_attributes` keys
differently, and the Explorer formatted errors with an `as Error` cast.
Both now use `useLegacyLogsRewrite` and the same state-driven banner, so
the Explorer picks up no-op detection and typed error extraction.

**Attribute keys are fetched on submit, not while typing.** The detected
source would otherwise feed a reactive query key, making every edit that
changed it cost another network call. `useLogsAttributeKeys` is
imperative and goes through `queryClient.fetchQuery`, so a source
already cached — including by the Explorer header and query panel, which
subscribe reactively — is reused. This also closes a gap where inline
edits never received keys at all, unlike full rewrites.

`getErrorMessage` gains an optional typed fallback and no longer
stringifies a bare object into `'[object Object]'`; every existing
caller already hand-rolled a fallback, except `QueueSettings`, which
interpolated the raw result and now passes one.

Nothing here is user-visible until the `sqlEditorLogsSource` flag is
enabled.

Tests: dialect selection and request-body shape, the ClickHouse prompt
content (including that the schema section does not restate the dialect
rules), the reducer's valid and invalid transitions,
`shouldOfferLegacyLogsRewrite`, on-submit key discovery with cache
reuse, and `getErrorMessage`.

## Additional context

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added an Assistant banner to help rewrite legacy BigQuery-style logs
queries into ClickHouse SQL.
* SQL assistance now adapts to the selected query type, including
relevant log attribute context.
* Rewrite suggestions can be reviewed as editor diffs before being
applied.

* **Bug Fixes**
* Improved rewrite failure handling, retry options, dismissal behavior,
and “no rewrite needed” messaging.
* Error notifications now provide a clearer fallback message when
details are unavailable.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-04 09:02:40 -04:00
Joshen Lim 2c53ca4a79 Support listing and reading custom reports from Assistant (#48530) 
## Context

This is pre-requisite work for adding support to managing custom reports
from the Assistant. Planning to break this into a number of PRs, briefly
- Adding read support for custom reports
- Adding write support for custom reports
- Adding run support for custom reports
  - Should be able to infer data from the results then

This PR starts with adding support for listing and reading custom
reports from the Assistant

## Other changes involved
- Updates setting up of the home page report to have better title and
description
- Swaps the variant of the ToggleGroup in the SQL block for custom
reports as the default variant blends into the background color of the
PopoverContent

## To test
- [ ] Assistant should be able to list custom reports + read its
contents
<img width="428" height="755" alt="image"
src="https://github.com/user-attachments/assets/6a15b660-c0ee-4a06-984c-87eff3943eec"
/>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added AI-assisted tools to list reports and retrieve report details,
including chart counts, layouts, configurations, and SQL-backed chart
information.
  - Added clearer empty-state messaging when no snippets are available.

- **Improvements**
- New homepage reports now use the name “Homepage Report” and include a
descriptive project-home summary.
  - Updated query controls with refreshed visual styling.
  - Improved content requests to support additional request context.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-03 13:59:50 +07:00
Joshen Lim 4901f081e5 Migrate remaining requests to pg-meta API to use query endpoint (#47758) 
## Context

Migrates the remaining API requests to the pg-meta endpoint to use the
query endpoint directly with the SQL from the pg-meta package. This
touches the following:
- policies
- publications
- triggers
- views
- materialized views
- types

## To test
Just need to verify that we're still fetching the data correctly on
these pages
- Database policies
- Database publications
- Database triggers
- Database tables (views + materialized views)
- Database types

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved and stabilized loading of database metadata (views, triggers,
RLS policies, publications, materialized views, and enum types),
including more reliable schema-scoped filtering.
* Updated policy loading behavior and related UI queries to consistently
use schema arrays, improving cache correctness and consistency.
* **Tests**
* Updated end-to-end test synchronization to wait for the correct
metadata responses using more specific request identifiers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-09 17:09:03 +08:00
Carel de WaalandAlaister Young 0acc0eb8b3 feat: Support Form - Sync AI assistant conversation to Front (#46778) 
# Sync AI assistant conversation to Front

## What & why

When a user submits a support ticket, an AI assistant chat opens so they
get help
immediately while waiting for a human agent. This PR mirrors every turn
of that chat into
the Front conversation the support form already created, so the support
team sees the full
context and Front automations (routing, emails, CSAT) can act on it.

Studio holds no Front credentials — it calls the platform endpoints (see
the platform PR)
to do the syncing. The assistant card is gated behind the
`supportAssistantFollowUp`
ConfigCat flag.

## How it works

1. **Submit** — `SupportFormV3` generates a stable `threadRef` (via the
`uuid` package —
`crypto.randomUUID()` is `undefined` in insecure contexts like
non-localhost HTTP and
would throw, silently aborting the submit) and sends it on
`/platform/feedback/send`.
The response returns the Front `conversationId`. Both are stored on
`SubmittedSupportRequest`.
2. **Open chat** — `SupportAssistantSuccessCardContent` opens a chat
seeded with
`supportMetadata` (`threadRef`, `frontConversationId`, subject,
category, severity, …).
   The first message is a `<support>…</support>` XML block.
3. **First user message** — the chat is tagged `isSupportChat = true`;
the `onFinish` hook
   fires `syncSupportChatToFront`.
4. **Subsequent turns** — each `onFinish` slices the unsynced delta,
strips the XML
metadata block from the seed message, and posts to the platform messages
endpoint.
5. **Escalation / resolve** — the `escalate_to_human` /
`resolve_support_conversation` tools
(and manual **Escalate**/**Resolve** buttons in the assistant input)
flip lifecycle status
via `setSupportLifecycleStatus` → `syncSupportLifecycleToFront`, which
calls the
escalation/resolve endpoints. Front rules act on `ai_support_status`.
The assistant only
   resolves after the user explicitly confirms the issue is fixed.

## Key design decisions

- **`threadRef` as the shared key** — one UUID travels as `threadRef` on
submit and as
`chatId` on every sync, so all messages thread into a single Front
conversation.
- **`conversationId` from the form response** — passed to all
sync/lifecycle calls so the
  platform skips lazy derivation and PATCHes custom fields directly.
- **Delta-only sync** — `lastSyncedMessageCount` tracks what's been
sent; the boundary is
snapshotted before the async call to avoid skipping messages that arrive
mid-flight.
- **Server-side de-dup** — stable `external_id` (`chatId:msg.id`) means
retries don't
  duplicate in Front.
- **Fire-and-forget** — sync failures log to Sentry, never break the
chat; `isSyncing`
resets on rehydration so the next `onFinish` retries the same delta.
Message and lifecycle
syncs use separate guards (`isSyncing` / `isLifecycleSyncing`) so an
in-flight message
  sync can't drop an escalate/resolve.
- **Lifecycle queued until the conversation exists** — if a lifecycle
transition is requested
before the initial message sync has returned a `frontConversationId`,
it's stored as
`pendingLifecycleStatus` and flushed once the id is assigned, rather
than dropped.
- **Tools return immediately** — the lifecycle tools return a stub to
the AI SDK; the real
Front call happens in `onFinish`, keeping async I/O out of the tool
execute path.
- **XML seed stripped before sync** — only the user's actual `<message>`
is sent to Front
  (or dropped entirely if the form already created the conversation).

## Changes

| Area | File(s) |
| --- | --- |
| Support form state | `SupportForm.state.ts` — `threadRef` /
`frontConversationId` on `SubmittedSupportRequest` |
| Support form submit | `support-ticket-send.ts` — sends `threadRef`,
reads `conversationId` |
| Support form UI | `SupportFormV3.tsx` — generates `threadRef`, stores
`conversationId` |
| AI assistant state | `ai-assistant-state.tsx` — `SupportChatMetadata`,
`setSupportLifecycleStatus`, `onFinish` wiring, tool handling |
| Message sync | `state/ai-chat-front-sync.ts` — delta tracking, message
filtering, initial vs. incremental |
| API data layer | `data/feedback/ai-chat-front-sync.ts` — typed
platform-client wrappers for the three conversation endpoints |
| Support tools | `lib/ai/tools/support-tools.ts` — `escalate_to_human`,
`resolve_support_conversation` |
| Tool integration | `lib/ai/tool-filter.ts`, `tools/index.ts`,
`generate-assistant-response.ts` |
| Success card | `SupportAssistantSuccessCardContent.tsx` — tags chat on
first engagement |
| Assistant panel UI | `AIAssistant.tsx` — Escalate/Resolve buttons,
disabled input on closed chats, support placeholders |



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Summary by CodeRabbit

- **New Features**
  - Support chats now include “Escalate to human” and “Resolve” actions.
- Support submissions can be associated with a stable Front thread via a
generated `threadRef`, preserving linkage across follow-ups.
- AI assistant responses and input hints adapt when support mode is
active.

- **Bug Fixes**
- Improved support chat state management and lifecycle handling to keep
conversation metadata and message history synchronized more reliably
with Front.

- **Chores**
- Added/updated coverage to reflect the new support-chat state and
syncing behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com>
 2026-07-08 12:41:53 +02:00
Pedro RodriguesandClaude Opus 4.8 c4c213ce3d feat(studio): switch dashboard assistant to remote MCP server (#47479) 
## I have read the
[CONTRIBUTING.md](<https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md>)
file.

YES

## What kind of change does this PR introduce?

Feature / refactor.

## What is the current behavior?

The dashboard assistant runs `@supabase/mcp-server-supabase` in-process
over an in-memory transport (`lib/ai/supabase-mcp.ts`).

## What is the new behavior?

The assistant connects to the **remote MCP server** over HTTP
(`@ai-sdk/mcp`), forwarding the dashboard session token as a bearer. URL
comes from `NEXT_PUBLIC_MCP_URL` with a local-dev fallback;
platform-only, and Nimbus works via the same env var.

* **Tool model unchanged:** UI-controlled `execute_sql` (with
`needsApproval`) and `deploy_edge_function` still come from Studio; the
allowlist (`TOOL_CATEGORY_MAP`) remains the gate keeping the remote's
write tools away from the assistant (`read_only` is defense-in-depth).
* **Attribution:** sends `x-source-name: supabase-studio` (+
`x-source-version`) → logged as `source_name`/`client_name`.
* **Connection lifecycle:** the HTTP client is closed via the request's
`AbortSignal` (tools execute later during streaming); `signal` is
required on `getTools`/`getMcpTools`.
* **Resilience:** a remote-MCP failure degrades to the remaining tools
instead of failing the assistant.
* **Drift protection:** relied-upon tools are typed against `keyof
typeof supabaseMcpToolSchemas`, so a package bump that renames/removes
one fails `pnpm typecheck`; a runtime check also warns if the deployed
server returns fewer tools.
* Adds unit tests for the above.

## Additional context

* Verified end-to-end against a local remote MCP server with a dashboard
token: `initialize` 200, tools listed, a tool executed, client closed
cleanly.
* The remote MCP (mgmt-api) already accepts dashboard session tokens
(GoTrue-JWT auth path) — no backend change needed. `NEXT_PUBLIC_MCP_URL`
must point at each env's `/mcp`.
* `@supabase/mcp-server-supabase` is kept — still used by the
self-hosted `/api/mcp` routes.

Closes
[AI-137](https://linear.app/supabase/issue/AI-137/switch-dashboard-assistant-to-remote-mcp)

## Rollout

* **Rollout:** merges with `USE_REMOTE_MCP` off (in-process); flip it to
`true` per environment (staging → prod → Nimbus) once each one's
prerequisites land.
* **Rollback:** unset `USE_REMOTE_MCP` and redeploy to fall back to the
in-process client — no revert needed.

## Summary by CodeRabbit

* **Bug Fixes**
* Improved AI request handling so tool loading and generation clean up
properly when a request is cancelled or the browser connection closes.
* Added safer fallback behavior when remote tool loading fails, so AI
features can continue with available tools instead of stopping entirely.
* Updated remote tool access to use the current project reference and
preserve the correct access headers.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* AI tools now connect more reliably to remote services and stop cleanly
when requests end or are canceled.
* Tool loading is more resilient, continuing with available tools if
remote access is unavailable.

* **Bug Fixes**
* Improved cleanup to prevent lingering connections during SQL
generation and policy workflows.
  * Added safer handling for remote tool changes and invalid responses.

* **Tests**
* Expanded automated coverage for remote tool setup, cancellation, and
fallback behavior.


<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-07 19:38:21 +01:00
Gildas Garcia 0e3364bbad Chore/cleanup studio deps (#47399) 
## Problem

Knip reported some unused dependencies. Some are actually used in
builds, etc but others are not.

## Solution

Remove the really unused dependencies

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Cleaned up unused dependencies and removed some obsolete test/support
files.
* Updated project ignore rules to better match current app structure and
generated files.
* **Bug Fixes**
* No user-facing behavior changed; this release is focused on
maintenance and cleanup.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-30 14:03:40 +02:00
Jordi EnricClaude Opus 4.8Joshen Lim
 		2aa1b52234 feat(studio): add feature to rewrite queries DEBUG-145 (#47266) 
## Problem

Moving the Logs Explorer to ClickHouse means users' saved BigQuery
queries no longer run.
<img width="2430" height="1010" alt="CleanShot 2026-06-29 at 11 36
04@2x"
src="https://github.com/user-attachments/assets/ae0ab155-7d3d-4ae9-81c3-22bf3a88cf8c"
/>

## Fix

Rewrite the query with AI instead of a SQL transpiler. AI handles the
long tail of nested fields and dialect differences far better than a
rule-based rewriter, and it needs no extra runtime dependency.

- `rewriteLogsSqlWithAI` posts the current query to
`/api/ai/code/complete` with `dialect: 'clickhouse'`. The endpoint skips
the Postgres schema and best-practices for that dialect and uses
logs-specific instructions and model so the output is ClickHouse logs
SQL (FROM `logs` + `source` filter, no `unnest` joins, nested fields
read from `log_attributes['...']`).
- The query's `source` is detected and its real `log_attributes` keys
are fetched and passed to the model, so it maps to exact paths instead
of guessing.
- The rewrite runs in the background and is proposed as a side-by-side
accept/discard diff in the editor. The AI Assistant panel is not opened.
- Entry points: a banner shown only for legacy-looking queries
(dismissal persisted), and a "Fix Query" button next to Field Reference.
- The Field Reference drawers discover `log_attributes` keys from real
data so the listed fields match what the source actually emits.

## Dependencies

Built on top of #47265 (Logs Explorer -> OTEL endpoint) — that is the
base branch of this PR. Merge #47265 first. Behind `otelLegacyLogs` (off
by default).

Part of DEBUG-145 (split from #47087).

## How to test

- Open the Logs Explorer with a BigQuery logs query (the templates have
some), click "Fix Query", and confirm the diff shows valid ClickHouse
SQL. Accept it and confirm the applied query runs.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an OTEL legacy logs workflow (behind a feature flag) with an
interactive banner and a “Fix Query” ClickHouse rewrite action,
including an accept/discard diff review overlay.
* Introduced OTEL-aware field reference rendering with dynamic discovery
of `log_attributes` keys and updated OTEL source insertion behavior.
* Enabled dialect-aware SQL completion for ClickHouse logs, using
logs-specific instructions and output constraints.
* **Bug Fixes**
* Improved rewrite flow validation and handling, including log source
detection and cleanup of AI-generated SQL formatting.
* **Tests**
* Added Vitest coverage for rewrite prompt generation,
detection/classification utilities, SQL fence stripping, OTEL field
mapping, and OTEL log attribute key discovery.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
 2026-06-29 14:31:18 +02:00
Alaister YoungAlaister YoungIvan Vasilov
 		9eab4f8fbf build(studio): Vite/TanStack-Start build pipeline behind flag (stack 1/6, from #46424) (#47107) 
**Stack 1/6** of the TanStack Start migration (#46424), split into
reviewable, independently-mergeable PRs.

> [!IMPORTANT]
> **Next stays the default and only active framework after this PR.**
This wires up the Vite/TanStack-Start build pipeline behind the
`STUDIO_FRAMEWORK` flag, but there are no TanStack routes yet — so the
TanStack build isn't functional or tested until later PRs in the stack.
Nothing about the Next build, dev, or deploy changes behaviourally here.

## What's in this PR
- **Dispatch:** `dev`/`build`/`start` now go through
`scripts/dispatch.js`, which runs the Next variant unless
`STUDIO_FRAMEWORK=tanstack`. The original commands are preserved as
`dev:next`/`build:next`/`start:next`.
- **Build pipeline:** `vite.config.ts`, `serve.js`, `smoke-server.mjs`,
vite/tanstack deps, `turbo.jsonc`.
- **`tsconfig.json`:** `jsx: react-jsx`, `moduleResolution: Bundler`,
`target: ES2022`. Because `include` is `**/*.ts(x)`, this re-typechecks
the whole app, so the companion adaptations below land with it.
- **Shared adaptations (companions to the tsconfig change):**
`BufferSource` casts, `packages/ui` unused-`React` import removals, etc.
- **Routing/middleware plumbing:** `next.config.ts` +
`redirects.shared.ts` (redirect rules now shared with `vercel.ts`),
`proxy.ts`/`start.ts` middleware + `hosted-api-allowlist.ts`.

## Verification
Run locally off `master`: frozen install ✓, `studio` typecheck ✓, **Next
build ✓** (compiles + generates all routes), lint ratchet ✓ ("some rules
improved"), prettier ✓.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a hosted API endpoint allowlist to return 404 for non-supported
`/api/*` routes.
* Introduced a TanStack route-migration checklist and expanded TanStack
Start routing support.
* **Improvements**
* Enhanced deployment refresh/detection by tightening cookie handling
for “latest deployment” updates.
* Centralized redirect/maintenance-mode rules for consistent platform vs
self-hosted behavior.
* Improved production serving with a dedicated static + proxy server and
a post-build smoke test.
* **Dependencies**
* Updated TanStack-related packages and React Table/query tooling
versions.
* **Documentation / Chores**
* Updated formatting and tooling config; added shared build environment
parsing utilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com>
Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
 2026-06-24 17:55:22 +08:00
Matt Rossman c54168842b fix(assistant): prevent parallel approval-required tool calls (#47008) 
Addresses the issue of parallel tool approvals freezing Assistant and
causing UX ambiguity around what happens if only part of multiple
dependent invocations is approved.

Two layers at which this is addressed:
1. Tightens prompt to clarify approval tools must be issued one per
step, not in parallel.
2. In case something slips past the prompt, this also auto-denies all
but the first `approval-required` tool call when the model issues
multiple in the same step, so the model is forced to reissue them
sequentially.


**Demo**

The following chats demo me explicitly asking Assistant to run those
approval-gated tools in parallel, and the Assistant correctly invokes
them sequentially instead.

| Parallel `execute_sql` request | Parallel `deploy_edge_function`
request |
|--------|--------|
| <img width="1820" height="4240" alt="CleanShot 2026-06-16 at 16 20
23@2x"
src="https://github.com/user-attachments/assets/3ddad61d-24d6-4e5b-8572-c261755f3a03"
/> | <img width="1738" height="2726" alt="CleanShot 2026-06-16 at 16 22
59@2x"
src="https://github.com/user-attachments/assets/77a0aa20-e3cb-4061-b270-ab2a96d1e64a"
/> |


As shown in [this
trace](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=project_logs&object_id=5a8d02e5-b3b6-40cc-ba76-ecee286478f4&r=a3a37857-95df-4a95-a5a8-818ea305b2a5&s=a3a37857-95df-4a95-a5a8-818ea305b2a5),
parallel tool calls are still allowed for context gathering tools that
don't require approval:

<img width="1051" height="517" alt="CleanShot 2026-06-16 at 16 25 42@2x"
src="https://github.com/user-attachments/assets/00d208b1-6131-4b8e-910e-e92fd2a79a5b"
/>


Closes AI-803

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Improvements**
* Approval-required operations are now processed sequentially rather
than in parallel within the AI Assistant.

* **Tests**
* Added comprehensive test coverage for the parallel approval prevention
logic.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-16 17:20:06 -04:00
Joshen Lim 1baaded0bb Consolidate execute-sql-query into execute-sql-mutation (#46944) 
## Context

Just some clean up as I was going through stuff
- `useExecuteSqlQuery` is deprecated and not used at all
- As such `execute-sql-query` is technically irrelevant, the more
relevant file is `execute-sql-mutation`
- Hence opting to consolidate `execute-sql-query` into
`execute-sql-mutation`
- Also removing `ExecuteSqlError` since its just re-exporting the
`ResponseError` type

There's a lot of file changes but its essentially just updating the
importing statements across the files
 2026-06-16 00:07:16 +08:00
Wen Bo Xie f041ec4124 feat: update API keys instruction for AI Assistant (#45401) 
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Updated authentication docs: require startup-time validation of
required env vars and introduce JSON-mapped publishable/secret key
handling.
* Added JWKS-based asymmetric JWT verification guidance and clarified
symmetric (HS256) handling; deprecated anon/service-role keys.
* Clarified Authorization control flow: when handlers must fully
authenticate vs. when module-scoped JWKS verification applies.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-03 16:44:11 -06:00
Matt Rossman 65fab30935 feat(ai): judge tool inputs, add storage guidance and permissive RLS evals (#46168) 
Adding broad RLS policies to public buckets can cause users to expose
more than they expected, like the ability to list all profile pictures
on an app. This patches Assistant with knowledge to follow our latest
guidance on restrictive RLS policies for storage buckets
https://github.com/supabase/supabase/pull/46172

**Changes**
- Adds Storage bucket evals for public website assets and avatar access
patterns to distinguish public vs private bucket use cases
- Adds eval for overly permissive table policies
- Adds `storage` knowledge so Assistant distinguishes public buckets,
private buckets, object reads, and object listing.
- Adds `includeToolCallInputs` option for scorer transcripts so LLM
judges can evaluate proposed SQL/tool actions.
- Bumps max step count to 10 since storage knowledge may incur another
tool call (also 10 is recommended
[here](https://vercel.com/academy/ai-sdk/multi-step-and-generative-ui#why-multi-step-is-required)
for complex multi-tool scenarios)

**References**
-
https://supabase.com/docs/guides/storage/buckets/fundamentals#public-buckets
- https://supabase.com/docs/guides/storage/security/access-control
- https://github.com/supabase/supabase/pull/46172

**Notes:**
- These prompt tweaks are not meant to be exhaustive fixes, they are
mainly hotfixes intended to hold us out until these cases can be
addressed more deeply in skills/docs and tracked in a central evals

Closes AI-676
Closes AI-756

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Storage knowledge resource for the assistant covering Supabase
Storage access patterns and RLS guidance.
* Added three evaluation cases: two for Storage (marketing assets,
avatars) and one for RLS policy generation for user profiles.

* **Improvements**
  * Evaluators now include tool call inputs when judging conversations.
* Assistant prompts and generation enhanced with richer Storage/RLS
guidance and extended streaming limits.

* **Tests**
* Added test ensuring tool call inputs are included in serialized thread
context.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46168?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-29 09:55:23 -04:00
Saxon Fletcher 80da153450 Fix for AI Assistant query and deploy confirmation (#46052) 
When Assistant requests confirmation to run a query or deploy an edge
function if the user doesn't skip or run and instead sends a follow-up
message it errors out. This allows follow-up messages and treats them as
"skips" which means adjusting confirmation message state as part of the
follow-up. This also uses toModelOutput to cleanse data based on
permissions.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **New Features**
* Enhanced tool approval workflow: pending approvals are now
automatically resolved as denied when submitting new messages
* Improved chat input state management with better handling of approval
states
  * Customizable loading messages for tool operations

* **Bug Fixes**
  * Fixed chat input availability during pending tool approval states
  * Improved tool execution feedback during approval workflows

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46052?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-20 09:09:28 +10:00
Matt Rossman 212ccf8135 fix(ai): contextualize cron schedule as SQL writes, score in "Tool Usage" (#45997) 
When Assistant tries to schedule crons in read-only mode, it succeeds
but creates the jobs under the `supabase_read_only_user`. This causes
permission errors when user try to delete or unschedule them from the
Cron dashboard. The root fix will be to enforce read-only transactions
for that user. In the meantime, this PR steers Assistant to avoid the
mistake.

**Changes**

- Prompts `execute_sql` to treat side-effecting function calls such as
`cron.schedule()` as write queries.
- Adds tool input assertions for "Tool Usage" scorer and a focused cron
regression eval.
- Updates eval mocks to show pg_cron extension as installed so it can
call `cron.schedule()`

**Verification**

See [this
trace](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=4a9e8c0e-83b7-4555-8502-365662c3ec8e&r=e041e69b-b70f-41d1-b88c-e8f7888c3de5&s=e041e69b-b70f-41d1-b88c-e8f7888c3de5)
from Braintrust where the new eval passes "Tool Usage", correctly using
`isWriteQuery` for the `cron.schedule()`

Closes AI-737


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Tool evaluation now validates tool inputs (including exact and
substring matches) in addition to tool presence.

* **Tests**
* Added a test confirming cron-scheduling behavior and that SQL
scheduling/enqueue calls are treated as write operations.

* **Chores**
  * Added pg_cron to mock extension data.
* Clarified description that SQL calls with side effects should be
treated as writes.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/45997?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-18 12:11:11 -04:00
Charis 12869fcd9f studio: flip executeSql signature to SafeSqlFragment (7/7) (#46007) 
## Summary

Final PR in the SafeSql migration stack. Stacked on top of #46006.

Tightens `executeSql`'s `sql` parameter from `string` to
`SafeSqlFragment`. Any future raw-string caller is now a compile error —
the SafeSql safety property becomes structural rather than
convention-based.

Also adapts the AI `execute_sql` tool to promote AI-generated SQL via
`acceptUntrustedSql(untrustedSql(sql))` inside the `execute` callback.
The tool's existing \`needsApproval: true\` gate ensures `execute` only
runs after the user has explicitly approved — that approval is the
gesture that promotes untrusted to safe.

## Test plan

- [x] `pnpm typecheck` passes
- [x] Grep for any remaining raw-string `executeSql` calls in
`apps/studio` returns nothing
- [x] Dev-server smoke: AI tool approval flow executes SQL
 2026-05-15 20:07:54 +00:00
Charis 39ec777d22 studio: SafeSql stragglers + remaining tests (6/7) (#46006) 
## Summary

Part 6 of 7 in the SafeSql migration stack. Picks up the few remaining
files that didn't fit cleanly into earlier batches:

- `components/Docs/Description.tsx` — comment statements built via
`safeSql`.
-
`components/Integrations/IntegrationOverviewTabV2/InstallIntegrationSheet.test.tsx`
— test fixture updated to `SafeSqlFragment`.
- `lib/ai/tools/studio-tools.test.ts` — test fixture updated.
- `lib/api/generate-v4.test.ts` — test fixture updated.

Sets up PR 7, which flips the `executeSql` signature itself.

## Test plan

- [x] `pnpm typecheck` passes
- [x] Specific Studio unit tests run on top of the stack
(`Policies.utils.test.ts`, `SidePanelEditor.utils.createTable.test.ts`,
`useQueryInsightsIssues.utils.test.ts`)
- [x] Dev-server smoke: Docs panel renders / accepts edits

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
  * Updated test suites to reflect internal type definition changes.

* **Refactor**
* Internal code improvements to enhance type safety and consistency
across the codebase.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46006)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-15 15:56:57 -04:00
Matt Rossman d143571586 feat(assistant): trace-level scorers + server-side tool execution with needsApproval (#45654) 
## Motivation

When Assistant runs a potentially destructive tool like `execute_sql`,
it stops the LLM request and prompts for client-side approval and
execution of the tool. After approval, a second request kicks off under
a separate trace. This has made scoring and
[Topics](https://www.braintrust.dev/blog/topics) classification
challenging, as the generated `output` is split across stateless
requests. The [span-level
scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans)
approach we've used thusfar (after the LLM call, we massage the result
into an `output` payload that's stuck onto the root span) has been
cumbersome and led to invalid scores / topics where only part of the
assistant response is considered. It's also inefficient, as we're
duplicating potentially large info (like the `search_docs` output) that
already exists within the trace.

An alternative to scoring spans is to [score
traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces).
Braintrust [best
practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices)
advise:

> Use span scope for evaluating individual operations or outputs. Use
trace scope for evaluating multi-turn conversations, overall workflow
completion, or when your scorer needs access to the full execution
context.

We've also received [direct
guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89)
from their team to use this approach.

## Changes

Migrates eval scorers from custom `AssistantEvalOutput` shape to
trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with
thread parsing that scores the full latest Assistant turn and passes
prior conversation separately where relevant.

Moves `execute_sql` and `deploy_edge_function` from client-side
execution after approval to AI SDK `needsApproval` + server-side
`execute()`. SQL results returned to the model are gated by AI opt-in
level, so row data is only included with `schema_and_log_and_data`;
otherwise the tool returns the no-data-permissions sentinel.

Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within
an "assistant" turn due to tool call requests/responses. For online
evals, this means we should configure automations to only score traces
with `metadata.isFinalStep = true` to ensure we're judging the complete
generated response.

Other minor kaizen changes:
- Renamed `promptProviderOptions` to `systemProviderOptions` to clarify
that this is associated with the "system" message and disambiguate from
the root `providerOptions`
- Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown`
span shapes from Braintrust, to more easily access typed inputs/output
on tool spans.
- Bumps AI SDK floor version `^6.0.116` → `^6.0.174`
- Tweaked the "Conciseness" scorer to not unfairly dock points for the
new `[called tool_name]` labels in serialized assistant response

## Verification

In the studio staging build, I asked Assistant to create a todos table
with 3 sample todos. I manually approved the `execute_sql` call and saw
Assistant generate text before & after the call.

In Braintrust I verified two traces were produced (see [filtered
logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})),
the first with `metadata.isFinalStep = false` and the second with
`metadata.isFinalStep = true`.

In the Braintrust staging scorers, I ran the preview Completeness scorer
on the second trace and verified it sees the complete Assistant response
including markers for tool calls ([link to
trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82))

<img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x"
src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a"
/>

I also tested the `deploy_edge_function` workflow and verified it still
prompts for permission and warns on deployment of existing functions.

**References**
- https://www.braintrust.dev/docs/evaluate/custom-code#score-traces
-
https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval

Supercedes https://github.com/supabase/supabase/pull/45556 and
https://github.com/supabase/supabase/pull/45339

Closes AI-473

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Tool actions (SQL execution, edge-function deploy) now require
explicit user Approve/Deny before proceeding.

* **Improvements**
* Assistant pauses for approval responses before sending follow-ups,
giving clearer control over risky actions.
  * Deploy/replace flows show confirmation and clearer replace warnings.
* Evaluation/scoring updated to use richer trace data for more accurate
assistant performance signals.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-12 15:24:21 -04:00
Greg Richardson 538f9e3e82 fix: prevent AI assistant from soliciting sensitive creds (#45692) 
Adds prompt guardrails and evals to prevent the AI assistant from asking
users to share sensitive data (API keys, `.env` contents, etc.) and to
warn when credentials are shared.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Stronger safety behavior: assistant now refuses requests to share full
environment files, asks for variable names only, and directs users to
secure secret-management tooling.
* Immediate warning and guidance if credentials or other sensitive
values are pasted in chat, without repeating exposed secrets.
* **Behavior**
* Clarified evaluation rules so responses more consistently follow the
new safety guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-07 13:22:19 -06:00
Greg Richardson 5f8906a20e fix: add destructive operation guardrails to AI assistant (#45194) 
Prevents the AI assistant from helping with local git/filesystem
operations, and adds explicit warnings before irreversible database
operations (DROP TABLE, DELETE without WHERE, etc.).

Adds a `safetyScorer` and eval cases to cover these behaviours.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a Safety metric to evaluations so assistant responses are scored
for safe handling of destructive or risky requests
* Assistant guidance updated to refuse destructive local VCS/filesystem
actions and require clear warnings for irreversible database operations

* **Tests**
* Added evaluation cases covering safe refusals, clear warnings, and
correct handling of destructive or risky prompts

* **Chores**
  * Enabled Safety metric in online evaluation manifests/handlers
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-06 09:24:21 -06:00
Matt Rossman 94d7c0d463 chore(studio): remove @supabase/mcp-utils dependency (#45438) 
**Changes**

Replaces our custom `StreamTransport` with
[InMemoryTransport](https://github.com/modelcontextprotocol/typescript-sdk/blob/4fbcfcd176b6b189970263c4625eb6e60db043d2/packages/core/src/util/inMemory.ts#)
from the official MCP SDK, removing the need for the
`@supabase/mcp-utils` dependency.

**Verification steps**

I verified Studio's AI Assistant still works as expected.

Closes AI-694

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
  * Updated the Model Context Protocol SDK dependency to version 1.29.0.
  * Removed unused AI utilities dependency.
* Optimized the internal AI service communication layer for improved
efficiency.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-30 15:46:50 -04:00
Matt Rossman 36ae9beb0c chore(ai): remove DPA signer killswitch for assistant tracing (#45134) 
Removes the temporary killswitch added when Braintrust was onboarded as
a subprocessor, to satisfy the 30-day DPA notice obligation. The window
has elapsed and legal has cleared removal.

Drops the `orgIsDpaSigned` check from `isTracingAllowed`, removes the
extra `/platform/organizations/{slug}/documents/dpa-signed` network hop
from `getOrgAIDetails`, and cleans up all call sites and tests.

Closes AI-596

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Refactor**
* Simplified AI tracing eligibility logic by removing DPA signing status
checks. Tracing authorization decisions now depend solely on region,
HIPAA addon status, and project sensitivity settings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-27 13:53:24 -04:00
Joshen Lim 7f5865872a Enforce noUnusedLocals and noUnusedParameters in tsconfig.json + fix all related issues (#45264) 
## Context

Enforce `noUnusedLocals` and `noUnusedParameters` in tsconfig.json + fix
all related issues
 2026-04-27 17:42:34 +08:00
Greg RichardsonandAli Waseem e38ba624bc feat(ai): update rls knowledge for 'secure by default' (#45072) 
Updates the RLS knowledge loaded by the dashboard AI assistant to
explain the new secure-by-default functionality.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Clarified PostgreSQL/RLS guidance in Studio: tables are now "secure by
default"—SQL-created tables aren’t exposed via the Data API unless
explicit grants are given to anon/authenticated/service_role and RLS is
enabled; added an “Exposing a Table to the Data API” workflow,
strengthened RLS prerequisites in best practices, and improved
troubleshooting/error-recovery guidance.
* **Tests / Evaluations**
* Added an evaluation case validating guidance for non-RLS tables
requiring explicit grants and RLS policies.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Ali Waseem <waseema393@gmail.com>
 2026-04-22 10:02:43 -06:00
Manuel RubioandAndrey A. 667be50982 fix(self-hosted): local AI assistant unable to read schemas and interpret/execute SQL (#44757) 
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Bug fix for #42574 

## What is the current behavior?

If you setup a self-hosted Supabase environment and ask local AI
assistant it is unable to interpret the results for executed SQL
queries.

## What is the new behavior?

Local AI assistant can now read schemas and interpret the results for
executed SQL queries.

## Additional context

I tested it using ollama with a smaller model which does not support
reasoning mode so It would be useful if someone can test it using an
actual OpenAI API KEY.

The main issue was that `executeSql` returns a wrapped `{ result: {...}
}` on `fallback-tools.ts` making the model think there's no result.
The solution is just to unwrap it like it's done at:
`apps/studio/data/database-functions/database-functions-query.ts`:

```
export async function getDatabaseFunctions(...) {
...
  const { result } = await executeSql(...)
...
  return result as DatabaseFunction[]
}
```

Also added a `false` default value in `execute_sql` tool schema so zod
validation does not reject a valid LLM tool call if the model omits the
field (which happened with the smaller model)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Refactor**
  * Optimized internal schema metadata query handling in AI tools.

* **Chores**
* Enhanced SQL execution tool robustness by providing a sensible default
for query type specification when not explicitly provided.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Andrey A. <56412611+aantti@users.noreply.github.com>
 2026-04-20 12:41:23 -04:00
Matt Rossman 8f69a10cc9 fix(studio): reliable schema-aware SQL editor AI completions (#44730) 
A variety of fixes and improvements to the Cmd+K AI completions endpoint
in the [SQL Editor](https://supabase.com/dashboard/project/_/sql/new):

- Pre-load table definitions for the public schema and any other schemas
referenced in the editor, so the model has real column names without
needing to fetch them dynamically
- Replace the generic tool suite with a single streamlined
`getSchemaDefinitions` tool the model can still call to look up
additional schemas on demand without behavior differences across
platform & self-hosted
- Swap generic chat system prompt for a purpose-built
`COMPLETION_PROMPT`; fix role (`assistant` → `user`) for consistency
with other endpoints
- Validate and type the request body with `zod`, which was previously
untyped (`any`)
- Improve Cmd+K behavior when nothing is selected — use the full editor
content as context, return the complete query rather than just the
changed fragment, and switch to a generation mode when the editor is
blank
- Escape single quotes in schema names when fetching entity definitions
in `pg-meta` to prevent schema names from breaking out of the SQL string
and injecting arbitrary content into the prompt

## Before

Before, the SQL Editor would often hallucinate tables / columns that
don't exist in the user's database making it less helpful if you don't
know the exact table/column names. Even with maximum Assistant opt-in
level on the org, it would often fail to call the necessary tools to
gather database context.

<img width="5062" height="1522" alt="image"
src="https://github.com/user-attachments/assets/fbe1130f-6b5a-41a8-99d7-7268880af188"
/>

<img width="2540" height="658" alt="image"
src="https://github.com/user-attachments/assets/a31c2967-7751-4fce-a9b7-60bd77660b1a"
/>

Sometimes it also silently fails and generates empty queries:

<img width="1352" height="398" alt="CleanShot 2026-04-09 at 17 46 06@2x"
src="https://github.com/user-attachments/assets/e17c103a-d47d-47e6-8c2e-101f0fae5651"
/>

Or echos back the user's prompt:

<img width="1368" height="282" alt="CleanShot 2026-04-09 at 23 04 56@2x"
src="https://github.com/user-attachments/assets/7dff6e64-f54e-45b5-8e86-5399e5a2fe41"
/>


## After

In this example, the completion correctly interpreted my request for
"completed" todos as a query on the `completed_foo` column in my
`public` schema, instead of assuming existence of a `completed` column.

<img width="1452" height="838" alt="CleanShot 2026-04-09 at 17 43 13@2x"
src="https://github.com/user-attachments/assets/7a575589-78b4-448d-810a-0330ff08ef8b"
/>

In this example, the completion was correctly aware of an `other` schema
because it was detected in my existing query. I didn't have to select
the text, it included the full query in context when unselected. Notice
how it correctly used the `is_done` column when I asked for "completed"
cakes:

<img width="1372" height="534" alt="CleanShot 2026-04-09 at 17 39 07@2x"
src="https://github.com/user-attachments/assets/e6b7eb6f-f3e8-4fa1-90a3-b5e34ddc14e4"
/>

Supersedes #44151

Closes AI-544
 2026-04-20 11:01:52 -04:00
Alaister YoungandAlaister Young 1b1d05ff96 chore: upgrade vite to v8 and vitest to v4 (#44833) 
Upgrade vite and vitest to their latest major versions across the
monorepo, along with related packages.

**Changed:**
- `vite` catalog: `^7.3.2` → `^8.0.8` (Rolldown replaces esbuild/Rollup)
- `vitest` catalog: `^3.2.0` → `^4.1.4`
- `@vitejs/plugin-react`: `^4.3.4` → `^6.0.1`
- `@vitest/coverage-v8`: `^3.2.0` → `^4.1.4`
- `@vitest/ui`: `^3.2.0` → `^4.1.4`
- `vite-tsconfig-paths`: `^4.3.2` / `^5.1.4` → `^6.1.1`

**Pinned to vite 7:**
- `apps/lite-studio` — `@react-router/dev` hasn't declared vite 8
support yet
- `blocks/vue` — Nuxt plugins (`vite-plugin-inspect`, `vite-dev-rpc`,
`vite-hot-client`, `vite-plugin-vue-tracer`) haven't declared vite 8
support yet

**Test fixes for vitest 4 breaking changes:**
- **`apps/studio/lib/api/snippets.utils.test.ts`** — Replaced
`vi.mock('fs/promises')` automock with an explicit factory. Vitest 4's
automocking doesn't create mock functions for getter-based exports on
Node built-ins, so `mockedFS.access.mockResolvedValue` etc. were
`undefined`.
- **`apps/studio/lib/api/self-hosted/functions/index.test.ts`** —
Changed `mockReturnValue` to `mockImplementation(function() { ... })`
for a constructor mock. Vitest 4 no longer allows `mockReturnValue` when
the mock is called with `new`.
- **`apps/studio/tests/pages/api/mcp/index.test.ts`** — Changed arrow
function to regular `function` in `mockImplementation` for
`StreamableHTTPServerTransport`. Arrow functions can't be constructors,
and vitest 4 now enforces this.
- **`packages/ui-patterns/vitest.setup.ts`** — Changed `ResizeObserver`
mock from arrow function to regular `function` for the same constructor
enforcement reason. This was crashing Radix popover rendering in jsdom.

## To test

- `pnpm test:studio` — all 226 test files should pass
- `pnpm --filter ui-patterns vitest run` — all 183 tests should pass
- `pnpm --filter www test -- --run` — all 19 tests should pass
- `pnpm --filter ui vitest run` — all tests should pass
- `pnpm --filter dev-tools vitest run` — all tests should pass
- `pnpm --filter ai-commands vitest run` — all tests should pass

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Standardized and updated development tooling versions and version
sources for consistent installs across the repo (Vite, Vitest,
vite-tsconfig-paths and related plugins/catalog entries).
* **Tests**
* Improved test mocks and typings (updated mock
factories/implementations and tightened spy/type assertions) to increase
test reliability and compatibility with updated tooling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com>
 2026-04-16 00:13:48 +09:00
Matt Rossman 4f90833ed9 feat(assistant): enable Braintrust tracing in production (#44644) 
Removes the temporary killswitch that was blocking tracing in production
environments.

Note this still excludes from tracing projects that are any of the
following:
- HIPAA sensitive
- Hosted in EU regions
- Within orgs where some owner previously signed our DPA

See [approval
message](https://supabase.slack.com/archives/C051L8U2EJF/p1775526578561789?thread_ts=1775144006.270549&cid=C051L8U2EJF)
for context.

Closes AI-450

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Tracing enablement now depends solely on the presence of required API
credentials instead of environment-based restrictions, enabling
diagnostic tracing functionality across all environments.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-09 08:47:27 -04:00
Charis 205cbe7d26 chore(studio}: enforce import order, remove bare import specifiers (#44585) 2026-04-07 20:34:10 -04:00
Matt Rossman 8aeacc6152 feat(assistant): disable Braintrust tracing for EU regions and DPA signers (#44504) 
**Changes**

- Extracted tracing conditional to an `isTracingAllowed` helper with
unit tests (the function is simple but sensitive hence the extra testing
precaution)
- Disables Braintrust tracing for projects in EU database regions
(region prefix `eu-`) to address GDPR data residency concerns
- Disables Braintrust tracing for orgs whose owners have signed the
previous DPA, as a stopgap during the 30-day notice period for the
updated DPA that adds Braintrust as a subprocessor
- Refactored `org-ai-details.ts` → `ai-details.ts`, splitting
`getOrgAIDetails` into separate org and project helpers to cleanly scope
the EU-region check at the project level

DPA check uses the newly added `/documents/dpa-signed` endpoint from
https://github.com/supabase/platform/pull/31060. This PR includes
regenerated `api.d.ts` and `platform.d.ts` from running `pnpm codegen`
in `packages/api-types` to get type safety on this new endpoint.

Note tracing is still yet to be activated in production, this is a
preparatory step.

**To verify**

Send a chat message and check for the `x-braintrust-span-id` response
header on `POST /api/ai/sql/generate-v4` — it should be absent for
DPA-signed orgs or EU-region projects, and present otherwise.

<img width="3594" height="1992" alt="CleanShot 2026-04-03 at 14 28
58@2x"
src="https://github.com/user-attachments/assets/4c91d7ad-2604-4531-a78e-dedf41632fa5"
/>

If you have access to the Braintrust dashboard, you can also verify
whether logs are produced or not in the Assistant project there.


Closes AI-570
Closes AI-569

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Tracks organization DPA signing and detects EU-region projects
* Assistant tracing now follows a combined compliance policy (HIPAA
addon, DPA, project sensitivity, region)
  * Added helpers to fetch org and project AI details

* **Documentation**
* Expanded API docs with additional examples and clarified parameter
descriptions
  * Added response schemas for subscription preview and document status

* **Tests**
* Added/updated tests covering DPA/region behavior and tracing policy
enforcement
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-07 13:09:34 -04:00
Matt Rossman 82deff37de feat(assistant): lazy load topic knowledge via load_knowledge tool (#44296) 
Moves knowledge (RLS, Edge Functions, PostgreSQL best practices,
Realtime) out of the static system prompt and into a `load_knowledge`
tool the model calls on demand, reducing prompt bloat. This is a
temporary stopgap until the [standard Supabase
agent-skills](https://github.com/supabase/agent-skills) are ready for
integration in Assistant.

- New always-available `load_knowledge` tool added to
`rendering-tools.ts`
- Updated `Message.Parts.tsx` so the "Ran load_knowledge" chip renders
in chat
- System prompt replaces the four knowledge blobs with an `## Available
Knowledge` block and is hardened to load knowledge for given topics
- New "Knowledge Usage" scorer and `requiredKnowledge` assertions check
that knowledge loads as expected in test scenarios
- Filters GraphQL error responses out of `output.docs` before
faithfulness scoring to reduce noise


See "Knowledge Usage" scoring 100% in evals with no major regressions:
https://github.com/supabase/supabase/pull/44296#issuecomment-4145760236

Sample trace showing the tool in action
([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=project_logs&object_id=5a8d02e5-b3b6-40cc-ba76-ecee286478f4&r=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1&s=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1))

<img width="2192" height="1730" alt="CleanShot 2026-03-30 at 13 53
59@2x"
src="https://github.com/user-attachments/assets/f483767c-34e0-401c-8089-5b9834fe696a"
/>


**References**
- https://ai-sdk.dev/cookbook/guides/agent-skills

Closes AI-508

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added dynamic knowledge loading capability enabling the AI assistant
to retrieve on-demand information about PostgreSQL best practices, Row
Level Security, Edge Functions, and Realtime.

* **Bug Fixes**
* Improved search results filtering to exclude error responses in tool
outputs.

* **Tests**
  * Enhanced evaluation metrics with knowledge usage scoring.
* Expanded test dataset cases to validate knowledge requirement
handling.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-02 16:09:06 -04:00
Matt Rossman 0c5f64fcba feat(assistant): upgrade default models to gpt-5.4-nano and gpt-5.3-codex (#44107) 
Replaces `gpt-5-mini` and `gpt-5` with `gpt-5.4-nano` and
`gpt-5.3-codex` respectively. Clients with stale model IDs in IndexedDB
will gracefully reset to the new defaults. While we can technically keep
the existing models around, we've
[opted](https://supabase.slack.com/archives/C051L8U2EJF/p1774283070517609?thread_ts=1773771991.871669&cid=C051L8U2EJF)
to replace them w/ the newer models for simplicity. Basic completion
endpoints use `'none'` reasoning level for optimal speed.

Rationale for these models is they provide they best balance of
intelligence/speed and cost. GPT-5.4-nano is less expensive (0.8x
price), faster, and smarter than GPT-5-mini. GPT-5.4-mini would be even
smarter but is 3x the price. GPT-5.3-Codex is ~1.4x the price of GPT-5,
while GPT-5.4 would be 2x price, but 5.3-Codex is still a big
intelligence boost from GPT-5.

See [eval
comparison](https://www.braintrust.dev/app/supabase.io/p/Assistant/experiments/mattrossman%2Fai-509-v2-upgrade-assistant-models-beyond-gpt-5-family-1774468619?c=master-1774458837&diff=between_experiments),
scores are relatively stable and conciseness naturally improves on
gpt-5.4-nano.

Other change:
- Fixed an eval test case to clarify that https://supabase.help is also
a correct URL for submitting support ticket, which was unfairly scored
as incorrect
[here](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=5244cccd-23b2-4f79-9dd2-287f1b40ebad&r=bac9b903-8bde-4c21-99dd-e0ed141c4f9e&s=f248fbf5-75bf-4aab-be0a-87a4298e6d11)

I sanity checked the Assistant, natural language filters, and SQL Editor
completions on staging preview.

References:
- https://openai.com/index/introducing-gpt-5-4-mini-and-nano/
- https://openai.com/index/introducing-gpt-5-3-codex/
- https://developers.openai.com/api/docs/pricing

Closes AI-509
 2026-03-26 14:35:54 +08:00
Matt Rossman adf8b0c67c feat(assistant): per-endpoint reasoningEffort + model config cleanup (#43981) 
We're exploring support for newer models like
[gpt-5.4-nano](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/)
in Assistant. This model doesn't support the `'minimal'` reasoning
effort level we use for gpt-5-mini which leads to vague errors.

<img width="595" height="263" alt="CleanShot 2026-03-18 at 17 13 05@2x"
src="https://github.com/user-attachments/assets/cf7c2370-322d-4a8a-be55-23e680db0aa0"
/>


Also, we've [previously
discussed](https://supabase.slack.com/archives/C0161K73J1J/p1771544464850199?thread_ts=1771493920.775699&cid=C0161K73J1J)
that reasoning adds unnecessary latency to otherwise simple AI
completion endpoints like `title-v2`. We want more control of reasoning
level independent of model/endpoint.

This PR aims to solve both problems by:
- making reasoning effort configurable on a per-request basis
- adding compile-time guardrails to prevent selecting an incompatible
reasoning level for models
- adding a `DEFAULT_COMPLETION_MODEL` with minimal reasoning that we can
update with newer models that support disabling reasoning (independent
of Assistant chat model reasoning)

Other improvements to our model config logic:
- Fixes bug in `onboarding/design.ts` and `assistant.eval.ts` where
`providerOptions` was being dropped
- `getModel()` now returns a bundled `modelParams` object (spread into
AI SDK calls) so `providerOptions` can't be accidentally omitted (this
[has happened
before](https://supabase.slack.com/archives/C0161K73J1J/p1771518443534309?thread_ts=1771493920.775699&cid=C0161K73J1J))
- Introduces an `ASSISTANT_MODELS` registry as a single source of truth
for assistant model config, eliminating hardcoded model IDs across the
codebase
- Aligns free/pro model conditional logic with `assistant.advance_model`
entitlement naming conventions instead of the `isLimited` pattern
- Adds `console.error` logging of Assistant stream errors so we can
interpret reasoning effort compatibility errors in the future (instead
of just opaque "Sorry, I'm having trouble responding right now" card)
- Removes unnecessary type casts and generally making the model config
logic stricter
- Removes pre-existing dead code: `anthropic` provider variant in
`GetModelParams` / `PROVIDERS` registry that was never implemented in
`getModel()`

Now if you try to select an unsupported reasoning level you get a type
error:

<img width="1306" height="320" alt="CleanShot 2026-03-20 at 14 37 24@2x"
src="https://github.com/user-attachments/assets/a6ac234b-5ea5-4d81-8e01-ac4be34a0800"
/>

And if for some reason an invalid reasoning level slips through, you now
get a server-side error surfacing the issue:

<img width="1268" height="204" alt="CleanShot 2026-03-20 at 14 58 14@2x"
src="https://github.com/user-attachments/assets/aadc1b7a-9495-475f-9741-39979bd27cd7"
/>

I've tested gpt-5 and gpt-5-mini are still working on the staging
preview and verified the models were selected properly in Braintrust
logs. Both models are available on my Pro test account, and my Free test
account shows the Pro upgrade CTA.


Closes AI-446
Closes AI-551
 2026-03-25 11:29:23 -04:00
Matt Rossman d29fbf6eb7 feat(assistant): upgrade AI SDK v5 → v6 (#43931) 
Upgrades `ai` from v5 to v6 and all related packages.

**Package bumps:**
- `ai`: `5.0.52` → `^6.0.116`
- `@ai-sdk/openai`: `2.0.32` → `^3.0.41`
- `@ai-sdk/react`: `2.0.52` → `^3.0.118`
- `@ai-sdk/provider`: `^2.0.0` → `^3.0.8`
- `@ai-sdk/provider-utils`: `^3.0.0` → `^4.0.19`
- `@ai-sdk/amazon-bedrock`: `^3.0.0` → `^4.0.81`
- `@ai-sdk/mcp`: N/A → `^1.0.25`
- `openai`: bumped to `^4.104.0`
- `braintrust`: `3.0.x` → `^3.4.0`

**Breaking change migrations:**
- `generateObject` removed in v6 — migrated 5 API routes to
`generateText` with `Output.object({ schema })`, returning
`result.output`
- `convertToModelMessages` is now async — added `await`
- MCP import path changed: `experimental_createMCPClient` from `ai` →
`createMCPClient` from `@ai-sdk/mcp`
- `openai()` defaults to Responses API — added `store: false` to
provider options for ZDR org compatibility

**Streaming fix:**
Added `Content-Encoding: none` header to `pipeUIMessageStreamToResponse`
calls. Without it, proxy middleware buffers the entire SSE response
before flushing, causing the full reply to appear at once.

**Zero Data Retention fix:**
In recent AI SDK versions, `openai()` default to Responses API instead
of the legacy chat completions API. This produces a 404 from OpenAI with
message `"Items are not persisted for Zero Data Retention organizations.
Remove this item from your input and try again."` The Responses API is
OpenAI's [recommended
endpoint](https://developers.openai.com/api/docs/guides/migrate-to-responses).

This PR adds `store: false` as mentioned in
https://github.com/vercel/ai/issues/10060 to avoid incompatible
persistence attempts.

**References:**
- https://ai-sdk.dev/docs/migration-guides/migration-guide-6-0
-
https://ai-sdk.dev/docs/troubleshooting/streaming-not-working-when-proxied
- https://github.com/vercel/ai/issues/10060

Closes AI-514
Related AI-509
 2026-03-19 09:07:33 -04:00
Joshen Lim be26feb9ba Chore/shift manual queries into pg meta 03 (#43951) 
## Context

Shifting more dashboard queries into pg-meta so that we centralize all
manually written queries in one place
Having them in packages/pg-meta also allows us to write tests for them

## To test

Just needs a smoke test on
- Table Editor
  - Fetching entities
  - Viewing definition 
- SQL Editor
  - View ongoing queries
  - Abort queries
- Integrations
  - Queues
- Database
  - Migrations
  -Triggers (Updating)
 2026-03-19 18:31:46 +08:00
Matt Rossman 25036af80e fix(assistant): sanitize backslash-escaped apostrophes in SQL (#43728) 
Fix for the LLM occasionally generating MySQL-style `\'` escapes in SQL,
which are invalid in PostgreSQL.

Example trace where this happened in the wild:
([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Assistant/review?tab=experiment&r=5fcf1b12-8584-455c-9e9a-bdc0fa3ed21c&s=5fcf1b12-8584-455c-9e9a-bdc0fa3ed21c&o=0627ada8-b567-4117-9fe8-49d847cb73a7&review=1))

**Changes**
- Adds `fixSqlBackslashEscapes` to convert `\'` → `''` before SQL is
executed
- Unit tests + adversarial eval dataset case

Compare the results of the adversarial test case:
- `master`: 0% SQL Validity
([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Dev%20(mattrossman%2FAssistant)/trace?object_type=experiment&object_id=b469cbf7-4d6f-429c-9819-6c4099294123&r=dce5a29b-2fde-44c3-80f8-4e14d1f657c0&s=dce5a29b-2fde-44c3-80f8-4e14d1f657c0))
- This branch: 100% SQL Validity
([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=160e9ce0-e320-4f6d-8aa7-c5ad7e01fbd2&r=d75ef0e3-90ed-42a7-9ef3-8bf69592f193&s=0eeca492-dbe6-451e-8d81-127caff30320))

Closes AI-400
 2026-03-17 13:44:14 -04:00
Ivan VasilovandJoshen Lim 9fa96977be chore: Minor prettier fixes (#43849) 
This PR fixes some prettier issues:
- Bump and unify all prettier versions to 3.7.3 across teh whole repo
- Bump the SQL prettier plugin
- When running `test:prettier`, check `mdx` files also
- Run the new prettier format on all files

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
 2026-03-17 11:17:42 +01:00
Matt Rossman f49ed14dfb fix(assistant): preserve template URL placeholders in responses (#43495) 
Fixes a rendering bug where template URLs like
`https://<project-ref>.supabase.co/...` were displayed incorrectly in
Assistant responses. The markdown parser was treating `<project-ref>` as
a tag and silently stripping it.

- Adds `wrapPlaceholderUrls()` preprocessing step that auto-wraps bare
URLs containing `<kebab-case>` placeholders in backticks before passing
to the markdown renderer
- Strengthens the prompt instruction to explicitly say "always format
template URLs as inline code using backticks"

Sample prompt:
```
What are the OAuth2 endpoints for Supabase projects? List the authorization, token, and JWKS URLs.
```

| Before | After |
|--------|--------|
| <img width="1314" height="841" alt="CleanShot 2026-03-06 at 15 32
13@2x"
src="https://github.com/user-attachments/assets/55eaa0c6-39fb-48e0-91a0-31903021a4c9"
/> | <img width="1421" height="827" alt="CleanShot 2026-03-06 at 15 31
37@2x"
src="https://github.com/user-attachments/assets/5335c71a-19c7-44a0-b027-6f2efd76eb8c"
/> |

Closes AI-470
 2026-03-11 13:56:43 -04:00
Matt Rossman 517171b246 feat(assistant): online evals support and CI workflows (#43194) 
Lays groundwork for online evals on Assistant chat logs.

https://www.braintrust.dev/docs/observe/score-online

### Changes

- New workflows:
- `braintrust-scorers-deploy.yml` keeps prod scorers in sync on push to
`master`
- `braintrust-preview-scorers-deploy.yml` deploys preview scorers to the
staging project for PRs labeled `preview-scorers`, posting a comment
with scorer links
([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222))
- `braintrust-preview-scorers-cleanup.yml` deletes preview scorers when
the PR is closed
([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000749847))
- Adds `evals/scorer-online.ts` entry point invoked with `pnpm
scorers:deploy`, registering scorers for online evals in the Braintrust
"Assistant" project
- Refactors scorer code to separate online-compatible scorers
(`scorer-online.ts`) from WASM-dependent ones (`scorer-wasm.ts`)
- "URL Validity" scorer now only checks Supabase domains to prevent
requests to untrusted origins
- Span `input` is now shaped `{ prompt: string }` instead of plain
`string` for compatibility with offline eval scorers
- Env vars `BRAINTRUST_STAGING_PROJECT_ID` and `BRAINTRUST_PROJECT_ID`
configured in GitHub repo settings
- `generateAssistantResponse` now uses `startSpan` + `withCurrent`
instead of `traced()` to manually manage the root span lifecycle — this
ensures `onFinish` logs output to the span _before_ `span.end()` is
called, which is when Braintrust triggers scoring automations

### Online Scorers

We share scoring logic across offline and online evals, but some of our
scorers aren't transferrable to an "online" setting due to runtime
challenges or ground truth requirements.

**Supported**
- Goal Completion
- Conciseness
- Completeness
- Docs Faithfulness
- URL Validity

**Unsupported**
- Correctness (requires ground truth output)
- Tool Usage (requires ground truth requiredTools)
- SQL Syntax (uses libpg-query WASM)
- SQL Identifier Quoting (uses libpg-query WASM)
 
### How to use these scorers

Going forward if you want to add/edit online eval scorers, add the
`preview-scorers` label to a PR. This deploys scorers to the [Assistant
(Staging
Scorers)](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)?v=Overview)
project in Braintrust with branch-specific slugs, and comments on the PR
([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222)).
From the Braintrust dashboard you can "Test" the scorer with traces from
any project.

<img width="1866" height="528" alt="CleanShot 2026-03-05 at 15 15 00@2x"
src="https://github.com/user-attachments/assets/4f15cebc-3f2d-4e8a-9ee2-fe8ef7bf4199"
/>

Once merged, scorers are deployed to the primary
[Assistant](https://www.braintrust.dev/app/supabase.io/p/Assistant)
project, and preview scorers are deleted from the staging project. Down
the road, scorers on the Assistant project will run automatically on a
sample of production traces.

Closes AI-437
 2026-03-09 13:05:26 -04:00
Matt Rossman 20d1ac0f83 feat(assistant): send user feedback to Braintrust traces (#43021) 
- Expose Braintrust span ID to client via `x-braintrust-span-id`
response header, captured in chat transport
- On feedback submission, call `logFeedback()` with `scores["User
Rating"]` (1/0), `comment`, and on the root span assign
`metadata.feedbackCategory`
- Silently skipped when tracing is disabled (HIPAA, missing env vars)
- Log `requestedModel` in trace metadata so we can see what model the
user selected vs what was actually used after throttling

Example traces:

- [Thumbs
up](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=project_logs&object_id=5a8d02e5-b3b6-40cc-ba76-ecee286478f4&r=0bb71680-784c-45c1-a234-cba0242562d6&s=0bb71680-784c-45c1-a234-cba0242562d6)
- [Thumbs down + negative
feedback](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=project_logs&object_id=5a8d02e5-b3b6-40cc-ba76-ecee286478f4&r=d5a78084-6c9a-4230-8615-1e864bb9bac7&s=d5a78084-6c9a-4230-8615-1e864bb9bac7)

<img width="645" height="173" alt="CleanShot 2026-02-19 at 13 30 25@2x"
src="https://github.com/user-attachments/assets/6c463e83-27c6-4afb-a8d0-a329ed61270a"
/>

Closes AI-442
 2026-02-19 15:00:07 -05:00
Ignacio Dobronich d5cc70560d chore: ai assistant model entitlement (#42955) 
### Changes
Replace hard-coded plan checks with `assistant.advance_model`
entitlement for AI model access control.

- `AIAssistant.tsx` & `ModelSelector.tsx`: Use
`useCheckEntitlements('assistant.advance_model')` instead of `plan.id
!== 'free'`

**Server-side:**
- `org-ai-details.ts`: Check `assistant.advance_model` entitlement via
new `checkEntitlement()` helper
- `entitlements-query.ts`: Add `checkEntitlement()` function with auth
headers support for server-side calls


### Test 1: Free Plan (No Entitlement)
- Open AI Assistant with a Free plan org
- Check model selector defaults to gpt-5-mini
- Click dropdown - verify gpt-5 shows "Upgrade" badge
- Click gpt-5 - should redirect to billing page
- Assert that you can send a message to the Assistant and that you get a
response
<img width="335" height="156" alt="image"
src="https://github.com/user-attachments/assets/2458cbd1-46ab-46cd-babb-09a47e163fd0"
/>


### Test 2: Paid Plan (Has Entitlement)
- Switch to Pro/Team/Enterprise org
- Check model selector defaults to gpt-5
- Click dropdown - verify gpt-5 shows checkmark (no badge)
- Switch to gpt-5-mini, then back to gpt-5 - should work without
redirect
- Assert that you can send a message to the Assistant and that you get a
response
 2026-02-19 14:28:31 -03:00
Matt Rossman e8ab92408f feat(assistant): enable Braintrust tracing for non-sensitive chats (#42963) 
Enables Braintrust tracing for AI Assistant chats to support debugging
and future online evals.

**Code Changes**

- Wraps `generateAssistantResponse` in a Braintrust `traced()` span,
logging the user's latest message as input along with metadata
(`chatId`, `chatName`, `projectRef`, `userId`, `orgId`, `planId`, etc.)
- Threads JWT claims from `apiWrapper` → handler to log `userId` in
Braintrust without an extra API call (+ expanded `apiWrapper` tests)
- Threads `orgId` and `planId` from `getOrgAIDetails` to log in
Braintrust

**Infrastructure Changes**

- Created a "Vercel" service account in Braintrust
- Added `BRAINTRUST_API_KEY` and `BRAINTRUST_PROJECT_ID` env vars to the
studio-staging project in Vercel using a service token for the above
service account
- Added an "Overview" view to the Logs tab in the Braintrust Assistant
project to surface the new metadata

**Precautions**

- HIPAA sensitive projects are excluded from logging (see
https://github.com/supabase/supabase/pull/42787 for the detection logic)
- Production is temporarily excluded from logging until we're confident
in the setup

**Testing steps**

- Chat with the AI Assistant in the [studio-staging preview
build](https://github.com/supabase/supabase/pull/42963#issuecomment-3917178023)
below
- Visit the [Logs tab in the Braintrust Assistant
project](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs)
and inspect the trace

<img width="4680" height="962" alt="CleanShot 2026-02-18 at 17 43 55@2x"
src="https://github.com/user-attachments/assets/c3a11b21-4e7f-4e90-bdab-a25ab8ee0d1f"
/>

<img width="2632" height="1288" alt="CleanShot 2026-02-18 at 17 45
04@2x"
src="https://github.com/user-attachments/assets/6c7b6ebc-5090-4ede-8f71-859ff7e386aa"
/>

**References**
- https://www.braintrust.dev/docs/integrations/sdk-integrations/vercel
- https://www.braintrust.dev/docs/instrument/custom-tracing

Closes AI-438
 2026-02-19 11:43:47 -05:00
Matt Rossman 2fc062a725 feat(assistant): detect HIPAA customers in assistant logic (#42787) 
Detects HIPAA customers server-side in the assistant code path. Threads
`isHipaaEnabled` boolean through `getOrgAIDetails` → `generate-v4` →
`generateAssistantResponse`. The motivation is to support online evals
down the road, where we'll want to exclude HIPAA projects from Assistant
tracing.

This PR follows existing patterns for checking if HIPAA is enabled for a
project (org has HIPAA addon + project is sensitive). Example
[[1]](https://github.com/supabase/supabase/blob/a5dd0a96716561443778f38a518b61d6cac95c19/apps/studio/components/interfaces/Settings/Addons/Addons.tsx#L75),
[[2]](https://github.com/supabase/supabase/blob/6858d4e18d9359d573fe3dff73bc4e5fa1cfe219/apps/studio/hooks/misc/useOrgOptedIntoAi.ts#L69).

```ts
const hasHipaaAddon = subscriptionHasHipaaAddon(subscription) && settings?.is_sensitive
```

(I call it `isHipaaEnabled` in this PR to avoid it being misunderstood
as just the org-level addon, rather it's a combo of that addon being
present AND high compliance being enabled on the project).

### Verification steps

<details><summary>Click to view the steps I followed to sanity check it
works with the local stack</summary>

Tested locally with `mise fullstack`:

1. Found my org's subscription ID:

   ```sh
docker exec platform-db-1 psql -U postgres -c "SELECT id, customer_id,
status FROM orb.subscriptions;"
   ```

2. Added HIPAA addon to it:

   ```sh
   docker exec platform-db-1 psql -U postgres -c "
     UPDATE orb.subscriptions
SET price_intervals = price_intervals || '[{\"price\": {\"unit_config\":
{\"unit_amount\": \"350.00\"}, \"external_price_id\":
\"addon_security_hipaa\", \"item\": {\"name\": \"HIPAA\"}}}]'::jsonb
     WHERE id = '<subscription_id>';"
   ```

2. Toggled on High Compliance (Project Settings → General)

3. Added a temporary log after `getOrgAIDetails` in `generate-v4.ts`:

   ```ts
   console.log('[HIPAA]', { isHipaaEnabled })
   ```

4. Sent a message in the AI Assistant → `isHipaaEnabled: true`

5. Toggled off High Compliance → resent → `isHipaaEnabled: false`

6. Removed addon from subscription, left project toggle on →
`isHipaaEnabled: false`

   ```sql
   -- Find addon index:
   SELECT ordinality - 1 as idx FROM orb.subscriptions,
jsonb_array_elements(price_intervals) WITH ORDINALITY AS elem(val,
ordinality)
     WHERE id = '<subscription_id>'
     AND val->'price'->>'external_price_id' = 'addon_security_hipaa';

   -- Remove by index:
UPDATE orb.subscriptions SET price_intervals = price_intervals - <idx>
     WHERE id = '<subscription_id>';
   ```

All three cases confirm `isHipaaEnabled` requires both the org addon and
the project-level toggle.


</details> 

Closes AI-434

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added HIPAA mode detection and exposed it in AI workflows.
* API request functions now accept optional custom authorization headers
for downstream calls.

* **Tests**
* Added tests covering HIPAA scenarios and verifying authorization
header propagation in related flows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-02-17 15:59:26 -05:00
Jordi Enric ac64a902c1 chore: adds tests (#42653) 2026-02-11 09:50:11 +01:00
Matt Rossman 4b8bab4d14 feat(assistant): score URL validity and fix support ticket URL guidance (#42227) 
**Logic changes**
- Adds function in `helpers.ts` to extract URLs from text via regex
- I also considering using a library like
[linkify-it](https://www.npmjs.com/package/linkify-it) for this but
figured it's not worth the extra dep
- Adds associated tests in `helpers.test.ts`
- Adds "URL Validity" scorer which performs a HEAD request for links in
Assistant response text and determins what portion of links have `.ok`
responses
- Adds eval case to check correctness of support ticket URL answers

**Prompt changes**
- Informs Assistant of https://supabase.com/dashboard/support/new being
the URL to create support tickets
- Encourages Assistant to "self-debug" issues before directing users to
create support tickets

See [Eval
Report](https://github.com/supabase/supabase/pull/42227#issuecomment-3807772871)
and
[Correctness](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=1ad0f9b0-5adb-436c-9812-a87aac62c036&r=1ef13459-a98c-4904-925e-6d81276cebb2&s=dbe5c607-a560-462b-8745-41d430744431)
analysis for new support ticket test case.

Resolves AI-384

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added URL validity scoring to evaluations and helper utilities for
extracting/cleaning URLs.
* Added evaluation cases for support-ticket URL handling and OAuth
callback guidance.

* **Documentation**
* Updated assistant guidance to prefer self-resolution, include
support-ticket direction, clarified data-recovery search steps, and
added template-URL notation.

* **Tests**
* Expanded URL extraction and related utility tests to cover many
formats and edge cases.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-01-30 09:53:21 -05:00
Matt Rossman eb259f1364 feat(assistant): score and improve SQL identifier quoting (#42122) 
* feat: SQL correctness scorer, override mock tables

* feat: replace "SQL Correctness" with "SQL Identifier Quoting" scorer

* fix(prompt): discourage simulating confirmation of execute_sql tool

this is already handled at the UI layer

* fix(prompt): encourage quotes on identifiers with caps

* feat: move extractIdentifiers to own file, add tests

* chore: shorten tests

* feat: extract ColumnDef column names in extractIdentifiers

* refactor: sqlIdentifierQuotingScorer with more thorough checks

* refactor: consolidate into `sql-identifier-quoting.ts`

* feat: support mocking schemas, eval test case with case sensitive schema

* fix: test cases that don't match default mock schema

* chore: format

* feat(prompt): mention special characters and reserved words

* feat: optional description in metadata, test with special characters

* feat: consolidated comprehensive test case

* fix(prompt): revert conflicting instruction
 2026-01-27 09:28:56 -05:00
Danny WhiteandCharis Lam 05dc676d36 feat(studio): incident-aware AI assistant (#41603) 
* handling and mock data support

* admonition

* only show in empty state

* rabbit

* mock without local overrides

* remove admonition

* make incident banner more prominent

* remove mock data

* move to tool

* prettier

* fix(studio): get_active_incidents tool

---------

Co-authored-by: Charis Lam <26616127+charislam@users.noreply.github.com>
 2026-01-21 13:52:29 +11:00
Matt Rossman 4553f09bb5 feat(assistant): hallucination scorers + corrective measures for storage versioning answers (#41655) 
* feat: "Docs Faithfulness" scorer

* feat: test case for storage object restoration

* feat: "Factuality" scorer

* feat: "Factuality" -> "Correctness"

* feat: update Storage recovery test case

* feat: finishReason in task output

* feat: encourage parallel tool calls + docs search, discourage superfluous context gathering

* prompt tuning (tool selection strategy)

* add data recovery section in chat prompt

* test: S3 versioning support correctness

* refactor: derive stepsSerialized/textOnly from shared steps data

* fix: input in correctness scorer
 2026-01-15 14:22:46 +07:00
Matt Rossmancoderabbitai[bot]Ali Waseem
 		072883bcec feat: assistant evals (#41311) 
* chore: bump `supabase` CLI

* chore: stricter message types in `generate-v4.ts`

* feat: tutorial eval

https://www.braintrust.dev/docs/evaluation

* feat: project ID for eval

* refactor: `generateAssistantResponse` out of `handlePost`

* refactor: generateAssistantResponse to lib/ai

* feat: factuality eval with assistant response

* chore: upgrade braintrust to v1.0.1

* chore: silence tsconfig warning

* feat: assertion scorer

* fix: aggregate tools across all steps

* refactor: strict tool names, remove need for `as const`

* refactor: generic tool name type in assertions

* feat: transfer mocks from `feature/braintrust`

* feat: LLM criteria assertion

* feat: braintrust evals workflow

* fix: BRAINTRUST_PROJECT_ID

* feat: `sql_similar` assertion

* fix: `OPENAI_API_KEY` in workflow env

* feat: split AssertionScorer into separate scorers

* feat: remove tutorial eval

* feat: 20 minute CI timeout

* feat: category in test case metadata

* feat: score with gpt-5

* refactor: dataset to own file, colocate scorers

* feat: "gpt-5.2-2025-12-11" for llm as a judge

* feat: SQL syntax scorer with `libpg-query`

* feat: `evals:setup` and `evals:run` scripts

* feat: `evals:setup` in CI

* feat: human readable scorer names

* chore: rename to "SQL Validity"

* feat: add 2 "sql_generation" test cases

* feat: update requiredTools in test cases

* chore: ignore Cursor MCP config

* feat: "Conciseness" score

* feat: "Completeness" scorer

* fix: generate-v4 test mocks

* feat: serialize "steps" for scorer inputs

* updated node mem options for typecheck

* updated runner

* remove ram update as actions handle this

* feat: read `BRAINTRUST_PROJECT_ID` from secrets

* feat: score helpfulness, remove old scorers

* feat: separate `evals:run` and `evals:upload` scripts

* feat: passthrough entire classifier result

* feat: use live `search_docs` impl, store docs result in metadata

* feat: reduce classifier options

* feat: filter workflow by `run-evals` PR label or `master` branch

* chore: cleanup stubbed mock tools

* fix: checkout actual branch with `ref:`

* fix: capture search_docs results from all content parts

* feat: simplify sql syntax score calculation

* feat: use AI SDK's UI message validator

* docs: justification for relative `extends`

* fix: cleanup leftover validatedMessages

* doc: note mock token isn't secret for snyk

* fix: mock ui message to pass validation

* feat: revert ignoring Cursor MCP config

Using `.git/info/exclude` instead until we have an opinion on this

* feat: add "tsconfig" as shared-data devDependency, revert relative path in tsconfig

* refactor: tool call parsing into function

* Update apps/studio/evals/assistant.eval.ts

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* refactor: organize mock schemas and tool factories

---------

Co-authored-by: Ali Waseem <waseema393@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2025-12-22 23:45:48 -05:00
Saxon FletcherandKevin Grüneberg 5579b8e7eb Added billing context (#40933) 
* added billing context

* Update apps/studio/lib/ai/prompts.ts

Co-authored-by: Kevin Grüneberg <k.grueneberg1994@gmail.com>

* direct links billing

---------

Co-authored-by: Kevin Grüneberg <k.grueneberg1994@gmail.com>
 2025-12-08 16:27:41 +10:00