dima/supabase
1
0 Fork 0
mirror of https://github.com/supabase/supabase.git synced 2026-10-06 01:45:10 +03:00
Code Issues Packages Projects Releases Wiki Activity
38,871 Commits 1,782 Branches 62 Tags
codex/fix-tanstack-e2e
Commit Graph
10 Commits
Author SHA1 Message Date
claude[bot]andClaude ea3a7743b1 feat(studio): default AI Assistant to GPT-6 Luna (AI-1245) (#50757) 
<!-- ccr-slack-attribution -->
_Requested by **Saxon Fletcher** · [Slack
thread](https://supabase.slack.com/archives/C051L8U2EJF/p1790104861710199?thread_ts=1790104861.710199&cid=C051L8U2EJF)_

## Problem

**Before:** The Assistant's base model is `gpt-5.6-luna` at medium
reasoning effort.

**After:** The base model is `gpt-6-luna`, its direct successor, still
at medium effort. It costs half as much: $0.10/$0.50 per MTok against
$0.20/$1.20.

Resolves
[AI-1245](https://linear.app/supabase/issue/AI-1245/move-the-assistants-base-model-to-gpt-6-luna).

## Solution

This swaps `gpt-5.6-luna` for `gpt-6-luna` in
`apps/studio/lib/ai/model.utils.ts`: the model ID union, the
reasoning-support map, `ASSISTANT_MODELS`,
`DEFAULT_ASSISTANT_BASE_MODEL_ID`, and the OpenAI provider registry. The
old ID is removed, not kept next to the new one. A stored selection of
`gpt-5.6-luna` is no longer a known ID, so the client and `generate-v4`
both fall back to the new default. The eval cost table (AI-1242) and
eval experiments (AI-1243) are out of scope.

Source for the model ID and supported efforts (none/low/medium
default/high/xhigh/max): [OpenAI model docs: GPT-6
Luna](https://developers.openai.com/api/docs/models/gpt-6-luna).
`@ai-sdk/openai@4.0.41` types model IDs as a union plus `string & {}`,
so no SDK bump is needed.

## Review instructions

1. Check the diff in `model.utils.ts`. Say so if you'd rather keep
`gpt-5.6-luna` selectable as a fallback.
2. AI-1245 asks for the Assistant evals before shipping. Add the
`run-evals` label to run `braintrust-evals.yml` on this PR, or run `pnpm
--filter studio evals:run` locally. They have not been run yet because
they need OpenAI/Braintrust credentials.
3. Already run: studio `typecheck`, eslint + prettier on the changed
files, vitest for `lib/ai`, `pages/api/ai` and `state/ai-assistant` (264
passed), and `evals:preflight`.

## Checklist

- [x] I have read
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
- [x] No docs topics changed

**AI disclosure:** Claude Code wrote this PR from start to finish. Saxon
Fletcher (@SaxonF) is the accountable human and must review it before
merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01W21zLTfzde6FFPyYGbGC6K


---
_Generated by [Claude
Code](https://claude.ai/code/session_01W21zLTfzde6FFPyYGbGC6K)_

Co-authored-by: Claude <noreply@anthropic.com>
 2026-09-23 10:03:27 +10:00
Saxon FletcherandCursor 9b17ce8f2c chore(studio): default assistant to GPT-5.6 Luna (#49749) 
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Feature / chore: hide assistant model selection in the UI and default
chats to GPT-5.6 Luna.

## What is the current behavior?

The assistant composer exposes a model picker. Paid orgs default to
`gpt-5.3-codex`; everyone else defaults to `gpt-5.4-nano`.

## What is the new behavior?

- The model picker is hidden in the assistant composer and Explorer
home.
- Chats default to `gpt-5.6-luna` with `reasoningEffort: medium`.
- Model selection plumbing is kept (registry, entitlements, `setModel`,
generate-v4 request body) so a requested model can still be honored when
provided.
- Other completion endpoints still use `gpt-5.4-nano`.

## Additional context

Model selector UI can be re-enabled by passing `selectedModel` /
`onSelectModel` to `AssistantChatForm`.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for the GPT-5.6 Luna model with medium reasoning
capability.
  * Made GPT-5.6 Luna the default assistant model.
* **Improvements**
* Simplified assistant chat by removing model selection from the primary
chat experience.
  * Updated model fallback behavior to use the standard assistant model.
* Chat forms can now optionally display model selection when configured.
* **Tests**
* Updated model coverage and assistant chat tests for the new defaults
and behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-09-02 10:24:37 +10:00
Jordi EnricClaude Opus 4.8Joshen Lim
 		2aa1b52234 feat(studio): add feature to rewrite queries DEBUG-145 (#47266) 
## Problem

Moving the Logs Explorer to ClickHouse means users' saved BigQuery
queries no longer run.
<img width="2430" height="1010" alt="CleanShot 2026-06-29 at 11 36
04@2x"
src="https://github.com/user-attachments/assets/ae0ab155-7d3d-4ae9-81c3-22bf3a88cf8c"
/>

## Fix

Rewrite the query with AI instead of a SQL transpiler. AI handles the
long tail of nested fields and dialect differences far better than a
rule-based rewriter, and it needs no extra runtime dependency.

- `rewriteLogsSqlWithAI` posts the current query to
`/api/ai/code/complete` with `dialect: 'clickhouse'`. The endpoint skips
the Postgres schema and best-practices for that dialect and uses
logs-specific instructions and model so the output is ClickHouse logs
SQL (FROM `logs` + `source` filter, no `unnest` joins, nested fields
read from `log_attributes['...']`).
- The query's `source` is detected and its real `log_attributes` keys
are fetched and passed to the model, so it maps to exact paths instead
of guessing.
- The rewrite runs in the background and is proposed as a side-by-side
accept/discard diff in the editor. The AI Assistant panel is not opened.
- Entry points: a banner shown only for legacy-looking queries
(dismissal persisted), and a "Fix Query" button next to Field Reference.
- The Field Reference drawers discover `log_attributes` keys from real
data so the listed fields match what the source actually emits.

## Dependencies

Built on top of #47265 (Logs Explorer -> OTEL endpoint) — that is the
base branch of this PR. Merge #47265 first. Behind `otelLegacyLogs` (off
by default).

Part of DEBUG-145 (split from #47087).

## How to test

- Open the Logs Explorer with a BigQuery logs query (the templates have
some), click "Fix Query", and confirm the diff shows valid ClickHouse
SQL. Accept it and confirm the applied query runs.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an OTEL legacy logs workflow (behind a feature flag) with an
interactive banner and a “Fix Query” ClickHouse rewrite action,
including an accept/discard diff review overlay.
* Introduced OTEL-aware field reference rendering with dynamic discovery
of `log_attributes` keys and updated OTEL source insertion behavior.
* Enabled dialect-aware SQL completion for ClickHouse logs, using
logs-specific instructions and output constraints.
* **Bug Fixes**
* Improved rewrite flow validation and handling, including log source
detection and cleanup of AI-generated SQL formatting.
* **Tests**
* Added Vitest coverage for rewrite prompt generation,
detection/classification utilities, SQL fence stripping, OTEL field
mapping, and OTEL log attribute key discovery.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
 2026-06-29 14:31:18 +02:00
Matt Rossman d143571586 feat(assistant): trace-level scorers + server-side tool execution with needsApproval (#45654) 
## Motivation

When Assistant runs a potentially destructive tool like `execute_sql`,
it stops the LLM request and prompts for client-side approval and
execution of the tool. After approval, a second request kicks off under
a separate trace. This has made scoring and
[Topics](https://www.braintrust.dev/blog/topics) classification
challenging, as the generated `output` is split across stateless
requests. The [span-level
scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans)
approach we've used thusfar (after the LLM call, we massage the result
into an `output` payload that's stuck onto the root span) has been
cumbersome and led to invalid scores / topics where only part of the
assistant response is considered. It's also inefficient, as we're
duplicating potentially large info (like the `search_docs` output) that
already exists within the trace.

An alternative to scoring spans is to [score
traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces).
Braintrust [best
practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices)
advise:

> Use span scope for evaluating individual operations or outputs. Use
trace scope for evaluating multi-turn conversations, overall workflow
completion, or when your scorer needs access to the full execution
context.

We've also received [direct
guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89)
from their team to use this approach.

## Changes

Migrates eval scorers from custom `AssistantEvalOutput` shape to
trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with
thread parsing that scores the full latest Assistant turn and passes
prior conversation separately where relevant.

Moves `execute_sql` and `deploy_edge_function` from client-side
execution after approval to AI SDK `needsApproval` + server-side
`execute()`. SQL results returned to the model are gated by AI opt-in
level, so row data is only included with `schema_and_log_and_data`;
otherwise the tool returns the no-data-permissions sentinel.

Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within
an "assistant" turn due to tool call requests/responses. For online
evals, this means we should configure automations to only score traces
with `metadata.isFinalStep = true` to ensure we're judging the complete
generated response.

Other minor kaizen changes:
- Renamed `promptProviderOptions` to `systemProviderOptions` to clarify
that this is associated with the "system" message and disambiguate from
the root `providerOptions`
- Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown`
span shapes from Braintrust, to more easily access typed inputs/output
on tool spans.
- Bumps AI SDK floor version `^6.0.116` → `^6.0.174`
- Tweaked the "Conciseness" scorer to not unfairly dock points for the
new `[called tool_name]` labels in serialized assistant response

## Verification

In the studio staging build, I asked Assistant to create a todos table
with 3 sample todos. I manually approved the `execute_sql` call and saw
Assistant generate text before & after the call.

In Braintrust I verified two traces were produced (see [filtered
logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})),
the first with `metadata.isFinalStep = false` and the second with
`metadata.isFinalStep = true`.

In the Braintrust staging scorers, I ran the preview Completeness scorer
on the second trace and verified it sees the complete Assistant response
including markers for tool calls ([link to
trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82))

<img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x"
src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a"
/>

I also tested the `deploy_edge_function` workflow and verified it still
prompts for permission and warns on deployment of existing functions.

**References**
- https://www.braintrust.dev/docs/evaluate/custom-code#score-traces
-
https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval

Supercedes https://github.com/supabase/supabase/pull/45556 and
https://github.com/supabase/supabase/pull/45339

Closes AI-473

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Tool actions (SQL execution, edge-function deploy) now require
explicit user Approve/Deny before proceeding.

* **Improvements**
* Assistant pauses for approval responses before sending follow-ups,
giving clearer control over risky actions.
  * Deploy/replace flows show confirmation and clearer replace warnings.
* Evaluation/scoring updated to use richer trace data for more accurate
assistant performance signals.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-12 15:24:21 -04:00
Matt Rossman 0c5f64fcba feat(assistant): upgrade default models to gpt-5.4-nano and gpt-5.3-codex (#44107) 
Replaces `gpt-5-mini` and `gpt-5` with `gpt-5.4-nano` and
`gpt-5.3-codex` respectively. Clients with stale model IDs in IndexedDB
will gracefully reset to the new defaults. While we can technically keep
the existing models around, we've
[opted](https://supabase.slack.com/archives/C051L8U2EJF/p1774283070517609?thread_ts=1773771991.871669&cid=C051L8U2EJF)
to replace them w/ the newer models for simplicity. Basic completion
endpoints use `'none'` reasoning level for optimal speed.

Rationale for these models is they provide they best balance of
intelligence/speed and cost. GPT-5.4-nano is less expensive (0.8x
price), faster, and smarter than GPT-5-mini. GPT-5.4-mini would be even
smarter but is 3x the price. GPT-5.3-Codex is ~1.4x the price of GPT-5,
while GPT-5.4 would be 2x price, but 5.3-Codex is still a big
intelligence boost from GPT-5.

See [eval
comparison](https://www.braintrust.dev/app/supabase.io/p/Assistant/experiments/mattrossman%2Fai-509-v2-upgrade-assistant-models-beyond-gpt-5-family-1774468619?c=master-1774458837&diff=between_experiments),
scores are relatively stable and conciseness naturally improves on
gpt-5.4-nano.

Other change:
- Fixed an eval test case to clarify that https://supabase.help is also
a correct URL for submitting support ticket, which was unfairly scored
as incorrect
[here](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=5244cccd-23b2-4f79-9dd2-287f1b40ebad&r=bac9b903-8bde-4c21-99dd-e0ed141c4f9e&s=f248fbf5-75bf-4aab-be0a-87a4298e6d11)

I sanity checked the Assistant, natural language filters, and SQL Editor
completions on staging preview.

References:
- https://openai.com/index/introducing-gpt-5-4-mini-and-nano/
- https://openai.com/index/introducing-gpt-5-3-codex/
- https://developers.openai.com/api/docs/pricing

Closes AI-509
 2026-03-26 14:35:54 +08:00
Matt Rossman adf8b0c67c feat(assistant): per-endpoint reasoningEffort + model config cleanup (#43981) 
We're exploring support for newer models like
[gpt-5.4-nano](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/)
in Assistant. This model doesn't support the `'minimal'` reasoning
effort level we use for gpt-5-mini which leads to vague errors.

<img width="595" height="263" alt="CleanShot 2026-03-18 at 17 13 05@2x"
src="https://github.com/user-attachments/assets/cf7c2370-322d-4a8a-be55-23e680db0aa0"
/>


Also, we've [previously
discussed](https://supabase.slack.com/archives/C0161K73J1J/p1771544464850199?thread_ts=1771493920.775699&cid=C0161K73J1J)
that reasoning adds unnecessary latency to otherwise simple AI
completion endpoints like `title-v2`. We want more control of reasoning
level independent of model/endpoint.

This PR aims to solve both problems by:
- making reasoning effort configurable on a per-request basis
- adding compile-time guardrails to prevent selecting an incompatible
reasoning level for models
- adding a `DEFAULT_COMPLETION_MODEL` with minimal reasoning that we can
update with newer models that support disabling reasoning (independent
of Assistant chat model reasoning)

Other improvements to our model config logic:
- Fixes bug in `onboarding/design.ts` and `assistant.eval.ts` where
`providerOptions` was being dropped
- `getModel()` now returns a bundled `modelParams` object (spread into
AI SDK calls) so `providerOptions` can't be accidentally omitted (this
[has happened
before](https://supabase.slack.com/archives/C0161K73J1J/p1771518443534309?thread_ts=1771493920.775699&cid=C0161K73J1J))
- Introduces an `ASSISTANT_MODELS` registry as a single source of truth
for assistant model config, eliminating hardcoded model IDs across the
codebase
- Aligns free/pro model conditional logic with `assistant.advance_model`
entitlement naming conventions instead of the `isLimited` pattern
- Adds `console.error` logging of Assistant stream errors so we can
interpret reasoning effort compatibility errors in the future (instead
of just opaque "Sorry, I'm having trouble responding right now" card)
- Removes unnecessary type casts and generally making the model config
logic stricter
- Removes pre-existing dead code: `anthropic` provider variant in
`GetModelParams` / `PROVIDERS` registry that was never implemented in
`getModel()`

Now if you try to select an unsupported reasoning level you get a type
error:

<img width="1306" height="320" alt="CleanShot 2026-03-20 at 14 37 24@2x"
src="https://github.com/user-attachments/assets/a6ac234b-5ea5-4d81-8e01-ac4be34a0800"
/>

And if for some reason an invalid reasoning level slips through, you now
get a server-side error surfacing the issue:

<img width="1268" height="204" alt="CleanShot 2026-03-20 at 14 58 14@2x"
src="https://github.com/user-attachments/assets/aadc1b7a-9495-475f-9741-39979bd27cd7"
/>

I've tested gpt-5 and gpt-5-mini are still working on the staging
preview and verified the models were selected properly in Braintrust
logs. Both models are available on my Pro test account, and my Free test
account shows the Pro upgrade CTA.


Closes AI-446
Closes AI-551
 2026-03-25 11:29:23 -04:00
Matt Rossman d29fbf6eb7 feat(assistant): upgrade AI SDK v5 → v6 (#43931) 
Upgrades `ai` from v5 to v6 and all related packages.

**Package bumps:**
- `ai`: `5.0.52` → `^6.0.116`
- `@ai-sdk/openai`: `2.0.32` → `^3.0.41`
- `@ai-sdk/react`: `2.0.52` → `^3.0.118`
- `@ai-sdk/provider`: `^2.0.0` → `^3.0.8`
- `@ai-sdk/provider-utils`: `^3.0.0` → `^4.0.19`
- `@ai-sdk/amazon-bedrock`: `^3.0.0` → `^4.0.81`
- `@ai-sdk/mcp`: N/A → `^1.0.25`
- `openai`: bumped to `^4.104.0`
- `braintrust`: `3.0.x` → `^3.4.0`

**Breaking change migrations:**
- `generateObject` removed in v6 — migrated 5 API routes to
`generateText` with `Output.object({ schema })`, returning
`result.output`
- `convertToModelMessages` is now async — added `await`
- MCP import path changed: `experimental_createMCPClient` from `ai` →
`createMCPClient` from `@ai-sdk/mcp`
- `openai()` defaults to Responses API — added `store: false` to
provider options for ZDR org compatibility

**Streaming fix:**
Added `Content-Encoding: none` header to `pipeUIMessageStreamToResponse`
calls. Without it, proxy middleware buffers the entire SSE response
before flushing, causing the full reply to appear at once.

**Zero Data Retention fix:**
In recent AI SDK versions, `openai()` default to Responses API instead
of the legacy chat completions API. This produces a 404 from OpenAI with
message `"Items are not persisted for Zero Data Retention organizations.
Remove this item from your input and try again."` The Responses API is
OpenAI's [recommended
endpoint](https://developers.openai.com/api/docs/guides/migrate-to-responses).

This PR adds `store: false` as mentioned in
https://github.com/vercel/ai/issues/10060 to avoid incompatible
persistence attempts.

**References:**
- https://ai-sdk.dev/docs/migration-guides/migration-guide-6-0
-
https://ai-sdk.dev/docs/troubleshooting/streaming-not-working-when-proxied
- https://github.com/vercel/ai/issues/10060

Closes AI-514
Related AI-509
 2026-03-19 09:07:33 -04:00
Saxon FletcherCharisJoshen Lim
 		1973615a42 Prompt GPT-5 Optimizations (#39031) 
* update onboarding

* update model and fix part issue

* action orientated assistant

* fix tool

* lock

* remove unused filter

* fix tests

* fix again

* update package

* update container

* fix tests

* refactor(ai assistant): break out message markdown and profile picture

* wip

* refactor(ai assistant): break up message component

* refactor: break ai assistant message down into multiple files

* add limitations prompt

* limitations prompt

* link prompt

* refactor: simplify ReportBlock state

* fix: styling of draggable report block header

When the drag handle is showing, it overlaps with the block header.
Decrease the opacity of the header so the handle can be seen and the two
can be distinguished.

* fix: minor tweaks to tool ui

* refactor: simplify DisplayBlockRenderer state

* fix: remove double deploy button in edge function block

When the confirm footer is shown, the deploy button on the top right should be
hidden (not just disabled) to avoid confusion.

* refactor, test: message sanitization by opt-in level

Refactor the message sanitization to have more type safety and be more testable.
Add tests to ensure:

- Message sanitization always runs on generate-v4
- Message sanitization correctly works by opt-in level

* Fix conflicts in pnpm lock

* Couple of nits and refactors

* Revert casing for report block snippet

* adjust sanitised prompt

* Fix tests

* prompt optimizer

* remove realtime for now

* Update apps/studio/lib/ai/prompts.ts

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

* Update apps/studio/lib/ai/prompts.ts

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

---------

Co-authored-by: Charis Lam <26616127+charislam@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
 2025-09-29 14:07:53 +10:00
Saxon Fletcher d276aca6cb Chore/ai flow model (#38805) 
* update onboarding

* update model and fix part issue

* lock

* remove unused filter

* fix tests

* fix again

* update package

* fix merge issue
 2025-09-23 12:46:20 +10:00
Saxon FletcherandJoshen Lim bf004b8a1d Choose provider and model (#38267) 
* choose provider and model

* fix test

* Fix lint

* Nit clean up

* add provider options and remove anthropic

* remove package

* use chat model and update prompt

* prompt rendering

* default to pro

* fix test

* low

* remove log

* remove redundant part.text

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
 2025-09-09 10:05:39 +00:00