dima/supabase
1
0 Fork 0
mirror of https://github.com/supabase/supabase.git synced 2026-10-06 01:45:10 +03:00
Code Issues Packages Projects Releases Wiki Activity
38,871 Commits 1,782 Branches 62 Tags
codex/fix-tanstack-e2e
Commit Graph
6 Commits
Author SHA1 Message Date
claude[bot]andClaude ea3a7743b1 feat(studio): default AI Assistant to GPT-6 Luna (AI-1245) (#50757) 
<!-- ccr-slack-attribution -->
_Requested by **Saxon Fletcher** · [Slack
thread](https://supabase.slack.com/archives/C051L8U2EJF/p1790104861710199?thread_ts=1790104861.710199&cid=C051L8U2EJF)_

## Problem

**Before:** The Assistant's base model is `gpt-5.6-luna` at medium
reasoning effort.

**After:** The base model is `gpt-6-luna`, its direct successor, still
at medium effort. It costs half as much: $0.10/$0.50 per MTok against
$0.20/$1.20.

Resolves
[AI-1245](https://linear.app/supabase/issue/AI-1245/move-the-assistants-base-model-to-gpt-6-luna).

## Solution

This swaps `gpt-5.6-luna` for `gpt-6-luna` in
`apps/studio/lib/ai/model.utils.ts`: the model ID union, the
reasoning-support map, `ASSISTANT_MODELS`,
`DEFAULT_ASSISTANT_BASE_MODEL_ID`, and the OpenAI provider registry. The
old ID is removed, not kept next to the new one. A stored selection of
`gpt-5.6-luna` is no longer a known ID, so the client and `generate-v4`
both fall back to the new default. The eval cost table (AI-1242) and
eval experiments (AI-1243) are out of scope.

Source for the model ID and supported efforts (none/low/medium
default/high/xhigh/max): [OpenAI model docs: GPT-6
Luna](https://developers.openai.com/api/docs/models/gpt-6-luna).
`@ai-sdk/openai@4.0.41` types model IDs as a union plus `string & {}`,
so no SDK bump is needed.

## Review instructions

1. Check the diff in `model.utils.ts`. Say so if you'd rather keep
`gpt-5.6-luna` selectable as a fallback.
2. AI-1245 asks for the Assistant evals before shipping. Add the
`run-evals` label to run `braintrust-evals.yml` on this PR, or run `pnpm
--filter studio evals:run` locally. They have not been run yet because
they need OpenAI/Braintrust credentials.
3. Already run: studio `typecheck`, eslint + prettier on the changed
files, vitest for `lib/ai`, `pages/api/ai` and `state/ai-assistant` (264
passed), and `evals:preflight`.

## Checklist

- [x] I have read
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
- [x] No docs topics changed

**AI disclosure:** Claude Code wrote this PR from start to finish. Saxon
Fletcher (@SaxonF) is the accountable human and must review it before
merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01W21zLTfzde6FFPyYGbGC6K


---
_Generated by [Claude
Code](https://claude.ai/code/session_01W21zLTfzde6FFPyYGbGC6K)_

Co-authored-by: Claude <noreply@anthropic.com>
 2026-09-23 10:03:27 +10:00
Saxon FletcherandCursor 9b17ce8f2c chore(studio): default assistant to GPT-5.6 Luna (#49749) 
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Feature / chore: hide assistant model selection in the UI and default
chats to GPT-5.6 Luna.

## What is the current behavior?

The assistant composer exposes a model picker. Paid orgs default to
`gpt-5.3-codex`; everyone else defaults to `gpt-5.4-nano`.

## What is the new behavior?

- The model picker is hidden in the assistant composer and Explorer
home.
- Chats default to `gpt-5.6-luna` with `reasoningEffort: medium`.
- Model selection plumbing is kept (registry, entitlements, `setModel`,
generate-v4 request body) so a requested model can still be honored when
provided.
- Other completion endpoints still use `gpt-5.4-nano`.

## Additional context

Model selector UI can be re-enabled by passing `selectedModel` /
`onSelectModel` to `AssistantChatForm`.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for the GPT-5.6 Luna model with medium reasoning
capability.
  * Made GPT-5.6 Luna the default assistant model.
* **Improvements**
* Simplified assistant chat by removing model selection from the primary
chat experience.
  * Updated model fallback behavior to use the standard assistant model.
* Chat forms can now optionally display model selection when configured.
* **Tests**
* Updated model coverage and assistant chat tests for the new defaults
and behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-09-02 10:24:37 +10:00
Matt Rossman d143571586 feat(assistant): trace-level scorers + server-side tool execution with needsApproval (#45654) 
## Motivation

When Assistant runs a potentially destructive tool like `execute_sql`,
it stops the LLM request and prompts for client-side approval and
execution of the tool. After approval, a second request kicks off under
a separate trace. This has made scoring and
[Topics](https://www.braintrust.dev/blog/topics) classification
challenging, as the generated `output` is split across stateless
requests. The [span-level
scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans)
approach we've used thusfar (after the LLM call, we massage the result
into an `output` payload that's stuck onto the root span) has been
cumbersome and led to invalid scores / topics where only part of the
assistant response is considered. It's also inefficient, as we're
duplicating potentially large info (like the `search_docs` output) that
already exists within the trace.

An alternative to scoring spans is to [score
traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces).
Braintrust [best
practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices)
advise:

> Use span scope for evaluating individual operations or outputs. Use
trace scope for evaluating multi-turn conversations, overall workflow
completion, or when your scorer needs access to the full execution
context.

We've also received [direct
guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89)
from their team to use this approach.

## Changes

Migrates eval scorers from custom `AssistantEvalOutput` shape to
trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with
thread parsing that scores the full latest Assistant turn and passes
prior conversation separately where relevant.

Moves `execute_sql` and `deploy_edge_function` from client-side
execution after approval to AI SDK `needsApproval` + server-side
`execute()`. SQL results returned to the model are gated by AI opt-in
level, so row data is only included with `schema_and_log_and_data`;
otherwise the tool returns the no-data-permissions sentinel.

Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within
an "assistant" turn due to tool call requests/responses. For online
evals, this means we should configure automations to only score traces
with `metadata.isFinalStep = true` to ensure we're judging the complete
generated response.

Other minor kaizen changes:
- Renamed `promptProviderOptions` to `systemProviderOptions` to clarify
that this is associated with the "system" message and disambiguate from
the root `providerOptions`
- Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown`
span shapes from Braintrust, to more easily access typed inputs/output
on tool spans.
- Bumps AI SDK floor version `^6.0.116` → `^6.0.174`
- Tweaked the "Conciseness" scorer to not unfairly dock points for the
new `[called tool_name]` labels in serialized assistant response

## Verification

In the studio staging build, I asked Assistant to create a todos table
with 3 sample todos. I manually approved the `execute_sql` call and saw
Assistant generate text before & after the call.

In Braintrust I verified two traces were produced (see [filtered
logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})),
the first with `metadata.isFinalStep = false` and the second with
`metadata.isFinalStep = true`.

In the Braintrust staging scorers, I ran the preview Completeness scorer
on the second trace and verified it sees the complete Assistant response
including markers for tool calls ([link to
trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82))

<img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x"
src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a"
/>

I also tested the `deploy_edge_function` workflow and verified it still
prompts for permission and warns on deployment of existing functions.

**References**
- https://www.braintrust.dev/docs/evaluate/custom-code#score-traces
-
https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval

Supercedes https://github.com/supabase/supabase/pull/45556 and
https://github.com/supabase/supabase/pull/45339

Closes AI-473

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Tool actions (SQL execution, edge-function deploy) now require
explicit user Approve/Deny before proceeding.

* **Improvements**
* Assistant pauses for approval responses before sending follow-ups,
giving clearer control over risky actions.
  * Deploy/replace flows show confirmation and clearer replace warnings.
* Evaluation/scoring updated to use richer trace data for more accurate
assistant performance signals.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-12 15:24:21 -04:00
Matt Rossman 0c5f64fcba feat(assistant): upgrade default models to gpt-5.4-nano and gpt-5.3-codex (#44107) 
Replaces `gpt-5-mini` and `gpt-5` with `gpt-5.4-nano` and
`gpt-5.3-codex` respectively. Clients with stale model IDs in IndexedDB
will gracefully reset to the new defaults. While we can technically keep
the existing models around, we've
[opted](https://supabase.slack.com/archives/C051L8U2EJF/p1774283070517609?thread_ts=1773771991.871669&cid=C051L8U2EJF)
to replace them w/ the newer models for simplicity. Basic completion
endpoints use `'none'` reasoning level for optimal speed.

Rationale for these models is they provide they best balance of
intelligence/speed and cost. GPT-5.4-nano is less expensive (0.8x
price), faster, and smarter than GPT-5-mini. GPT-5.4-mini would be even
smarter but is 3x the price. GPT-5.3-Codex is ~1.4x the price of GPT-5,
while GPT-5.4 would be 2x price, but 5.3-Codex is still a big
intelligence boost from GPT-5.

See [eval
comparison](https://www.braintrust.dev/app/supabase.io/p/Assistant/experiments/mattrossman%2Fai-509-v2-upgrade-assistant-models-beyond-gpt-5-family-1774468619?c=master-1774458837&diff=between_experiments),
scores are relatively stable and conciseness naturally improves on
gpt-5.4-nano.

Other change:
- Fixed an eval test case to clarify that https://supabase.help is also
a correct URL for submitting support ticket, which was unfairly scored
as incorrect
[here](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=5244cccd-23b2-4f79-9dd2-287f1b40ebad&r=bac9b903-8bde-4c21-99dd-e0ed141c4f9e&s=f248fbf5-75bf-4aab-be0a-87a4298e6d11)

I sanity checked the Assistant, natural language filters, and SQL Editor
completions on staging preview.

References:
- https://openai.com/index/introducing-gpt-5-4-mini-and-nano/
- https://openai.com/index/introducing-gpt-5-3-codex/
- https://developers.openai.com/api/docs/pricing

Closes AI-509
 2026-03-26 14:35:54 +08:00
Matt Rossman adf8b0c67c feat(assistant): per-endpoint reasoningEffort + model config cleanup (#43981) 
We're exploring support for newer models like
[gpt-5.4-nano](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/)
in Assistant. This model doesn't support the `'minimal'` reasoning
effort level we use for gpt-5-mini which leads to vague errors.

<img width="595" height="263" alt="CleanShot 2026-03-18 at 17 13 05@2x"
src="https://github.com/user-attachments/assets/cf7c2370-322d-4a8a-be55-23e680db0aa0"
/>


Also, we've [previously
discussed](https://supabase.slack.com/archives/C0161K73J1J/p1771544464850199?thread_ts=1771493920.775699&cid=C0161K73J1J)
that reasoning adds unnecessary latency to otherwise simple AI
completion endpoints like `title-v2`. We want more control of reasoning
level independent of model/endpoint.

This PR aims to solve both problems by:
- making reasoning effort configurable on a per-request basis
- adding compile-time guardrails to prevent selecting an incompatible
reasoning level for models
- adding a `DEFAULT_COMPLETION_MODEL` with minimal reasoning that we can
update with newer models that support disabling reasoning (independent
of Assistant chat model reasoning)

Other improvements to our model config logic:
- Fixes bug in `onboarding/design.ts` and `assistant.eval.ts` where
`providerOptions` was being dropped
- `getModel()` now returns a bundled `modelParams` object (spread into
AI SDK calls) so `providerOptions` can't be accidentally omitted (this
[has happened
before](https://supabase.slack.com/archives/C0161K73J1J/p1771518443534309?thread_ts=1771493920.775699&cid=C0161K73J1J))
- Introduces an `ASSISTANT_MODELS` registry as a single source of truth
for assistant model config, eliminating hardcoded model IDs across the
codebase
- Aligns free/pro model conditional logic with `assistant.advance_model`
entitlement naming conventions instead of the `isLimited` pattern
- Adds `console.error` logging of Assistant stream errors so we can
interpret reasoning effort compatibility errors in the future (instead
of just opaque "Sorry, I'm having trouble responding right now" card)
- Removes unnecessary type casts and generally making the model config
logic stricter
- Removes pre-existing dead code: `anthropic` provider variant in
`GetModelParams` / `PROVIDERS` registry that was never implemented in
`getModel()`

Now if you try to select an unsupported reasoning level you get a type
error:

<img width="1306" height="320" alt="CleanShot 2026-03-20 at 14 37 24@2x"
src="https://github.com/user-attachments/assets/a6ac234b-5ea5-4d81-8e01-ac4be34a0800"
/>

And if for some reason an invalid reasoning level slips through, you now
get a server-side error surfacing the issue:

<img width="1268" height="204" alt="CleanShot 2026-03-20 at 14 58 14@2x"
src="https://github.com/user-attachments/assets/aadc1b7a-9495-475f-9741-39979bd27cd7"
/>

I've tested gpt-5 and gpt-5-mini are still working on the staging
preview and verified the models were selected properly in Braintrust
logs. Both models are available on my Pro test account, and my Free test
account shows the Pro upgrade CTA.


Closes AI-446
Closes AI-551
 2026-03-25 11:29:23 -04:00
Jordi Enric ac64a902c1 chore: adds tests (#42653) 2026-02-11 09:50:11 +01:00