Adding broad RLS policies to public buckets can cause users to expose
more than they expected, like the ability to list all profile pictures
on an app. This patches Assistant with knowledge to follow our latest
guidance on restrictive RLS policies for storage buckets
https://github.com/supabase/supabase/pull/46172
**Changes**
- Adds Storage bucket evals for public website assets and avatar access
patterns to distinguish public vs private bucket use cases
- Adds eval for overly permissive table policies
- Adds `storage` knowledge so Assistant distinguishes public buckets,
private buckets, object reads, and object listing.
- Adds `includeToolCallInputs` option for scorer transcripts so LLM
judges can evaluate proposed SQL/tool actions.
- Bumps max step count to 10 since storage knowledge may incur another
tool call (also 10 is recommended
[here](https://vercel.com/academy/ai-sdk/multi-step-and-generative-ui#why-multi-step-is-required)
for complex multi-tool scenarios)
**References**
-
https://supabase.com/docs/guides/storage/buckets/fundamentals#public-buckets
- https://supabase.com/docs/guides/storage/security/access-control
- https://github.com/supabase/supabase/pull/46172
**Notes:**
- These prompt tweaks are not meant to be exhaustive fixes, they are
mainly hotfixes intended to hold us out until these cases can be
addressed more deeply in skills/docs and tracked in a central evals
Closes AI-676
Closes AI-756
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added Storage knowledge resource for the assistant covering Supabase
Storage access patterns and RLS guidance.
* Added three evaluation cases: two for Storage (marketing assets,
avatars) and one for RLS policy generation for user profiles.
* **Improvements**
* Evaluators now include tool call inputs when judging conversations.
* Assistant prompts and generation enhanced with richer Storage/RLS
guidance and extended streaming limits.
* **Tests**
* Added test ensuring tool call inputs are included in serialized thread
context.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46168?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
When Assistant tries to schedule crons in read-only mode, it succeeds
but creates the jobs under the `supabase_read_only_user`. This causes
permission errors when user try to delete or unschedule them from the
Cron dashboard. The root fix will be to enforce read-only transactions
for that user. In the meantime, this PR steers Assistant to avoid the
mistake.
**Changes**
- Prompts `execute_sql` to treat side-effecting function calls such as
`cron.schedule()` as write queries.
- Adds tool input assertions for "Tool Usage" scorer and a focused cron
regression eval.
- Updates eval mocks to show pg_cron extension as installed so it can
call `cron.schedule()`
**Verification**
See [this
trace](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=4a9e8c0e-83b7-4555-8502-365662c3ec8e&r=e041e69b-b70f-41d1-b88c-e8f7888c3de5&s=e041e69b-b70f-41d1-b88c-e8f7888c3de5)
from Braintrust where the new eval passes "Tool Usage", correctly using
`isWriteQuery` for the `cron.schedule()`
Closes AI-737
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tool evaluation now validates tool inputs (including exact and
substring matches) in addition to tool presence.
* **Tests**
* Added a test confirming cron-scheduling behavior and that SQL
scheduling/enqueue calls are treated as write operations.
* **Chores**
* Added pg_cron to mock extension data.
* Clarified description that SQL calls with side effects should be
treated as writes.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/45997?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Adds prompt guardrails and evals to prevent the AI assistant from asking
users to share sensitive data (API keys, `.env` contents, etc.) and to
warn when credentials are shared.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Stronger safety behavior: assistant now refuses requests to share full
environment files, asks for variable names only, and directs users to
secure secret-management tooling.
* Immediate warning and guidance if credentials or other sensitive
values are pasted in chat, without repeating exposed secrets.
* **Behavior**
* Clarified evaluation rules so responses more consistently follow the
new safety guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Prevents the AI assistant from helping with local git/filesystem
operations, and adds explicit warnings before irreversible database
operations (DROP TABLE, DELETE without WHERE, etc.).
Adds a `safetyScorer` and eval cases to cover these behaviours.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Safety metric to evaluations so assistant responses are scored
for safe handling of destructive or risky requests
* Assistant guidance updated to refuse destructive local VCS/filesystem
actions and require clear warnings for irreversible database operations
* **Tests**
* Added evaluation cases covering safe refusals, clear warnings, and
correct handling of destructive or risky prompts
* **Chores**
* Enabled Safety metric in online evaluation manifests/handlers
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Updates the RLS knowledge loaded by the dashboard AI assistant to
explain the new secure-by-default functionality.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Clarified PostgreSQL/RLS guidance in Studio: tables are now "secure by
default"—SQL-created tables aren’t exposed via the Data API unless
explicit grants are given to anon/authenticated/service_role and RLS is
enabled; added an “Exposing a Table to the Data API” workflow,
strengthened RLS prerequisites in best practices, and improved
troubleshooting/error-recovery guidance.
* **Tests / Evaluations**
* Added an evaluation case validating guidance for non-RLS tables
requiring explicit grants and RLS policies.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Ali Waseem <waseema393@gmail.com>
Moves knowledge (RLS, Edge Functions, PostgreSQL best practices,
Realtime) out of the static system prompt and into a `load_knowledge`
tool the model calls on demand, reducing prompt bloat. This is a
temporary stopgap until the [standard Supabase
agent-skills](https://github.com/supabase/agent-skills) are ready for
integration in Assistant.
- New always-available `load_knowledge` tool added to
`rendering-tools.ts`
- Updated `Message.Parts.tsx` so the "Ran load_knowledge" chip renders
in chat
- System prompt replaces the four knowledge blobs with an `## Available
Knowledge` block and is hardened to load knowledge for given topics
- New "Knowledge Usage" scorer and `requiredKnowledge` assertions check
that knowledge loads as expected in test scenarios
- Filters GraphQL error responses out of `output.docs` before
faithfulness scoring to reduce noise
See "Knowledge Usage" scoring 100% in evals with no major regressions:
https://github.com/supabase/supabase/pull/44296#issuecomment-4145760236
Sample trace showing the tool in action
([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=project_logs&object_id=5a8d02e5-b3b6-40cc-ba76-ecee286478f4&r=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1&s=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1))
<img width="2192" height="1730" alt="CleanShot 2026-03-30 at 13 53
59@2x"
src="https://github.com/user-attachments/assets/f483767c-34e0-401c-8089-5b9834fe696a"
/>
**References**
- https://ai-sdk.dev/cookbook/guides/agent-skills
Closes AI-508
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added dynamic knowledge loading capability enabling the AI assistant
to retrieve on-demand information about PostgreSQL best practices, Row
Level Security, Edge Functions, and Realtime.
* **Bug Fixes**
* Improved search results filtering to exclude error responses in tool
outputs.
* **Tests**
* Enhanced evaluation metrics with knowledge usage scoring.
* Expanded test dataset cases to validate knowledge requirement
handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Logic changes**
- Adds function in `helpers.ts` to extract URLs from text via regex
- I also considering using a library like
[linkify-it](https://www.npmjs.com/package/linkify-it) for this but
figured it's not worth the extra dep
- Adds associated tests in `helpers.test.ts`
- Adds "URL Validity" scorer which performs a HEAD request for links in
Assistant response text and determins what portion of links have `.ok`
responses
- Adds eval case to check correctness of support ticket URL answers
**Prompt changes**
- Informs Assistant of https://supabase.com/dashboard/support/new being
the URL to create support tickets
- Encourages Assistant to "self-debug" issues before directing users to
create support tickets
See [Eval
Report](https://github.com/supabase/supabase/pull/42227#issuecomment-3807772871)
and
[Correctness](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=1ad0f9b0-5adb-436c-9812-a87aac62c036&r=1ef13459-a98c-4904-925e-6d81276cebb2&s=dbe5c607-a560-462b-8745-41d430744431)
analysis for new support ticket test case.
Resolves AI-384
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added URL validity scoring to evaluations and helper utilities for
extracting/cleaning URLs.
* Added evaluation cases for support-ticket URL handling and OAuth
callback guidance.
* **Documentation**
* Updated assistant guidance to prefer self-resolution, include
support-ticket direction, clarified data-recovery search steps, and
added template-URL notation.
* **Tests**
* Expanded URL extraction and related utility tests to cover many
formats and edge cases.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* feat: SQL correctness scorer, override mock tables
* feat: replace "SQL Correctness" with "SQL Identifier Quoting" scorer
* fix(prompt): discourage simulating confirmation of execute_sql tool
this is already handled at the UI layer
* fix(prompt): encourage quotes on identifiers with caps
* feat: move extractIdentifiers to own file, add tests
* chore: shorten tests
* feat: extract ColumnDef column names in extractIdentifiers
* refactor: sqlIdentifierQuotingScorer with more thorough checks
* refactor: consolidate into `sql-identifier-quoting.ts`
* feat: support mocking schemas, eval test case with case sensitive schema
* fix: test cases that don't match default mock schema
* chore: format
* feat(prompt): mention special characters and reserved words
* feat: optional description in metadata, test with special characters
* feat: consolidated comprehensive test case
* fix(prompt): revert conflicting instruction
Matt Rossman
