dima/supabase
1
0 Fork 0
mirror of https://github.com/supabase/supabase.git synced 2026-10-16 06:39:09 +03:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into docs/billing-revamp

Browse Source
This commit is contained in:
Kevin Grüneberg committed 2025-02-12 11:45:09 +08:00
2 parents 0af1639631 25a4424869
commit e80c59c2dd
13 files changed
+1010 -92
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts
+4
View File
@@ -1747,6 +1747,10 @@ export const ai = {
url: '/guides/ai/vector-indexes',
items: vectorIndexItems,
},
{
name: 'Automatic embeddings',
url: '/guides/ai/automatic-embeddings',
},
{
name: 'Engineering for scale',
url: '/guides/ai/engineering-for-scale',
apps/docs/content/guides/ai/automatic-embeddings.mdx
+817
View File
@@ -0,0 +1,817 @@
---
id: 'ai-automatic-embeddings'
title: 'Automatic embeddings'
description: 'Automate embedding generation and updates in Postgres'
sidebar_label: 'Automatic embeddings'
---
Vector embeddings enable powerful [semantic search](/docs/guides/ai/semantic-search) capabilities in Postgres, but managing them alongside your content has traditionally been complex. This guide demonstrates how to automate embedding generation and updates using Supabase [Edge Functions](/docs/guides/functions), [pgmq](/docs/guides/database/extensions/pgmq), [pg_net](/docs/guides/database/extensions/pg_net), and [pg_cron](/docs/guides/cron).
## Understanding the challenge
When implementing semantic search with pgvector, developers typically need to:
1. Generate embeddings via an external API (like OpenAI)
2. Store these embeddings alongside the content
3. Keep embeddings in sync when content changes
4. Handle failures and retries in the embedding generation process
While Postgres [full-text search](/docs/guides/database/full-text-search) can handle this internally through synchronous calls to `to_tsvector` and [triggers](https://www.postgresql.org/docs/current/textsearch-features.html#TEXTSEARCH-UPDATE-TRIGGERS), semantic search requires asynchronous API calls to a provider like OpenAI to generate vector embeddings. This guide demonstrates how to use triggers, queues, and Supabase Edge Functions to bridge this gap.
## Understanding the architecture
We'll leverage the following Postgres and Supabase features to create the automated embedding system:
1. [pgvector](/docs/guides/database/extensions/pgvector): Stores and queries vector embeddings
1. [pgmq](/docs/guides/queues): Queues embedding generation requests for processing and retries
1. [pg_net](/docs/guides/database/extensions/pg_net): Handles asynchronous HTTP requests to Edge Functions directly from Postgres
1. [pg_cron](/docs/guides/cron): Automatically processes and retries embedding generations
1. [Triggers](/docs/guides/database/postgres/triggers): Detects content changes and enqueues embedding generation requests
1. [Edge Functions](/docs/guides/functions): Generates embeddings via an API like OpenAI (customizable)
We'll design the system to:
1. Be generic, so that it can be used with any table and content. This allows you to configure embeddings in multiple places, each with the ability to customize the input used for embedding generation. These will all use the same queue infrastructure and Edge Function to generate the embeddings.
1. Handle failures gracefully, by retrying failed jobs and providing detailed information about the status of each job.
## Implementation
We'll start by setting up the infrastructure needed to queue and process embedding generation requests. Then we'll create an example table with triggers to enqueue these embedding requests whenever content is inserted or updated.
### Step 1: Enable extensions
First, let's enable the required extensions:
<Tabs
scrollable
size="small"
type="underlined"
defaultActiveId="sql"
queryGroup="database-method"
>
<TabPanel id="sql" label="SQL">
```sql
-- For vector operations
create extension if not exists vector
with
schema extensions;
-- For queueing and processing jobs
-- (pgmq will create its own schema)
create extension if not exists pgmq;
-- For async HTTP requests
create extension if not exists pg_net
with
schema extensions;
-- For scheduled processing and retries
-- (pg_cron will create its own schema)
create extension if not exists pg_cron;
-- For clearing embeddings during updates
create extension if not exists hstore
with
schema extensions;
```
Even though the SQL code is `create extension`, this is the equivalent of "enabling the extension".
To disable an extension, call `drop extension`.
</TabPanel>
<TabPanel id="dashboard" label="Dashboard">
1. Go to the [Extensions](/dashboard/project/_/database/extensions) page in the Dashboard.
2. Search for and enable the following extensions:
- `vector`
- `pgmq`
- `pg_net`
- `pg_cron`
- `hstore`
</TabPanel>
</Tabs>
### Step 2: Create utility functions
Before we set up our embedding logic, we need to create some utility functions:
```sql
-- Schema for utility functions
create schema util;
-- Utility function to get the Supabase project URL (required for Edge Functions)
create function util.project_url()
returns text
language plpgsql
security definer
as $$
declare
secret_value text;
begin
-- Retrieve the project URL from Vault
select decrypted_secret into secret_value from vault.decrypted_secrets where name = 'project_url';
return secret_value;
end;
$$;
-- Generic function to invoke any Edge Function
create or replace function util.invoke_edge_function(
name text,
body jsonb,
timeout_milliseconds int = 5 * 60 * 1000 -- default 5 minute timeout
)
returns void
language plpgsql
as $$
declare
headers_raw text;
auth_header text;
begin
-- If we're in a PostgREST session, reuse the request headers for authorization
headers_raw := current_setting('request.headers', true);
-- Only try to parse if headers are present
auth_header := case
when headers_raw is not null then
(headers_raw::json->>'authorization')
else
null
end;
-- Perform async HTTP request to the edge function
perform net.http_post(
url => util.project_url() || '/functions/v1/' || name,
headers => jsonb_build_object(
'Content-Type', 'application/json',
'Authorization', auth_header
),
body => body,
timeout_milliseconds => timeout_milliseconds
);
end;
$$;
-- Generic trigger function to clear a column on update
create or replace function util.clear_column()
returns trigger
language plpgsql as $$
declare
clear_column text := TG_ARGV[0];
begin
NEW := NEW #= hstore(clear_column, NULL);
return NEW;
end;
$$;
```
Here we create:
- A schema `util` to store utility functions.
- A function to retrieve the Supabase project URL from [Vault](/docs/guides/database/vault). We'll add this secret next.
- A generic function to invoke any Edge Function with a given name and request body.
- A generic trigger function to clear a column on update. This function accepts the column name as an argument and sets it to `NULL` in the `NEW` record. We'll explain how to use this function later.
Every project has a unique API URL that is required to invoke Edge Functions. Let's go ahead and add the project URL secret to Vault depending on your environment.
When working with a local Supabase stack, add the following to your `supabase/seed.sql` file:
```sql
select
vault.create_secret('http://api.supabase.internal:8000', 'project_url');
```
When deploying to the cloud platform, open the [SQL editor](/dashboard/project/_/sql/new) and run the following, replacing `<project-url>` with your [project's API URL](/dashboard/project/_/settings/api):
```sql
select
vault.create_secret('<project-url>', 'project_url');
```
### Step 3: Create queue and triggers
Our goal is to automatically generate embeddings whenever content is inserted or updated within a table. We can use triggers and queues to achieve this. Our approach is to automatically queue embedding jobs whenever records are inserted or updated in a table, then process them asynchronously using a cron job. If a job fails, it will remain in the queue and be retried in the next scheduled task.
First we create a `pgmq` queue for processing embedding requests:
```sql
-- Queue for processing embedding jobs
select pgmq.create('embedding_jobs');
```
Next we create a trigger function to queue embedding jobs. We'll use this function to handle both insert and update events:
```sql
-- Generic trigger function to queue embedding jobs
create or replace function util.queue_embeddings()
returns trigger
language plpgsql
as $$
declare
content_function text = TG_ARGV[0];
embedding_column text = TG_ARGV[1];
begin
perform pgmq.send(
queue_name => 'embedding_jobs',
msg => jsonb_build_object(
'id', NEW.id,
'schema', TG_TABLE_SCHEMA,
'table', TG_TABLE_NAME,
'contentFunction', content_function,
'embeddingColumn', embedding_column
)
);
return NEW;
end;
$$;
```
Our `util.queue_embeddings` trigger function is generic and can be used with any table and content function. It accepts two arguments:
1. `content_function`: The name of a function that returns the text content to be embedded. The function should accept a single row as input and return text (see the `embedding_input` example).
This allows you to customize the text input passed to the embedding model - for example, you could concatenate multiple columns together like `title` and `content` and use the result as input.
2. `embedding_column`: The name of the destination column where the embedding will be stored.
Note that the `util.queue_embeddings` trigger function requires a `for each row` clause to work correctly. See [Usage](#usage) for an example of how to use this trigger function with your table.
Next we'll create a function to process the embedding jobs. This function will read jobs from the queue, group them into batches, and invoke the Edge Function to generate embeddings. We'll use `pg_cron` to schedule this function to run every 10 seconds.
```sql
-- Function to process embedding jobs from the queue
create or replace function util.process_embeddings(
batch_size int = 10,
max_requests int = 10,
timeout_milliseconds int = 5 * 60 * 1000 -- default 5 minute timeout
)
returns void
language plpgsql
as $$
declare
job_batches jsonb[];
batch jsonb;
begin
with
-- First get jobs and assign batch numbers
numbered_jobs as (
select
message || jsonb_build_object('jobId', msg_id) as job_info,
(row_number() over (order by 1) - 1) / batch_size as batch_num
from pgmq.read(
queue_name => 'embedding_jobs',
vt => timeout_milliseconds / 1000,
qty => max_requests * batch_size
)
),
-- Then group jobs into batches
batched_jobs as (
select
jsonb_agg(job_info) as batch_array,
batch_num
from numbered_jobs
group by batch_num
)
-- Finally aggregate all batches into array
select array_agg(batch_array)
from batched_jobs
into job_batches;
-- Invoke the embed edge function for each batch
foreach batch in array job_batches loop
perform util.invoke_edge_function(
name => 'embed',
body => batch,
timeout_milliseconds => timeout_milliseconds
);
end loop;
end;
$$;
-- Schedule the embedding processing
select
cron.schedule(
'process-embeddings',
'10 seconds',
$$
select util.process_embeddings();
$$
);
```
Let's discuss some common questions about this approach:
#### Why not generate all embeddings in a single Edge Function request?
While this is possible, it can lead to long processing times and potential timeouts. Batching allows us to process multiple embeddings concurrently and handle failures more effectively.
#### Why not one request per row?
This approach can lead to API rate limiting and performance issues. Batching provides a balance between efficiency and reliability.
#### Why queue requests instead of processing them immediately?
Queuing allows us to handle failures gracefully, retry requests, and manage concurrency more effectively. Specifically we are using `pgmq`'s visibility timeouts to ensure that failed requests are retried.
#### How do visibility timeouts work?
Every time we read a message from the queue, we set a visibility timeout which tells `pgmq` to hide the message from other readers for a certain period. If the Edge Function fails to process the message within this period, the message becomes visible again and will be retried by the next scheduled task.
#### How do we handle retries?
We use `pg_cron` to schedule a task that reads messages from the queue and processes them. If the Edge Function fails to process a message, it becomes visible again after a timeout and can be retried by the next scheduled task.
#### Is 10 seconds a good interval for processing?
This interval is a good starting point, but you may need to adjust it based on your workload and the time it takes to generate embeddings. You can adjust the `batch_size`, `max_requests`, and `timeout_milliseconds` parameters to optimize performance.
### Step 4: Create the Edge Function
Finally we'll create the Edge Function to generate embeddings. We'll use OpenAI's API in this example, but you can replace it with any other embedding generation service.
Use the Supabase CLI to create a new Edge Function:
```bash
supabase functions new embed
```
This will create a new directory `supabase/functions/embed` with an `index.ts` file. Replace the contents of this file with the following:
_supabase/functions/embed/index.ts_:
```typescript
// Setup type definitions for built-in Supabase Runtime APIs
import 'jsr:@supabase/functions-js/edge-runtime.d.ts'
// We'll use the OpenAI API to generate embeddings
import OpenAI from 'jsr:@openai/openai'
import { z } from 'npm:zod'
// We'll make a direct Postgres connection to update the document
import postgres from 'https://deno.land/x/postgresjs@v3.4.5/mod.js'
// Initialize OpenAI client
const openai = new OpenAI({
// We'll need to manually set the `OPENAI_API_KEY` environment variable
apiKey: Deno.env.get('OPENAI_API_KEY'),
})
// Initialize Postgres client
const sql = postgres(
// `SUPABASE_DB_URL` is a built-in environment variable
Deno.env.get('SUPABASE_DB_URL')!
)
const jobSchema = z.object({
jobId: z.number(),
id: z.number(),
schema: z.string(),
table: z.string(),
contentFunction: z.string(),
embeddingColumn: z.string(),
})
const failedJobSchema = jobSchema.extend({
error: z.string(),
})
type Job = z.infer<typeof jobSchema>
type FailedJob = z.infer<typeof failedJobSchema>
type Row = {
id: string
content: unknown
}
const QUEUE_NAME = 'embedding_jobs'
// Listen for HTTP requests
Deno.serve(async (req) => {
if (req.method !== 'POST') {
return new Response('expected POST request', { status: 405 })
}
if (req.headers.get('content-type') !== 'application/json') {
return new Response('expected json body', { status: 400 })
}
// Use Zod to parse and validate the request body
const parseResult = z.array(jobSchema).safeParse(await req.json())
if (parseResult.error) {
return new Response(`invalid request body: ${parseResult.error.message}`, {
status: 400,
})
}
const pendingJobs = parseResult.data
// Track jobs that completed successfully
const completedJobs: Job[] = []
// Track jobs that failed due to an error
const failedJobs: FailedJob[] = []
async function processJobs() {
let currentJob: Job | undefined
while ((currentJob = pendingJobs.shift()) !== undefined) {
try {
await processJob(currentJob)
completedJobs.push(currentJob)
} catch (error) {
failedJobs.push({
...currentJob,
error: error instanceof Error ? error.message : JSON.stringify(error),
})
}
}
}
try {
// Process jobs while listening for worker termination
await Promise.race([processJobs(), catchUnload()])
} catch (error) {
// If the worker is terminating (e.g. wall clock limit reached),
// add pending jobs to fail list with termination reason
failedJobs.push(
...pendingJobs.map((job) => ({
...job,
error: error instanceof Error ? error.message : JSON.stringify(error),
}))
)
}
// Log completed and failed jobs for traceability
console.log('finished processing jobs:', {
completedJobs: completedJobs.length,
failedJobs: failedJobs.length,
})
return new Response(
JSON.stringify({
completedJobs,
failedJobs,
}),
{
// 200 OK response
status: 200,
// Custom headers to report job status
headers: {
'Content-Type': 'application/json',
'X-Completed-Jobs': completedJobs.length.toString(),
'X-Failed-Jobs': failedJobs.length.toString(),
},
}
)
})
/**
* Generates an embedding for the given text.
*/
async function generateEmbedding(text: string) {
const response = await openai.embeddings.create({
model: 'text-embedding-3-small',
input: text,
})
const [data] = response.data
if (!data) {
throw new Error('failed to generate embedding')
}
return data.embedding
}
/**
* Processes an embedding job.
*/
async function processJob(job: Job) {
const { jobId, id, schema, table, contentFunction, embeddingColumn } = job
// Fetch content for the schema/table/row combination
const [row]: [Row] = await sql`
select
id,
${sql(contentFunction)}(t) as content
from
${sql(schema)}.${sql(table)} t
where
id = ${id}
`
if (!row) {
throw new Error(`row not found: ${schema}.${table}/${id}`)
}
if (typeof row.content !== 'string') {
throw new Error(`invalid content - expected string: ${schema}.${table}/${id}`)
}
const embedding = await generateEmbedding(row.content)
await sql`
update
${sql(schema)}.${sql(table)}
set
${sql(embeddingColumn)} = ${JSON.stringify(embedding)}
where
id = ${id}
`
await sql`
select pgmq.delete(${QUEUE_NAME}, ${jobId}::bigint)
`
}
/**
* Returns a promise that rejects if the worker is terminating.
*/
function catchUnload() {
return new Promise((reject) => {
addEventListener('beforeunload', (ev: any) => {
reject(new Error(ev.detail?.reason))
})
})
}
```
The Edge Function listens for incoming HTTP requests from `pg_net` and processes each embedding job. It is a generic worker that can handle embedding jobs for any table and column. It uses OpenAI's API to generate embeddings and updates the corresponding row in the database. It also deletes the job from the queue once it has been processed.
The function is designed to process multiple jobs independently. If one job fails, it will not affect the processing of other jobs. The function returns a `200 OK` response with a list of completed and failed jobs. We can use this information to diagnose failed jobs. See [Troubleshooting](#troubleshooting) for more details.
You will need to set the `OPENAI_API_KEY` environment variable to authenticate with OpenAI. When running the Edge Function locally, you can add it to a `.env` file:
_.env_:
```
OPENAI_API_KEY=your-api-key
```
When you're ready to deploy the Edge Function, set can set the environment variable using the Supabase CLI:
```shell
supabase secrets set --env-file .env
```
or
```shell
supabase secrets set OPENAI_API_KEY=<your-api-key>
```
Alternatively, you can replace the `generateEmbedding` function with your own embedding generation logic.
See [Deploy to Production](/docs/guides/functions/deploy) for more information on how to deploy the Edge Function.
## Usage
Now that the infrastructure is in place, let's go through an example of how to use this system to automatically generate embeddings for a table of documents. You can use this approach with multiple tables and customize the input for each embedding generation as needed.
### 1. Create table to store documents with embeddings
We'll set up a new `documents` table that will store our content and embeddings:
```sql
-- Table to store documents with embeddings
create table documents (
id integer primary key generated always as identity,
title text not null,
content text not null,
embedding halfvec,
created_at timestamp with time zone default now()
);
-- Index for vector search over document embeddings
create index on documents using hnsw (embedding halfvec_cosine_ops);
```
Our `documents` table stores the title and content of each document along with its vector embedding. We use a `halfvec` column to store the embeddings, which is a `pgvector` data type that stores float values in half precision (16 bits) to save space.
We use an [HNSW index](/docs/guides/ai/vector-indexes/hnsw-indexes) on the vector column. Note that we are choosing `halfvec_cosine_ops` as the index method, which means our future queries will need to use cosine distance (`<=>`) to find similar embeddings. Also note that HNSW indexes support a maximum of 4000 dimensions for `halfvec` vectors, so keep this in mind when choosing an embedding model. If your model generates embeddings with more than 4000 dimensions, you will need to reduce the dimensionality before indexing them. See [Matryoshka embeddings](/blog/matryoshka-embeddings) for a potential solution to shortening dimensions.
Also note that the table must have a primary key column named `id` for our triggers to work correctly with the `util.queue_embeddings` function and for our Edge Function to update the correct row.
### 2. Create triggers to enqueue embedding jobs
Now we'll set up the triggers to enqueue embedding jobs whenever content is inserted or updated:
```sql
-- Customize the input for embedding generation
-- e.g. Concatenate title and content with a markdown header
create or replace function embedding_input(doc documents)
returns text
language plpgsql
immutable
as $$
begin
return '# ' || doc.title || E'\n\n' || doc.content;
end;
$$;
-- Trigger for insert events
create trigger embed_documents_on_insert
after insert
on documents
for each row
execute function util.queue_embeddings('embedding_input', 'embedding');
-- Trigger for update events
create trigger embed_documents_on_update
after update of title, content -- must match the columns in embedding_input()
on documents
for each row
execute function util.queue_embeddings('embedding_input', 'embedding');
```
We create 2 triggers:
1. `embed_documents_on_insert`: Enqueues embedding jobs whenever new rows are inserted into the `documents` table.
2. `embed_documents_on_update`: Enqueues embedding jobs whenever the `title` or `content` columns are updated in the `documents` table.
Both of these triggers use the same `util.queue_embeddings` function that will queue the embedding jobs for processing. They accept 2 arguments:
1. `embedding_input`: The name of the function that generates the input for embedding generation. This function allows you to customize the text input passed to the embedding model (e.g. concatenating the title and content). The function should accept a single row as input and return text.
2. `embedding`: The name of the destination column where the embedding will be stored.
Note that the update trigger only fires when the `title` or `content` columns are updated. This is to avoid unnecessary updates to the embedding column when other columns are updated. Make sure that these columns match the columns used in the `embedding_input` function.
{/* supa-mdx-lint-disable-next-line Rule001HeadingCase */}
#### (Optional) Clearing embeddings on update
Note that our trigger will enqueue new embedding jobs when content is updated, but it will not clear any existing embeddings. This means that an embedding can be temporarily out of sync with the content until the new embedding is generated and updated.
If it is more important to have _accurate_ embeddings than _any_ embedding, you can add another trigger to clear the existing embedding until the new one is generated:
```sql
-- Trigger to clear the embedding column on update
create trigger clear_document_embedding_on_update
before update of title, content -- must match the columns in embedding_input()
on documents
for each row
execute function util.clear_column('embedding');
```
`util.clear_column` is a generic trigger function we created earlier that can be used to clear any column in a table.
- It accepts the column name as an argument. This column must be nullable.
- It requires a `before` trigger with a `for each row` clause.
- It requires the `hstore` extension we created earlier.
This example will clear the `embedding` column whenever the `title` or `content` columns are updated (note the `of title, content` clause). This ensures that the embedding is always in sync with the title and content, but it will result in temporary gaps in search results until the new embedding is generated.
We intentionally use a `before` trigger because it allows us to modify the record before it's written to disk, avoiding an extra `update` statement that would be needed with an `after` trigger.
### 3. Insert and update documents
Let's insert a new document and update its content to see the embedding generation in action:
```sql
-- Insert a new document
insert into documents (title, content)
values
('Understanding Vector Databases', 'Vector databases are specialized...');
-- Immediately check the embedding column
select id, embedding
from documents
where title = 'Understanding Vector Databases';
```
You should observe that the `embedding` column is initially `null` after inserting the document. This is because the embedding generation is asynchronous and will be processed by the Edge Function in the next scheduled task.
Wait up to 10 seconds for the next task to run, then check the `embedding` column again:
```sql
select id, embedding
from documents
where title = 'Understanding Vector Databases';
```
You should see the generated embedding for the document.
Next let's update the content of the document:
```sql
-- Update the content of the document
update documents
set content = 'Vector databases allow you to query...'
where title = 'Understanding Vector Databases';
-- Immediately check the embedding column
select id, embedding
from documents
where title = 'Understanding Vector Databases';
```
You should observe that the `embedding` column is reset to `null` after updating the content. This is because of the trigger we added to clear existing embeddings whenever the content is updated. The embedding will be regenerated by the Edge Function in the next scheduled task.
Wait up to 10 seconds for the next task to run, then check the `embedding` column again:
```sql
select id, embedding
from documents
where title = 'Understanding Vector Databases';
```
You should see the updated embedding for the document.
Finally we'll update the title of the document:
```sql
-- Update the title of the document
update documents
set title = 'Understanding Vector Databases with Supabase'
where title = 'Understanding Vector Databases';
```
You should observe that the `embedding` column is once again reset to `null` after updating the title. This is because the trigger we added to clear existing embeddings fires when either the `content` or `title` columns are updated. The embedding will be regenerated by the Edge Function in the next scheduled task.
Wait up to 10 seconds for the next task to run, then check the `embedding` column again:
```sql
select id, embedding
from documents
where title = 'Understanding Vector Databases with Supabase';
```
You should see the updated embedding for the document.
## Troubleshooting
The `embed` Edge Function processes a batch of embedding jobs and returns a `200 OK` response with a list of completed and failed jobs in the body. For example:
```json
{
"completedJobs": [
{
"jobId": "1",
"id": "1",
"schema": "public",
"table": "documents",
"contentFunction": "embedding_input",
"embeddingColumn": "embedding"
}
],
"failedJobs": [
{
"jobId": "2",
"id": "2",
"schema": "public",
"table": "documents",
"contentFunction": "embedding_input",
"embeddingColumn": "embedding",
"error": "error connecting to openai api"
}
]
}
```
It also returns the number of completed and failed jobs in the response headers. For example:
```
X-Completed-Jobs: 1
X-Failed-Jobs: 1
```
You can also use the `X-Deno-Execution-Id` header to trace the execution of the Edge Function within the [dashboard](/dashboard/project/_/functions) logs.
Each failed job includes an `error` field with a description of the failure. Reasons for a job failing could include:
- An error generating the embedding via external API
- An error connecting to the database
- The edge function being terminated (e.g. due to a wall clock limit)
- Any other error thrown during processing
`pg_net` stores HTTP responses in the `net._http_response` table, which can be queried to diagnose issues with the embedding generation process.
```sql
select
*
from
net._http_response
where
(headers->>'X-Failed-Jobs')::int > 0;
```
## Conclusion
Automating embedding generation and updates in Postgres allow you to build powerful semantic search capabilities without the complexity of managing embeddings manually.
By combining Postgres features like triggers, queues, and other extensions with Supabase Edge Functions, we can create a robust system that handles embedding generation asynchronously and retries failed jobs automatically.
This system can be customized to work with any content and embedding generation service, providing a flexible and scalable solution for semantic search in Postgres.
## See also
- [What are embeddings?](/docs/guides/ai/concepts)
- [Semantic search](/docs/guides/ai/semantic-search)
- [Vector indexes](/docs/guides/ai/vector-indexes)
- [Supabase Edge Functions](/docs/guides/functions)
apps/docs/content/guides/database/prisma.mdx
+9 -9
View File
@@ -98,11 +98,11 @@ If you plan to solely use Prisma instead of the Supabase Data API (PostgREST), t
<TabPanel id="yarn_initiate" label="yarn">
```bash
yarn init -y
yarn install prisma typescript ts-node @types/node --save-dev
yarn add prisma typescript ts-node @types/node --save-dev
yarn dlx tsc --init
npx tsc --init
yarn dlx prisma init
npx prisma init
```
</TabPanel>
<TabPanel id="bun_initiate" label="bun">
@@ -241,7 +241,7 @@ If you plan to solely use Prisma instead of the Supabase Data API (PostgREST), t
</TabPanel>
<TabPanel id="yarn_migrate" label="yarn">
```bash
yarn dlx prisma migrate dev --name first_prisma_migration
npx prisma migrate dev --name first_prisma_migration
```
</TabPanel>
@@ -318,7 +318,7 @@ If you plan to solely use Prisma instead of the Supabase Data API (PostgREST), t
<TabPanel id="yarn_sync" label="yarn">
```bash
yarn dlx prisma db pull
npx prisma db pull
```
Create a migration file
@@ -328,7 +328,7 @@ If you plan to solely use Prisma instead of the Supabase Data API (PostgREST), t
Synchronize the migrations
```bash
yarn dlx prisma migrate diff \
npx prisma migrate diff \
--from-empty \
--to-schema-datamodel prisma/schema.prisma \
--script > prisma/migrations/0_init_supabase/migration.sql
@@ -338,7 +338,7 @@ If you plan to solely use Prisma instead of the Supabase Data API (PostgREST), t
</Admonition>
```bash
yarn dlx prisma migrate resolve --applied 0_init_supabase
npx prisma migrate resolve --applied 0_init_supabase
```
</TabPanel>
@@ -403,8 +403,8 @@ If you plan to solely use Prisma instead of the Supabase Data API (PostgREST), t
</TabPanel>
<TabPanel id="yarn_client" label="yarn">
```sh
yarn install @prisma/client
yarn dlx prisma generate
yarn add @prisma/client
npx prisma generate
```
</TabPanel>
<TabPanel id="bun_client" label="bun">
apps/docs/spec/cli_v1_commands.yaml
+1 -1
View File
@@ -3006,7 +3006,7 @@ commands:
Linked project is up to date.
- id: self-hosted
name: Self hosted
code: supabase db push --db-url "postgres://user:pass@127.0.0.1:5432/postgres"
code: supabase db push --db-url "postgres://user.tenant:pass@127.0.0.1:5432"
response: |
Pushing migration 20230410135622_create_employees_table.sql...
Finished supabase db push.
apps/docs/spec/supabase_kt_v3.yml
+1 -1
View File
@@ -2883,7 +2883,7 @@ functions:
isSpotlight: true
code: |
```kotlin
val supabase = createSupabaseClient(supabaseURL = "https://xyzcompany.supabase.co'", supabaseKey = "public-anon-key") { ... }
val supabase = createSupabaseClient(supabaseUrl = "https://xyzcompany.supabase.co'", supabaseKey = "public-anon-key") { ... }
val auth = supabase.auth
```
- id: sign-up
apps/studio/components/interfaces/Auth/Users/DeleteUserModal.tsx
+61
View File
@@ -0,0 +1,61 @@
import { useParams } from 'common'
import { toast } from 'sonner'
import { useUserDeleteMutation } from 'data/auth/user-delete-mutation'
import { User } from 'data/auth/users-infinite-query'
import { timeout } from 'lib/helpers'
import ConfirmationModal from 'ui-patterns/Dialogs/ConfirmationModal'
interface DeleteUserModalProps {
visible: boolean
selectedUser?: User
onClose: () => void
onDeleteSuccess?: () => void
}
export const DeleteUserModal = ({
visible,
selectedUser,
onClose,
onDeleteSuccess,
}: DeleteUserModalProps) => {
const { ref: projectRef } = useParams()
const { mutate: deleteUser, isLoading: isDeleting } = useUserDeleteMutation({
onSuccess: () => {
toast.success(`Successfully deleted ${selectedUser?.email}`)
onDeleteSuccess?.()
},
})
const handleDeleteUser = async () => {
await timeout(200)
if (!projectRef) return console.error('Project ref is required')
if (selectedUser?.id === undefined) {
return toast.error(`Failed to delete user: User ID not found`)
}
deleteUser({ projectRef, userId: selectedUser.id })
}
return (
<ConfirmationModal
visible={visible}
variant="destructive"
title="Confirm to delete user"
loading={isDeleting}
confirmLabel="Delete"
onCancel={() => onClose()}
onConfirm={() => handleDeleteUser()}
alert={{
title: 'Deleting a user is irreversible',
description:
'This will remove the selected the user from the project and all associated data.',
}}
>
<p className="text-sm text-foreground-light">
This is permanent! Are you sure you want to delete the user{' '}
{selectedUser?.email ?? selectedUser?.phone ?? 'this user'}?
</p>
</ConfirmationModal>
)
}
apps/studio/components/interfaces/Auth/Users/UserOverview.tsx
+11 -35
View File
@@ -10,7 +10,6 @@ import { ButtonTooltip } from 'components/ui/ButtonTooltip'
import CopyButton from 'components/ui/CopyButton'
import { useAuthConfigQuery } from 'data/auth/auth-config-query'
import { useUserDeleteMFAFactorsMutation } from 'data/auth/user-delete-mfa-factors-mutation'
import { useUserDeleteMutation } from 'data/auth/user-delete-mutation'
import { useUserResetPasswordMutation } from 'data/auth/user-reset-password-mutation'
import { useUserSendMagicLinkMutation } from 'data/auth/user-send-magic-link-mutation'
import { useUserSendOTPMutation } from 'data/auth/user-send-otp-mutation'
@@ -24,6 +23,7 @@ import { Admonition } from 'ui-patterns/admonition'
import ConfirmationModal from 'ui-patterns/Dialogs/ConfirmationModal'
import { PROVIDERS_SCHEMAS } from '../AuthProvidersFormValidation'
import { BanUserModal } from './BanUserModal'
import { DeleteUserModal } from './DeleteUserModal'
import { UserHeader } from './UserHeader'
import { PANEL_PADDING } from './UserPanel'
import { providerIconMap } from './Users.utils'
@@ -109,13 +109,6 @@ export const UserOverview = ({ user, onDeleteSuccess }: UserOverviewProps) => {
toast.error(`Failed to send OTP: ${err.message}`)
},
})
const { mutate: deleteUser } = useUserDeleteMutation({
onSuccess: () => {
toast.success(`Successfully deleted ${user?.email}`)
setIsDeleteModalOpen(false)
onDeleteSuccess()
},
})
const { mutate: deleteUserMFAFactors } = useUserDeleteMFAFactorsMutation({
onSuccess: () => {
toast.success("Successfully deleted the user's factors")
@@ -129,15 +122,6 @@ export const UserOverview = ({ user, onDeleteSuccess }: UserOverviewProps) => {
},
})
const handleDelete = async () => {
await timeout(200)
if (!projectRef) return console.error('Project ref is required')
if (user.id === undefined) {
return toast.error(`Failed to delete user: User ID not found`)
}
deleteUser({ projectRef, userId: user.id })
}
const handleDeleteFactors = async () => {
await timeout(200)
if (!projectRef) return console.error('Project ref is required')
@@ -179,7 +163,7 @@ export const UserOverview = ({ user, onDeleteSuccess }: UserOverviewProps) => {
<Separator />
)}
<div className={cn('flex flex-col gap-1.5', PANEL_PADDING)}>
<div className={cn('flex flex-col gap-y-1', PANEL_PADDING)}>
<RowData property="User UID" value={user.id} />
<RowData
property="Created at"
@@ -403,23 +387,15 @@ export const UserOverview = ({ user, onDeleteSuccess }: UserOverviewProps) => {
</div>
</div>
<ConfirmationModal
<DeleteUserModal
visible={isDeleteModalOpen}
variant="destructive"
title="Confirm to delete user"
confirmLabel="Delete"
onCancel={() => setIsDeleteModalOpen(false)}
onConfirm={() => handleDelete()}
alert={{
title: 'Deleting a user is irreversible',
description: 'This will remove the user from the project and all associated data.',
selectedUser={user}
onClose={() => setIsDeleteModalOpen(false)}
onDeleteSuccess={() => {
setIsDeleteModalOpen(false)
onDeleteSuccess()
}}
>
<p className="text-sm text-foreground-light">
This is permanent! Are you sure you want to delete the user{' '}
<span className="text-foreground">{user.email ?? user.phone ?? 'this user'}</span>?
</p>
</ConfirmationModal>
/>
<ConfirmationModal
visible={isDeleteFactorsModalOpen}
@@ -466,7 +442,7 @@ export const RowData = ({ property, value }: { property: string; value?: string
return (
<>
<div className="flex items-center gap-x-2 group justify-between">
<p className=" text-foreground-lighter text-sm">{property}</p>
<p className=" text-foreground-lighter text-xs">{property}</p>
{typeof value === 'boolean' ? (
<div className="h-[26px] flex items-center justify-center min-w-[70px]">
{value ? (
@@ -481,7 +457,7 @@ export const RowData = ({ property, value }: { property: string; value?: string
</div>
) : (
<div className="flex items-center gap-x-2 h-[26px] font-mono min-w-[40px]">
<p className="text-sm">{!value ? '-' : value}</p>
<p className="text-xs">{!value ? '-' : value}</p>
{!!value && (
<CopyButton
iconOnly
apps/studio/components/interfaces/Auth/Users/Users.utils.tsx
+77 -38
View File
@@ -1,11 +1,20 @@
import dayjs from 'dayjs'
import { UserIcon } from 'lucide-react'
import { Clipboard, Trash, UserIcon } from 'lucide-react'
import { UIEvent } from 'react'
import { Column, useRowSelection } from 'react-data-grid'
import { User } from 'data/auth/users-infinite-query'
import { BASE_PATH } from 'lib/constants'
import { Checkbox_Shadcn_, cn } from 'ui'
import { copyToClipboard } from 'lib/helpers'
import {
Checkbox_Shadcn_,
cn,
ContextMenu_Shadcn_,
ContextMenuContent_Shadcn_,
ContextMenuItem_Shadcn_,
ContextMenuSeparator_Shadcn_,
ContextMenuTrigger_Shadcn_,
} from 'ui'
import { HeaderCell } from './UsersGridComponents'
import { ColumnConfiguration, USERS_TABLE_COLUMNS } from './UsersV2'
@@ -248,11 +257,13 @@ export const formatUserColumns = ({
users,
visibleColumns = [],
setSortByValue,
onSelectDeleteUser,
}: {
config: ColumnConfiguration[]
users: User[]
visibleColumns?: string[]
setSortByValue: (val: string) => void
onSelectDeleteUser: (user: User) => void
}) => {
const columnOrder = config.map((c) => c.id) ?? USERS_TABLE_COLUMNS.map((c) => c.id)
@@ -319,42 +330,70 @@ export const formatUserColumns = ({
}
return (
<div
className={cn(
'w-full flex items-center text-xs',
col.id.includes('provider') ? 'capitalize' : ''
)}
>
{/* [Joshen] Not convinced this is the ideal way to display the icons, but for now */}
{col.id === 'providers' &&
row.provider_icons.map((icon: string, idx: number) => {
const provider = row.providers[idx]
return (
<div
className="min-w-6 min-h-6 rounded-full border flex items-center justify-center bg-surface-75"
style={{
marginLeft: idx === 0 ? 0 : `-8px`,
zIndex: row.provider_icons.length - idx,
}}
>
<img
key={`${user?.id}-${provider}`}
width={16}
src={icon}
alt={`${provider} auth icon`}
className={cn(provider === 'github' && 'dark:invert')}
/>
</div>
)
})}
{col.id === 'last_sign_in_at' && !isConfirmed ? (
<p className="text-foreground-lighter">Waiting for verification</p>
) : (
<p className={cn(col.id === 'providers' && 'ml-1')}>
{formattedValue === null ? '-' : formattedValue}
</p>
)}
</div>
<ContextMenu_Shadcn_>
<ContextMenuTrigger_Shadcn_ asChild>
<div
className={cn(
'w-full flex items-center text-xs',
col.id.includes('provider') ? 'capitalize' : ''
)}
>
{/* [Joshen] Not convinced this is the ideal way to display the icons, but for now */}
{col.id === 'providers' &&
row.provider_icons.map((icon: string, idx: number) => {
const provider = row.providers[idx]
return (
<div
className="min-w-6 min-h-6 rounded-full border flex items-center justify-center bg-surface-75"
style={{
marginLeft: idx === 0 ? 0 : `-8px`,
zIndex: row.provider_icons.length - idx,
}}
>
<img
key={`${user?.id}-${provider}`}
width={16}
src={icon}
alt={`${provider} auth icon`}
className={cn(provider === 'github' && 'dark:invert')}
/>
</div>
)
})}
{col.id === 'last_sign_in_at' && !isConfirmed ? (
<p className="text-foreground-lighter">Waiting for verification</p>
) : (
<p className={cn(col.id === 'providers' && 'ml-1')}>
{formattedValue === null ? '-' : formattedValue}
</p>
)}
</div>
</ContextMenuTrigger_Shadcn_>
<ContextMenuContent_Shadcn_ onClick={(e) => e.stopPropagation()}>
<ContextMenuItem_Shadcn_
className="gap-x-2"
onFocusCapture={(e) => e.stopPropagation()}
onSelect={() => {
const value = col.id === 'providers' ? row.providers.join(', ') : formattedValue
copyToClipboard(value)
}}
>
<Clipboard size={12} />
<span>Copy {col.id === 'id' ? col.name : col.name.toLowerCase()}</span>
</ContextMenuItem_Shadcn_>
<ContextMenuSeparator_Shadcn_ />
<ContextMenuItem_Shadcn_
className="gap-x-2"
onFocusCapture={(e) => e.stopPropagation()}
onSelect={() => {
if (user) onSelectDeleteUser(user)
}}
>
<Trash size={12} />
<span>Delete user</span>
</ContextMenuItem_Shadcn_>
</ContextMenuContent_Shadcn_>
</ContextMenu_Shadcn_>
)
},
}
apps/studio/components/interfaces/Auth/Users/UsersV2.tsx
+24 -5
View File
@@ -16,7 +16,7 @@ import { FormHeader } from 'components/ui/Forms/FormHeader'
import { authKeys } from 'data/auth/keys'
import { useUserDeleteMutation } from 'data/auth/user-delete-mutation'
import { useUsersCountQuery } from 'data/auth/users-count-query'
import { useUsersInfiniteQuery } from 'data/auth/users-infinite-query'
import { User, useUsersInfiniteQuery } from 'data/auth/users-infinite-query'
import { useLocalStorageQuery } from 'hooks/misc/useLocalStorage'
import { LOCAL_STORAGE_KEYS } from 'lib/constants'
import {
@@ -44,6 +44,7 @@ import { Input } from 'ui-patterns/DataInputs/Input'
import ConfirmationModal from 'ui-patterns/Dialogs/ConfirmationModal'
import { GenericSkeletonLoader } from 'ui-patterns/ShimmeringLoader'
import AddUserDropdown from './AddUserDropdown'
import { DeleteUserModal } from './DeleteUserModal'
import { UserPanel } from './UserPanel'
import { MAX_BULK_DELETE, PROVIDER_FILTER_OPTIONS } from './Users.constants'
import { formatUserColumns, formatUsersData, isAtBottom } from './Users.utils'
@@ -83,13 +84,16 @@ export const UsersV2 = () => {
const [search, setSearch] = useState('')
const [filter, setFilter] = useState<Filter>('all')
const [filterKeywords, setFilterKeywords] = useState('')
const [selectedUsers, setSelectedUsers] = useState<Set<any>>(new Set([]))
const [selectedColumns, setSelectedColumns] = useState<string[]>([])
const [selectedProviders, setSelectedProviders] = useState<string[]>([])
const [selectedUser, setSelectedUser] = useState<string>()
const [sortByValue, setSortByValue] = useState<string>('created_at:desc')
const [selectedUser, setSelectedUser] = useState<string>()
const [selectedUsers, setSelectedUsers] = useState<Set<any>>(new Set([]))
const [selectedUserToDelete, setSelectedUserToDelete] = useState<User>()
const [showDeleteModal, setShowDeleteModal] = useState(false)
const [isDeletingUsers, setIsDeletingUsers] = useState(false)
const [
columnConfiguration,
setColumnConfiguration,
@@ -143,7 +147,7 @@ export const UsersV2 = () => {
const totalUsers = countData ?? 0
const users = useMemo(() => data?.pages.flatMap((page) => page.result) ?? [], [data?.pages])
// [Joshen] Only relevant for when selecting one user only
const selectedUserToDelete = users.find((u) => u.id === [...selectedUsers][0])
const selectedUserFromCheckbox = users.find((u) => u.id === [...selectedUsers][0])
const handleScroll = (event: UIEvent<HTMLDivElement>) => {
const isScrollingHorizontally = xScroll.current !== event.currentTarget.scrollLeft
@@ -199,6 +203,11 @@ export const UsersV2 = () => {
500
)
const onSelectDeleteUser = (user: User) => {
setSelectedUsers(new Set([user.id]))
setShowDeleteModal(true)
}
const handleDeleteUsers = async () => {
if (!projectRef) return console.error('Project ref is required')
const userIds = [...selectedUsers]
@@ -237,6 +246,7 @@ export const UsersV2 = () => {
users: users ?? [],
visibleColumns: selectedColumns,
setSortByValue,
onSelectDeleteUser,
})
setColumns(columns)
if (columns.length < USERS_TABLE_COLUMNS.length) {
@@ -377,6 +387,7 @@ export const UsersV2 = () => {
users: users ?? [],
visibleColumns: value,
setSortByValue,
onSelectDeleteUser,
})
setSelectedColumns(value)
@@ -580,12 +591,20 @@ export const UsersV2 = () => {
{selectedUsers.size === 1 ? (
<span className="text-foreground">
{' '}
{selectedUserToDelete?.email ?? selectedUserToDelete?.phone ?? 'this user'}
{selectedUserFromCheckbox?.email ?? selectedUserFromCheckbox?.phone ?? 'this user'}
</span>
) : null}
?
</p>
</ConfirmationModal>
{/* [Joshen] For deleting via context menu, the dialog above is dependent on the selectedUsers state */}
<DeleteUserModal
visible={!!selectedUserToDelete}
selectedUser={selectedUserToDelete}
onClose={() => setSelectedUserToDelete(undefined)}
onDeleteSuccess={() => setSelectedUserToDelete(undefined)}
/>
</>
)
}
apps/www/_blog/2024-08-12-github-copilot-extension-for-vs-code.mdx
+1 -1
View File
@@ -64,7 +64,7 @@ List the Storage buckets in your Supabase project.
We're excited to continue adding more features that will make your development experience with Supabase even more delightful - and for this we need your help! If you have any feedback, feature requests, or bug reports, please [open an issue on GitHub](https://github.com/supabase-community/supabase-vscode-extension/issues).
The extension requires you to have the Supabase CLI installed and have your project running locally. In a future release, we will integrate the [Supabase Managamenet API](/docs/reference/api/introduction) into the extension to make connecting to your hosted Supabase projects as seamless as possible.
The extension requires you to have the Supabase CLI installed and have your project running locally. In a future release, we will integrate the [Supabase Management API](/docs/reference/api/introduction) into the extension to make connecting to your hosted Supabase projects as seamless as possible.
## Contributing to Supabase
apps/www/_blog/2024-08-13-supabase-realtime-broadcast-and-presence-authorization.mdx
+1 -1
View File
@@ -212,7 +212,7 @@ Realtime uses the private flag client's define when creating channel, takes the
light:
'/images/blog/2024-08-13-supabase-realtime-broadcast-and-presence-authorization/one-user-connection--light.png',
}}
alt="Flow of checks done against the databse to determine the policies for a user connection."
alt="Flow of checks done against the database to determine the policies for a user connection."
/>
## How is this approach performant?
apps/www/_blog/2024-12-06-launch-week-13-top-10.mdx
+1 -1
View File
@@ -35,7 +35,7 @@ A feature tri-fecta: Edge Functions now support Websockets for things like OpenA
## #4: CLI v2: Config as Code
We realeased v2 of the CLI, adding support for Configuration as Code. You can now use the CLI in your GitHub actions to keep all of your project configuration in sync.
We released v2 of the CLI, adding support for Configuration as Code. You can now use the CLI in your GitHub actions to keep all of your project configuration in sync.
## #5: High Performance Disks
supa-mdx-lint/Rule003Spelling.toml
+2
View File
@@ -42,6 +42,7 @@ allow_list = [
"[Dd]evs?",
"[Dd]iff(s|ing|ed)?",
"[Dd]ropdown",
"[Ee]nqueues?",
"[Ee]nums?",
"[Ee]nv",
"[Ff]atals",
@@ -54,6 +55,7 @@ allow_list = [
"[Hh]ypertables?",
"[KMG]bps",
"[KMG]iB",
"[Mm]atryoshka",
"[Mm]essageBird",
"[Mm]icroservices?",
"[Mm]iddlewares?",