mirror of
https://github.com/supabase/supabase.git
synced 2026-10-05 09:25:06 +03:00
Danny WhiteandChris Chinchilla
9c03ad6b02
## What kind of change does this PR introduce? Docs update. Related to DEPR-551. ## What is the current behavior? Docs MDX still uses the legacy `label` prop for Admonitions, even though #45618 added `title` and kept `label` only as a backwards-compatible alias after #45302 was reverted in #45535. ## What is the new behavior? Migrates Docs-owned Admonitions from `label=` to `title=` without changing rendered copy, component APIs, Studio callsites, design-system examples, or the legacy `label` alias. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Standardized admonition headings across the docs by switching how admonition headings are provided (preserving all visible guidance and examples). Content and instructions remain unchanged; this ensures consistent rendering of callouts and improves uniformity across guides and reference pages. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46053?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Chris Chinchilla <chris.ward@supabase.io>
281 lines
13 KiB
Plaintext
281 lines
13 KiB
Plaintext
---
|
|
id: 'external-replication-setup'
|
|
title: 'Set up external replication'
|
|
description: 'Set up external replication using Postgres logical replication.'
|
|
subtitle: 'Configure publications and destinations for external replication.'
|
|
sidebar_label: 'Setting up'
|
|
---
|
|
|
|
<Admonition type="caution" title="Private Alpha">
|
|
|
|
External replication is currently in private alpha. Access is limited and features may change.
|
|
|
|
</Admonition>
|
|
|
|
External replication is powered by [Supabase ETL](https://github.com/supabase/etl) and uses **Postgres logical replication** to stream changes from your database to external data systems. It provides a managed interface through the Dashboard to configure and monitor replication pipelines.
|
|
|
|
## Setup overview
|
|
|
|
External replication requires two main components: a **Postgres publication** (defines what to replicate) and a **destination** (where data is sent). Follow these steps to set up your replication pipeline.
|
|
|
|
<Admonition type="tip">
|
|
|
|
If you already have a Postgres publication set up, you can skip to [Step 2: Enable external replication](#step-2-enable-external-replication).
|
|
|
|
</Admonition>
|
|
|
|
### Step 1: Create a Postgres publication
|
|
|
|
A Postgres publication defines which tables and change types will be replicated from your database. You create publications using SQL.
|
|
|
|
#### Creating a publication
|
|
|
|
The following SQL examples assume you have `users` and `orders` tables in your database.
|
|
|
|
##### Publication for specific tables
|
|
|
|
```sql
|
|
-- Create publication for both tables
|
|
create publication pub_users_orders
|
|
for table users, orders;
|
|
```
|
|
|
|
This publication tracks all changes (INSERT, UPDATE, DELETE, TRUNCATE) for both the `users` and `orders` tables.
|
|
|
|
##### Publication for all tables in a schema
|
|
|
|
```sql
|
|
-- Create a publication for all tables in the public schema
|
|
create publication pub_all_public for tables in schema public;
|
|
```
|
|
|
|
This tracks changes for all existing and future tables in the `public` schema.
|
|
|
|
##### Publication for all tables
|
|
|
|
```sql
|
|
-- Create a publication for all tables
|
|
create publication pub_all_tables for all tables;
|
|
```
|
|
|
|
This tracks changes for all tables in your database.
|
|
|
|
#### Advanced publication options
|
|
|
|
##### Selecting specific columns
|
|
|
|
You can replicate only a subset of columns from a table:
|
|
|
|
```sql
|
|
-- Replicate only specific columns from the users table
|
|
create publication pub_users_subset
|
|
for table users (id, email, created_at);
|
|
```
|
|
|
|
This only replicates the `id`, `email`, and `created_at` columns from the `users` table.
|
|
|
|
##### Filtering rows with a predicate
|
|
|
|
You can filter which rows to replicate using a `WHERE` clause:
|
|
|
|
```sql
|
|
-- Only replicate active users
|
|
create publication pub_active_users
|
|
for table users where (status = 'active');
|
|
|
|
-- Only replicate recent orders
|
|
create publication pub_recent_orders
|
|
for table orders where (created_at > '2024-01-01');
|
|
```
|
|
|
|
#### Viewing publications in the Dashboard
|
|
|
|
After creating a publication via SQL, you can view it in the Dashboard:
|
|
|
|
1. Navigate to the [**Database > Publications**](/dashboard/project/_/database/publications) section of the Dashboard
|
|
2. You'll see all your publications listed with their tables
|
|
|
|
### Step 2: Enable external replication
|
|
|
|
Before creating an external replication pipeline, enable external replication for your project:
|
|
|
|
1. Navigate to the [**Database > Replication**](/dashboard/project/_/database/replication) section of the Dashboard
|
|
2. Click **Add destination** to show the replication side panel
|
|
3. Select an external replication destination, such as **BigQuery**
|
|
4. Click **Enable external replication**
|
|
|
|
<Image
|
|
alt="Enable external replication"
|
|
src="/docs/img/database/replication/replication-enable-replication.png"
|
|
width={3560}
|
|
height={2156}
|
|
/>
|
|
|
|
### Step 3: Configure a destination
|
|
|
|
Once external replication is enabled and you have a Postgres publication, configure a destination. The destination is where your replicated data will be stored, while the pipeline is the active Postgres replication process that continuously streams changes from your database to that destination.
|
|
|
|
#### Choose and configure your destination
|
|
|
|
Follow these steps to configure your destination. Each destination has its own setup requirements and behavior. **BigQuery** is currently available, and we are working on new destinations.
|
|
|
|
1. Navigate to the [**Database > Replication**](/dashboard/project/_/database/replication) section of the Dashboard
|
|
2. Click **Add destination** if the destination side panel isn't already open
|
|
|
|
3. Configure the general settings:
|
|
- **Destination name**: A name to identify this destination
|
|
- **Publication**: The publication to replicate data from (created in [Step 1](#step-1-create-a-postgres-publication))
|
|
- **Destination type**: Select the destination you want to use
|
|
|
|
4. Configure the destination-specific settings. See the destination guide for required credentials, permissions, and limitations:
|
|
- [BigQuery](/docs/guides/database/replication/bigquery)
|
|
|
|
5. Optionally expand **Advanced settings** to tune pipeline behavior. These settings apply to the pipeline rather than the destination:
|
|
|
|
| Setting | Default | Description |
|
|
| ------------------------------ | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
| **Batch wait time** | `10000` milliseconds | Maximum time the pipeline waits to collect additional changes before flushing them. Lower values reduce replication latency. Higher values can improve batching efficiency. |
|
|
| **Table sync workers** | `4` workers | Number of tables copied in parallel during the initial snapshot phase. Each worker uses one replication slot, up to `N + 1` total replication slots while syncing. |
|
|
| **Copy connections per table** | `2` connections | Number of parallel database connections each table copy can use during the initial sync. Increasing this can speed up large table copies, but uses more database connections. |
|
|
| **Invalidated slot behavior** | `Error` | What the pipeline does when its replication slot is invalidated. **Error** blocks startup so you can recover manually. **Recreate** rebuilds the slot and starts replication from scratch. |
|
|
|
|
Leave these settings at their defaults unless you need to tune initial copy speed, latency, or recovery behavior.
|
|
|
|
6. Click **Create and start** to begin replication
|
|
|
|
Your replication pipeline now starts copying data from your database to your destination.
|
|
|
|
### Step 4: Monitor your pipeline
|
|
|
|
After creating a destination, the replication pipeline starts and appears in the destinations list. You can monitor the pipeline's status and performance from the Dashboard.
|
|
|
|
<Image
|
|
alt="External replication destinations list"
|
|
src="/docs/img/database/replication/replication-destinations-list.png"
|
|
width={3560}
|
|
height={2146}
|
|
|
|
/>
|
|
|
|
For comprehensive monitoring instructions including pipeline states, metrics, and logs, see the [external replication monitoring guide](/docs/guides/database/replication/external-replication-monitoring).
|
|
|
|
### Managing your pipeline
|
|
|
|
You can manage your pipeline from the destinations list using the actions menu.
|
|
|
|
<Image
|
|
alt="Pipeline actions"
|
|
src="/docs/img/database/replication/replication-pipeline-actions.png"
|
|
width={3560}
|
|
height={2146}
|
|
|
|
/>
|
|
|
|
Available actions:
|
|
|
|
- **Start**: Begin replication for a stopped pipeline
|
|
- **Stop**: Pause replication (changes will queue up in the WAL)
|
|
- **Restart**: Stop and start the pipeline (required after publication changes)
|
|
- **Edit destination**: Modify destination settings like credentials or advanced options
|
|
- **Delete**: Remove the destination and permanently stop replication
|
|
|
|
### Disabling external replication
|
|
|
|
To turn off external replication for a project, delete all external replication pipelines first. After all pipelines are removed, open the three-dot actions menu on the Replication page and click **Disable external replication**.
|
|
|
|
<Image
|
|
alt="Disable external replication from the Replication page actions menu"
|
|
src="/docs/img/database/replication/replication-disable-external-replication.png"
|
|
width={3560}
|
|
height={2156}
|
|
/>
|
|
|
|
For cleanup details, see [What happens when you disable external replication?](/docs/guides/database/replication/external-replication-faq#what-happens-when-you-disable-external-replication).
|
|
|
|
### Adding or removing tables
|
|
|
|
If you need to modify which tables are replicated after your replication pipeline is already running, follow these steps:
|
|
|
|
<Admonition type="note">
|
|
|
|
If your Postgres publication uses `FOR ALL TABLES` or `FOR TABLES IN SCHEMA`, new tables in that scope are automatically included in the publication. However, you still **must restart the replication pipeline** for the changes to take effect.
|
|
|
|
</Admonition>
|
|
|
|
#### Adding tables to replication
|
|
|
|
1. Add the table to your publication using SQL:
|
|
|
|
```sql
|
|
-- Add a single table to an existing publication
|
|
alter publication pub_users_orders add table products;
|
|
|
|
-- Or add multiple tables at once
|
|
alter publication pub_users_orders add table products, categories;
|
|
```
|
|
|
|
2. **Restart the replication pipeline** using the actions menu (see [Managing your pipeline](#managing-your-pipeline)) for the changes to take effect.
|
|
|
|
#### Removing tables from replication
|
|
|
|
1. Remove the table from your Postgres publication using SQL:
|
|
|
|
```sql
|
|
-- Remove a single table from a publication
|
|
alter publication pub_users_orders drop table orders;
|
|
|
|
-- Or remove multiple tables at once
|
|
alter publication pub_users_orders drop table orders, products;
|
|
```
|
|
|
|
2. **Restart the replication pipeline** using the actions menu (see [Managing your pipeline](#managing-your-pipeline)) for the changes to take effect.
|
|
|
|
<Admonition type="note">
|
|
|
|
When a table is deleted at the destination, the behavior depends on the destination. In general, the pipeline tries to recreate the table so replication can continue. To permanently delete a table, pause the pipeline first or remove it from the publication before deleting. See the [external replication FAQ](/docs/guides/database/replication/external-replication-faq#what-happens-if-a-table-is-deleted-at-the-destination) for details.
|
|
|
|
</Admonition>
|
|
|
|
### Schema change support
|
|
|
|
Schema change support depends on the destination. BigQuery is currently the only destination with beta schema change support. See [BigQuery schema change support](/docs/guides/database/replication/bigquery#schema-change-support) for supported and unsupported changes.
|
|
|
|
### How it works
|
|
|
|
Once configured, external replication:
|
|
|
|
1. **Captures** changes from your Postgres database using Postgres publications and logical replication
|
|
2. **Streams** the changes through the replication pipeline
|
|
3. **Loads** the data to your destination
|
|
|
|
External replication automatically optimizes how changes are delivered to the destination. The replication pipeline currently performs data extraction and loading only, without transformation - your data is replicated as-is to the destination.
|
|
|
|
### Troubleshooting
|
|
|
|
If you encounter issues during setup:
|
|
|
|
- **Publication not appearing**: Ensure you created the Postgres publication via SQL and refresh the dashboard
|
|
- **Tables not showing in publication**: Verify your tables have primary keys (required for Postgres logical replication)
|
|
- **Pipeline failed to start**: Check the error message in the status view for specific details
|
|
- **No data being replicated**: Verify your Postgres publication includes the correct tables and event types
|
|
|
|
For more troubleshooting help, see the [external replication FAQ](/docs/guides/database/replication/external-replication-faq).
|
|
|
|
### Limitations
|
|
|
|
External replication has the following limitations:
|
|
|
|
- **Primary keys required**: Tables must have primary keys (Postgres logical replication requirement)
|
|
- **Custom data types**: Not supported
|
|
- **Schema changes**: Currently in beta and limited to BigQuery
|
|
- **No data transformation**: Data is replicated as-is without transformation
|
|
- **Data duplicates**: Duplicates can occur when stopping a pipeline if your database has transactions that take longer than a few minutes to complete. See [Can data duplicates occur during pipeline operations?](/docs/guides/database/replication/external-replication-faq#can-data-duplicates-occur-during-pipeline-operations) for details
|
|
|
|
Destination-specific limitations, such as BigQuery's row size limits, are documented in each destination guide.
|
|
|
|
### Next steps
|
|
|
|
- [Set up BigQuery](/docs/guides/database/replication/bigquery)
|
|
- [Monitor external replication](/docs/guides/database/replication/external-replication-monitoring)
|
|
- [View external replication FAQ](/docs/guides/database/replication/external-replication-faq)
|