This commit is contained in:
Chris Chinchilla committed 2025-08-29 19:09:09 +02:00
1 parent 102c4ee18c
commit dbe74e3256
2 files changed
+14 -9

No files matched your search

@@ -3,7 +3,7 @@ title: ETL
description: Replicate your Supabase database to external destinations
---
`ETL` simplifies the process of syncing your data to an external destination. This could be another database, a data warehouse, or a specialized service such as customer management or business analytics software.
ETL simplifies the process of syncing your data to an external destination. This could be another database, a data warehouse, or a specialized service such as customer management or business analytics software.
<Admonition type="caution">
@@ -53,21 +53,26 @@ Before setting up replication to BigQuery, you need:
1. Navigate to the [Database](https://supabase.com/dashboard/project/_/database/tables) section in your Supabase Dashboard and select the **Replication** tab.
2. Click **Add destination** to open the New Destination panel.
3. Click **Enable**.
4. Configure the replication settings:
3. Configure the replication settings:
- **Name**: Enter a descriptive name for your replication destination
- **Publication**: Select an existing publication or create a new one
- **Type**: Select "BigQuery" from the destination types
- **Project Id**: Enter the GCP project ID
- **Project's Dataset Id**: Enter the ID of the BigQuery dataset
- **Service Account Key**: Enter your GCP service account key
5. Review the [Advanced Settings](#advanced-settings) to change the limits for the replication.
4. Review the [Advanced Settings](#advanced-settings) to change the limits for the replication.
- **Max size**: Changes are sent in batches. Max size limits the size of an individual batch.
- **Max fill seconds**: Number of seconds after which a batch should be sent, even if it hasn't reached the max size.
- **Max staleness**: See BigQuery's [`max_staleness`](https://cloud.google.com/bigquery/docs/change-data-capture#manage_table_staleness) option
- **Max size**: Changes are sent in batches. Max size limits the size of an individual batch.
- **Max fill seconds**: Number of seconds after which a batch should be sent, even if it hasn't reached the max size.
- **Max staleness**: See BigQuery's [`max_staleness`](https://cloud.google.com/bigquery/docs/change-data-capture#manage_table_staleness) option
6. Click **Create** to start the replication process.
5. Click **Create** to start the replication process.
<Admonition type="info">
Due to ingestion latency in BigQuery, there may be a delay in the dataflow.
</Admonition>
## Monitoring replication
@@ -3,7 +3,7 @@ title: Manual Replication
subtitle: Set up your own replication pipeline for more control
---
ETL enables you to build replication pipelines from Postgres databases to various analytics-focused destinations such as BigQuery, DuckDB, Iceberg, and other specialized systems optimized for OLAP workloads, unlocking fast, scalable analysis with minimal setup and operational overhead.
Postgres comes with built-in support for replication. If `pg_replicate` doesn't provide a destination that you need, or if you need more control over the replication process, you can set up your own replication pipeline using Postgres primitives.
## Replication and publications