mirror of
https://github.com/supabase/supabase.git
synced 2026-10-18 23:59:08 +03:00
update links and refine sections
This commit is contained in:
2 files changed
+11
-17
No files matched your search
+1
-1
@@ -23,7 +23,7 @@ When using logical replication, you have a few active components:
|
||||
- `replication slot` - A slot used for replicating the data from a single publication. The slot, when created, will specify the output format of the changes
|
||||
- `subscription` - A subscription is made from an external system (i.e. another Postgresql database) and must specify the name of the `publication`. If you do not specify a replication slot, one is automatically created for you
|
||||
|
||||
#### Logical Replication Output Format
|
||||
## Logical Replication Output Format
|
||||
|
||||
Logical replication is typically output in 2 forms, `pgoutput` and `wal2json`. The output method is how Postgresql sends changes to any active replication slot.
|
||||
|
||||
@@ -5,33 +5,27 @@ description: 'Performing Extract Transform Load (ETL) with Supabase'
|
||||
sidebar_label: 'Overview'
|
||||
---
|
||||
|
||||
# Replication, Analytics and ETL
|
||||
# Logical Replication and Change Data Capture (CDC)
|
||||
|
||||
You have launched your app. You want to understand what works and what doesn't so you start some queries and soon notice that your queries are taking minutes or that you need to combine these data with another system you are using. Not to worry, you are in the right place.
|
||||
|
||||
## How?
|
||||
|
||||
OK, so now we know why you should not use your primary database for this, what should you use? This is not an easy question and it comes with the "it depends" answer so let's go through some options:
|
||||
|
||||
### Read Replicas
|
||||
## Read Replicas
|
||||
|
||||
[Read-Replicas](/guides/platform/read-replicas) are available on Supabase and allow you to create a second database that is a direct, read-only copy of your primary database that stays in sync. This can be great for many things, like load-balancing, and, in this case, they can be great for analytics workloads.
|
||||
|
||||
### Logical Replication
|
||||
## Logical Replication
|
||||
|
||||
Your use case might go beyond what a Read-Replica can offer and you may need to replicate your data from Supabase into a Data Warehouse, like Clickhouse or Google BigQuery. This could be because of long running queries, the amount of data or because you need to combine your Supabase data with data from other systems.
|
||||
|
||||
When you want to do this, this is where Extract Transform Load (ETL) comes in.
|
||||
This is done using a method called [Logical Replication](/guides/database/replication-cdc/logical-replication).
|
||||
|
||||
1. Extract - Getting the data out of your source, typically using a method called [Logical Replication](https://www.postgresql.org/docs/current/logical-replication.html)
|
||||
2. Transform - Modifying the data for ingestion
|
||||
3. Load - Loading the transformed data into the destination
|
||||
In most cases you will want to use a tool, such as Estuary or Airbyte or Fivetran, to susbscribe to your database, capture the changes and write them to a destination of your choice, ranging from S3 to Google BigQuery to Google Sheets to Salesforce.
|
||||
|
||||
To perform each of these steps, some destinations will pull data directly from Postgresql but, in most cases, you will want to use a tool, such as Estuary or Airbyte or Fivetran. These perform the `Extract` and `Transform` in much the same way, as well as support a large number of destinations to `Load` your data into, ranging from S3 to Salesforce.
|
||||
Regardless of the tool you choose, it is very likely you will be using logical replication, so make sure you read the [introduction]((/guides/database/replication-cdc/logical-replication) to that, as well as the [FAQ](/guides/databae/replication-cdc/faq)
|
||||
|
||||
Regardless of the tool you choose, it is very likely you will be using logical replication, so let's take a moment to coer that.
|
||||
To learn how to configure Supabase with a specific tool, select a services from the navigation menu.
|
||||
|
||||
## Resources
|
||||
|
||||
To get started, choose the tool you want to use below:
|
||||
|
||||
<DatabaseReplicationConfig />
|
||||
- [Managing logical replication](/guides/database/managing-replication)
|
||||
- [Replicating to an external Postgres database](/guides/database/postgres/setup-replicstion-external)
|
||||
Reference in new issue
Block a user