mirror of
https://github.com/supabase/supabase.git
synced 2026-10-13 21:29:08 +03:00
99 lines
6.5 KiB
Plaintext
99 lines
6.5 KiB
Plaintext
---
|
|
id: 'replication'
|
|
title: 'Database Replication'
|
|
description: 'Replicate your Postgres database to external destinations using logical replication.'
|
|
subtitle: 'An introduction to Postgres replication and change data capture.'
|
|
sidebar_label: 'Overview'
|
|
---
|
|
|
|
Database replication is the process of copying changes from your database to another location. It's also referred to as change data capture (CDC): capturing all the changes that occur to your data.
|
|
|
|
Supabase uses **Postgres logical replication**, a native Postgres feature that streams changes from your database's Write-Ahead Log (WAL) to external destinations in near real-time.
|
|
|
|
## Use cases
|
|
|
|
You might use database replication for:
|
|
|
|
- **Analytics and Data Warehousing**: Replicate your operational database to analytics platforms for complex analysis without impacting your application's performance.
|
|
- **Data Integration**: Keep your data synchronized across different systems and services in your tech stack.
|
|
- **Backup and Disaster Recovery**: Maintain up-to-date copies of your data in different locations.
|
|
|
|
## Destination types
|
|
|
|
Supabase supports replicating your database to two types of destinations:
|
|
|
|
### Managed destinations
|
|
|
|
<Admonition type="caution" label="Private Alpha">
|
|
|
|
Managed destinations are currently in private alpha. Access is limited and features may change.
|
|
|
|
</Admonition>
|
|
|
|
Supabase provides managed replication to popular data platforms powered by [Supabase ETL](https://github.com/supabase/etl), an open source tool for Postgres logical replication. Configure your destination through the Dashboard, and Supabase ETL handles the replication pipeline for you.
|
|
|
|
**Supported managed destinations:**
|
|
|
|
- **[Analytics Buckets](/docs/guides/storage/analytics/introduction)**: Specialized storage buckets in Supabase Storage designed for analytical workloads. They provide S3-compatible storage and use the Apache Iceberg open table format.
|
|
- **BigQuery**: Google's fully managed data warehouse
|
|
|
|
<Admonition type="note">
|
|
|
|
Destination availability varies based on our planned roll-out strategy. The destinations you can access depend on your project and access level.
|
|
|
|
</Admonition>
|
|
|
|
- [Set up managed destinations](/docs/guides/database/replication/etl-replication-setup)
|
|
|
|
### Manual destinations
|
|
|
|
Configure your own replication destination using external tools and Postgres's native logical replication. This gives you full control over the replication process and allows you to replicate to any system that supports Postgres logical replication.
|
|
|
|
- [Set up manual destinations](/docs/guides/database/replication/manual-replication-setup)
|
|
|
|
## Related features
|
|
|
|
Choose the data syncing method based on your use case:
|
|
|
|
- For realtime features and syncing data to clients (browsers, mobile apps), see [Realtime](/docs/guides/realtime)
|
|
- For deploying read-only databases across multiple regions, see [Read Replicas](/docs/guides/platform/read-replicas)
|
|
|
|
## Concepts and terms
|
|
|
|
### Write-Ahead Log (WAL)
|
|
|
|
Postgres uses a system called the Write-Ahead Log (WAL) to manage changes to the database. As you make changes, they are appended to the WAL, which is a series of files (also called "segments") where the file size can be specified. Once one segment is full, Postgres will start appending to a new segment. After a period of time, a checkpoint occurs and Postgres synchronizes the WAL with your database. Once the checkpoint is complete, then the WAL files can be removed from disk and free up space.
|
|
|
|
### Logical replication and WAL
|
|
|
|
Logical replication is a method of replication where Postgres uses the WAL files and transmit those changes to another Postgres database, or a system that supports reading WAL files.
|
|
|
|
### LSN
|
|
|
|
LSN is a Log Sequence Number that is used to identify the position of a WAL file in the WAL directory. It is often used to determine the progress of replication in subscribers and calculate the lag of a replication slot.
|
|
|
|
## Logical replication architecture
|
|
|
|
When setting up logical replication, three key components are involved:
|
|
|
|
- `publication` - A set of tables on your primary database that will be `published`
|
|
- `replication slot` - A slot used for replicating the data from a single publication. The slot, when created, will specify the output format of the changes
|
|
- `subscription` - A subscription is created from an external system (i.e. another Postgres database) and must specify the name of the `publication`. If you do not specify a replication slot, one is automatically created
|
|
|
|
## Logical replication output format
|
|
|
|
Logical replication is typically output in 2 forms, `pgoutput` and `wal2json`. The output method is how Postgres sends changes to any active replication slot.
|
|
|
|
## Logical replication configuration
|
|
|
|
When using logical replication, Postgres is then configured to keep WAL files around for longer than it needs them. If the files are removed too quickly, then your `replication slot` will become inactive and, if the database receives a large number of changes in a short time, then the `replication slot` can become lost as it was not able to keep up.
|
|
|
|
In order to mitigate this, Postgres has many options and settings that can be [tweaked](/docs/guides/database/custom-postgres-config) to manage the WAL usage effectively. Not all of these settings are user configurable as they can impact the stability of your database. For those that are, these should be considered as advanced configuration and not changed without understanding that they can cause additional disk space and resources to be used, as well as incur additional costs.
|
|
|
|
| Setting | Description | User-facing | Default |
|
|
| ---------------------------------------------------------------------------------------- | ------------------------------------------------------ | ----------- | ------- |
|
|
| [`max_replication_slots`](https://postgresqlco.nf/doc/en/param/max_replication_slots/) | Max count of replication slots allowed | No | |
|
|
| [`wal_keep_size`](https://postgresqlco.nf/doc/en/param/wal_keep_size/) | Minimum size of WAL files to keep for replication | No | |
|
|
| [`max_slot_wal_keep_size`](https://postgresqlco.nf/doc/en/param/max_slot_wal_keep_size/) | Max WAL size that can be reserved by replication slots | No | |
|
|
| [`checkpoint_timeout`](https://postgresqlco.nf/doc/en/param/checkpoint_timeout/) | Max time between WAL checkpoints | No | |
|