mirror of
https://github.com/supabase/supabase.git
synced 2026-10-05 01:15:03 +03:00
Miranda Limonczenko
7ce4ee53ae
Closes [DOCS-1289](https://linear.app/supabase/issue/DOCS-1289/get-the-linter-to-fix-what-it-flags-or-retirereplace-the-linter) Stacked on #50600, which points contributors at the authoring skills. Merge that one first. ## Problem Contributors experienced friction with the linter. They felt nickle and dimed for tiny nits and felt detracted from the work itself. PRs would become noisy with tiny one-word suggestions. Additionally, our homegrown linter is not very intelligent, causing frequent overrides. ## Solution This removes the linter entirely in favor of directing contributors to use SKILLS instead. The removal entails... - **CI.** Delete the three `docs_lint` workflows: the PR check, the external-PR comment companion, and the nightly `--fix` bot. Drop the stale `zizmor.yml` ignore entry for the deleted workflow. - **Tooling.** Delete `supa-mdx-lint.config.toml` and the 14 rule files. Drop the `lint:mdx` script and the `@supabase/supa-mdx-lint` dependency from docs, learn, and ui-library, and regenerate the lockfile. - **Content.** Remove the 181 directives. A separate commit carries Prettier's reformatting of the tables and blank lines those comments had suppressed, so the deletion commit stays readable. No prose changes. - **Style guide.** The word list states each rule directly instead of describing what the linter flagged. Every term survives, including the phrase groups that mirrored `Rule004ExcludeWords`. - **Skills.** `write-the-docs`, `edit-the-docs`, and `review-the-docs` drop `pnpm lint:mdx` from their self-review commands and check the word list directly. `ask-the-docs`'s CI reference drops both workflows. ## Manual testing 1. Run `git grep -i supa-mdx-lint -- . ':!pnpm-lock.yaml'`. No matches. 2. Run `pnpm install --frozen-lockfile --lockfile-only`. It passes, so the lockfile matches the three trimmed manifests. 3. Run `git diff master...HEAD --name-only --diff-filter=ACMR | grep -E '\.(md|mdx)$' | xargs npx prettier --config prettier.config.mjs --check`. All changed markdown passes. 4. Open the [reformatted filter table](https://docs-git-docs-retire-mdx-linter-supabase.vercel.app/docs/guides/observability/logs#filter-events) on the preview and compare it with [production](https://supabase.com/docs/guides/observability/logs#filter-events). The table renders the same. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Documentation guidance now uses manual prose and terminology review with the shared word list. * Clarified storage configuration and common Realtime channel mistakes. * Improved table formatting, text wrapping, and selected reference links. * Updated documentation authoring and review guidance. * **Chores** * Retired automated MDX linting from workflows and local validation commands. * Removed lint-suppression markers throughout documentation without changing instructions. * Added targeted documentation review guidance for pull requests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
98 lines
7.6 KiB
Plaintext
98 lines
7.6 KiB
Plaintext
---
|
|
id: 'replication'
|
|
title: 'Database replication'
|
|
description: 'Compare read replicas, Supabase Pipelines, and manual replication.'
|
|
subtitle: 'An introduction to database replication and change data capture.'
|
|
sidebar_label: 'Overview'
|
|
---
|
|
|
|
Replication keeps data synchronized with another location. Logical replication products such as Supabase Pipelines use change data capture (CDC) to read database changes and apply them to a destination.
|
|
|
|
## Replication methods
|
|
|
|
Supabase supports three replication methods. Choose based on whether you need another Supabase Postgres database, a managed replication pipeline to a destination system, or full control over your own logical replication setup.
|
|
|
|
### Read replicas
|
|
|
|
Read replicas are additional Supabase Postgres databases kept in sync with your primary database. Use them when you want read-only query capacity, lower latency in another region, or to isolate analytical reads from application writes while staying inside Supabase Postgres.
|
|
|
|
See [Set up read replicas](/docs/guides/platform/read-replicas).
|
|
|
|
### Pipelines
|
|
|
|
<$Partial path="pipelines-public-alpha.mdx" />
|
|
|
|
Supabase Pipelines is a managed CDC product for moving data from Supabase Postgres to supported destination systems. It uses Postgres logical replication with the open-source [Supabase ETL engine](https://github.com/supabase/etl). A destination is where your replicated data is stored; a pipeline copies existing rows for the tables selected for initial sync, then uses ongoing replication (CDC) to send subsequent database changes to that destination.
|
|
|
|
See [Set up Pipelines](/docs/guides/database/replication/pipelines).
|
|
|
|
### Manual replication
|
|
|
|
Manual replication uses the same underlying Postgres logical replication features as Pipelines, but you configure and operate the pieces yourself. Use this path when you want to connect tools such as Airbyte, Estuary, Fivetran, Materialize, Stitch, AWS DMS, or another system that supports Postgres logical replication.
|
|
|
|
See [Set up manual replication](/docs/guides/database/replication/manual-replication-setup).
|
|
|
|
## Supported destinations
|
|
|
|
| Destination | Status |
|
|
| ---------------------------------------------------------- | ------------- |
|
|
| [BigQuery](/docs/guides/database/replication/bigquery) | Public alpha |
|
|
| [ClickHouse](/docs/guides/database/replication/clickhouse) | Private alpha |
|
|
| [DuckLake](/docs/guides/database/replication/ducklake) | Private alpha |
|
|
| [Snowflake](/docs/guides/database/replication/snowflake) | Private alpha |
|
|
|
|
[Request access](/go/supabase-pipelines-new-destinations) to destinations in private alpha.
|
|
|
|
## Use cases
|
|
|
|
You might use database replication for:
|
|
|
|
- **Analytics and data warehousing**: Run analytical queries on replicated data in a separate platform. Initial sync and ongoing replication still use resources on the source database.
|
|
- **Data integration**: Keep your data synchronized across different systems and services in your tech stack.
|
|
- **Operational reporting**: Maintain a copy of selected application data that you can query in another system.
|
|
|
|
## Related features
|
|
|
|
For realtime features and syncing data to browsers and mobile apps, see [Realtime](/docs/guides/realtime).
|
|
|
|
Realtime also uses Postgres changes, but it is intended for broadcasting database updates to clients rather than maintaining a copy of your database in another system.
|
|
|
|
## Concepts and terms
|
|
|
|
### Write-Ahead Log (WAL)
|
|
|
|
Postgres records database changes in the Write-Ahead Log (WAL) before writing them to data files. WAL is stored in files called segments. A checkpoint writes modified data pages to disk so crash recovery can start from a recent position. Older WAL segments can be recycled or removed once they are no longer needed for recovery, archiving, or replication.
|
|
|
|
### Logical replication and WAL
|
|
|
|
Logical replication is a method of replication where Postgres uses WAL files to transmit changes to another Postgres database, or to a system that supports reading WAL files.
|
|
|
|
### LSN
|
|
|
|
LSN is a Log Sequence Number that identifies a position in the WAL. It is often used to determine the progress of replication in subscribers and calculate the lag of a replication slot.
|
|
|
|
## Logical replication architecture
|
|
|
|
Logical replication uses these components:
|
|
|
|
- **Publication**: Defines the source tables and change types to publish, with optional column lists and row filters.
|
|
- **Replication slot**: Tracks a consumer's progress and retains WAL it still needs. A slot uses an output plugin to decode changes; it is not tied to a single publication.
|
|
- **Consumer**: Reads decoded changes and applies them to a destination. Another Postgres database can use a subscription to manage this connection and its publications. Pipelines connects directly to the replication stream without creating a Postgres subscription.
|
|
|
|
## Logical replication output format
|
|
|
|
Logical replication is typically output in two forms, `pgoutput` and `wal2json`. The output method is how Postgres sends changes to any active replication slot.
|
|
|
|
## Logical replication configuration
|
|
|
|
Logical replication slots retain WAL until their consumers confirm that they no longer need it. If Postgres removes required WAL before a consumer catches up, the slot can become unusable and the consumer may need a new initial copy.
|
|
|
|
Postgres settings control replication capacity and WAL retention. Higher retention limits reduce the risk of losing required WAL, but can consume more database storage. Treat these settings as advanced configuration and check available disk before changing them. On Supabase, configure the supported settings with the [Supabase CLI](/docs/guides/database/custom-postgres-config#managing-postgres-configuration-with-the-cli).
|
|
|
|
| Setting | Description | Supabase configuration |
|
|
| ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- |
|
|
| [`max_replication_slots`](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-REPLICATION-SLOTS) | Maximum number of replication slots. Pipelines needs one main slot and can temporarily use one additional slot per active table-sync worker. | CLI only |
|
|
| [`wal_keep_size`](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-WAL-KEEP-SIZE) | Minimum amount of old WAL retained for standby servers. This setting is separate from the per-slot retention limit. | CLI only |
|
|
| [`max_slot_wal_keep_size`](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-SLOT-WAL-KEEP-SIZE) | Maximum WAL that replication slots can retain at checkpoint time. `-1` lets slots retain unlimited WAL. A finite limit can invalidate a slot when its consumer falls too far behind. | CLI only |
|
|
| [`checkpoint_timeout`](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-CHECKPOINT-TIMEOUT) | Maximum time between automatic WAL checkpoints. Slot WAL limits are enforced at checkpoint time. | CLI only |
|