Files
supabase/apps/docs/content/guides/database/replication/pipelines-faq.mdx
T

105 lines
7.6 KiB
Plaintext

---
id: 'pipelines-faq'
title: 'Pipelines FAQ'
description: 'Frequently asked questions about Supabase Pipelines.'
subtitle: 'Common questions and answers about Supabase Pipelines.'
sidebar_label: 'FAQ'
---
<$Partial path="pipelines-public-alpha.mdx" />
{/* supa-mdx-lint-disable-next-line Rule001HeadingCase */}
## Which plans support Pipelines?
Pipelines requires a Pro, Team, or Enterprise plan. During public alpha, availability varies by organization; an eligible plan does not guarantee access. If unavailable, request access from **Database > Replication** or contact your account manager.
## What destinations are supported?
BigQuery is in public alpha. ClickHouse, DuckLake, and Snowflake are in private alpha and require access approval. See [supported destinations](/docs/guides/database/replication#supported-destinations) for their status and access request.
## Does the destination's region affect performance?
Yes. Distance between the source, pipeline, and destination adds network latency and can reduce throughput. Choose resources near the [managed pipeline region](/docs/guides/database/replication/pipelines#region), prioritizing the destination if you can optimize only one side.
{/* supa-mdx-lint-disable-next-line Rule001HeadingCase */}
## What does Pipelines install in the database?
Pipelines installs objects in your project's Postgres database to track replication and support schema changes:
- An `etl` schema containing internal tables for replication state and progress, source table schemas, destination mappings, and migration history.
- Helper functions in `etl` that read table definitions and prepare schema-change messages.
- A database event trigger, `supabase_etl_ddl_message_trigger`, that runs after `ALTER TABLE` and `ALTER PUBLICATION`. It writes schema-change information for published tables to the write-ahead log (WAL), so Pipelines can process supported changes alongside row changes.
Each pipeline also uses replication slots to track its position in WAL. See [initial sync and table-sync slots](/docs/guides/database/replication/pipelines-monitoring#initial-sync-and-table-sync-slots) for how these retain changes during replication.
<Admonition type="caution">
The `etl` schema is reserved for Pipelines. If your application already uses a schema named `etl`, rename it and update references to it before enabling Pipelines. Do not add application objects to the Pipelines-managed schema or include it in a publication.
</Admonition>
To remove the installed objects, delete all pipelines, then [disable Pipelines](#what-happens-when-you-disable-pipelines).
{/* supa-mdx-lint-disable-next-line Rule001HeadingCase */}
## What does Pipelines check before creating a pipeline?
The Dashboard validates source access, replication capacity, publication tables, and destination connectivity and requirements. **Required** issues block creation; **Warnings** require review. See [creation checks](/docs/guides/database/replication/pipelines#creation-checks).
## What schema changes are supported?
Support differs by destination. Check the [schema-change guides](/docs/guides/database/replication/pipelines#schema-change-support) before changing column types, defaults, keys, or constraints.
## Can data be processed more than once?
Yes. Pipelines provides at-least-once processing: recovery can replay acknowledged data, and consumers of append-only histories must tolerate repeated events. Destination deduplication does not provide an exactly-once processing guarantee.
See the data models for [BigQuery](/docs/guides/database/replication/bigquery#how-it-works), [ClickHouse](/docs/guides/database/replication/clickhouse#choose-a-table-engine), [DuckLake](/docs/guides/database/replication/ducklake#how-replication-works), and [Snowflake](/docs/guides/database/replication/snowflake#append-only-change-history), and the [billing implications](/docs/guides/platform/manage-your-usage/pipelines#how-data-processed-is-measured).
## Why is a table not being replicated?
Check that:
- The table is published. If you added it after starting the pipeline, [restart the pipeline to discover it](/docs/guides/database/replication/pipelines#adding-or-removing-tables).
- The table and published columns satisfy the destination's primary-key and replica-identity requirements.
- [Publication row filters](/docs/guides/database/replication/pipelines#filtering-rows-with-a-predicate) include the expected rows.
- Missing columns aren't generated; Pipelines skips generated columns.
If new changes arrive but existing rows are missing, check the [initial sync selection](/docs/guides/database/replication/pipelines#choosing-which-tables-to-copy). [Changing a row filter](/docs/guides/database/replication/pipelines#other-publication-changes) also does not copy newly included historical rows.
If inserts work but updates or deletes fail, review the source table requirements for [BigQuery](/docs/guides/database/replication/bigquery#source-table-requirements), [ClickHouse](/docs/guides/database/replication/clickhouse#source-table-requirements), [DuckLake](/docs/guides/database/replication/ducklake#source-table-requirements), or [Snowflake](/docs/guides/database/replication/snowflake#source-table-requirements).
## Why is a table in error state?
An initial sync or ongoing replication operation failed, for example after an unsupported schema change. Some errors retry automatically; others require restarting replication for the affected tables from scratch. See [Table errors](/docs/guides/database/replication/pipelines-monitoring#table-errors) for diagnosis and [Restarting tables](/docs/guides/database/replication/pipelines-monitoring#restarting-tables) for the procedure and its effects.
## Why is a pipeline failed or stopped?
**Failed** reports a startup or runtime error and can recover automatically. **Stopped** requires a manual start. Follow [pipeline error recovery](/docs/guides/database/replication/pipelines-monitoring#pipeline-errors) to diagnose the cause before restarting.
## Why is replication lag increasing?
Postgres is producing WAL faster than Pipelines confirms progress. Follow [Investigate the lag](/docs/guides/database/replication/pipelines-monitoring#investigate-the-lag) to distinguish source activity, destination throughput, and connectivity problems.
## What does a `Lost` slot status mean?
Required WAL is gone, so replication cannot continue from that slot. Follow [slot recovery](/docs/guides/database/replication/pipelines-monitoring#respond-based-on-the-slot-status); recovery scope depends on whether the main slot or a table-sync slot was lost.
## What happens if a table is deleted at the destination?
Deleting or modifying managed objects can stop replication and require a new initial sync. To remove one safely, [remove the source table from the publication](/docs/guides/database/replication/pipelines#removing-tables-from-replication) and restart the pipeline before deleting its destination table.
## What happens when a project becomes inactive or moves to the Free Plan?
Project inactivity stops its pipelines. Start them manually after restarting the project. Downgrading to the Free Plan deletes its pipelines.
{/* supa-mdx-lint-disable-next-line Rule001HeadingCase */}
## What happens when you disable Pipelines?
Disabling Pipelines removes its database event trigger and the entire Pipelines-managed `etl` schema, including its tables and helper functions. Your source application tables and existing destination data remain.
Delete all pipelines first, then follow [Disabling Pipelines](/docs/guides/database/replication/pipelines#disabling-pipelines). Deleting the pipelines alone does not remove the shared database installation.