mirror of
https://github.com/supabase/supabase.git
synced 2026-10-07 10:25:06 +03:00
## Problem Pipeline destination guides live beside the Pipelines overview, so their sidebar hierarchy and URLs do not reflect that they belong to Pipelines. ## Solution Moves the BigQuery, ClickHouse, DuckLake, and Snowflake guides under `/database/replication/pipelines/`, redirects the old URLs in both docs preview (`apps/docs/next.config.mjs`) and production (`apps/www/lib/redirects.js`), and updates internal documentation links. The matching Studio changes, including destination-aware links from the creation sheet, will follow in a separate PR. ## To test - [Pipelines overview](https://docs-git-dnywh-docsnest-pipeline-destinations-supabase.vercel.app/docs/guides/database/replication/pipelines) - Destination guides: [BigQuery](https://docs-git-dnywh-docsnest-pipeline-destinations-supabase.vercel.app/docs/guides/database/replication/pipelines/bigquery), [ClickHouse](https://docs-git-dnywh-docsnest-pipeline-destinations-supabase.vercel.app/docs/guides/database/replication/pipelines/clickhouse), [DuckLake](https://docs-git-dnywh-docsnest-pipeline-destinations-supabase.vercel.app/docs/guides/database/replication/pipelines/ducklake), [Snowflake](https://docs-git-dnywh-docsnest-pipeline-destinations-supabase.vercel.app/docs/guides/database/replication/pipelines/snowflake) - [Old Snowflake URL](https://docs-git-dnywh-docsnest-pipeline-destinations-supabase.vercel.app/docs/guides/database/replication/snowflake) ## Review instructions 1. Open the Pipelines overview and confirm the four destination guides appear beneath Pipelines in the sidebar. 2. Open each destination guide and confirm its nested URL and content load correctly. 3. Open the old Snowflake URL and confirm it redirects to its new location. ## Checklist - [x] I have read [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) - [x] If I wrote a new docs topic or edited an existing topic, I used the `/write-the-docs` or `/edit-the-docs` skill, which references [WORD_LIST](https://github.com/supabase/supabase/blob/master/apps/docs/WORD_LIST.md) and the docs [CONTRIBUTING](https://github.com/supabase/supabase/blob/master/apps/docs/CONTRIBUTING.md) guide <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Moved BigQuery, ClickHouse, DuckLake, and Snowflake replication guides to a dedicated pipelines section and updated related navigation and links. * **Bug Fixes** * Added permanent redirects so existing links to the four destination guides continue to work. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
97 lines
7.4 KiB
Plaintext
97 lines
7.4 KiB
Plaintext
---
|
|
id: 'pipelines-faq'
|
|
title: 'Pipelines FAQ'
|
|
description: 'Frequently asked questions about Supabase Pipelines.'
|
|
subtitle: 'Common questions and answers about Supabase Pipelines.'
|
|
sidebar_label: 'FAQ'
|
|
---
|
|
|
|
<$Partial path="pipelines-public-alpha.mdx" />
|
|
|
|
## Which plans support Pipelines?
|
|
|
|
Pipelines requires a Pro, Team, or Enterprise plan. During public alpha, availability varies by organization; an eligible plan does not guarantee access. If unavailable, request access from **Database > Pipelines** or contact your account manager.
|
|
|
|
## What destinations are supported?
|
|
|
|
BigQuery is in public alpha. ClickHouse, DuckLake, and Snowflake are in private alpha and require access approval. See [supported destinations](/docs/guides/database/replication#supported-destinations) for their status and access request.
|
|
|
|
## Does the destination's region affect performance?
|
|
|
|
Yes. Distance between the source, pipeline, and destination adds network latency and can reduce throughput. Choose resources near the [managed pipeline region](/docs/guides/database/replication/pipelines#region), prioritizing the destination if you can optimize only one side.
|
|
|
|
## What does Pipelines install in the database?
|
|
|
|
Pipelines installs objects in your project's Postgres database to track replication and support schema changes:
|
|
|
|
- An `etl` schema containing internal tables for replication state and progress, source table schemas, destination mappings, and migration history.
|
|
- Helper functions in `etl` that read table definitions and prepare schema-change messages.
|
|
- A database event trigger, `supabase_etl_ddl_message_trigger`, that runs after `ALTER TABLE` and `ALTER PUBLICATION`. It writes schema-change information for published tables to the write-ahead log (WAL), so Pipelines can process supported changes alongside row changes.
|
|
|
|
Each pipeline also uses replication slots to track its position in WAL. See [initial sync and table-sync slots](/docs/guides/database/replication/pipelines-monitoring#initial-sync-and-table-sync-slots) for how these retain changes during replication.
|
|
|
|
<Admonition type="caution">
|
|
|
|
The `etl` schema is reserved for Pipelines. If your application already uses a schema named `etl`, rename it and update references to it before enabling Pipelines. Do not add application objects to the Pipelines-managed schema or include it in a publication.
|
|
|
|
</Admonition>
|
|
|
|
To remove the installed objects, delete all pipelines, then [disable Pipelines](#what-happens-when-you-disable-pipelines).
|
|
|
|
## What does Pipelines check before creating a pipeline?
|
|
|
|
The Dashboard validates source access, replication capacity, publication tables, and destination connectivity and requirements. **Required** issues block creation; **Warnings** require review. See [creation checks](/docs/guides/database/replication/pipelines#creation-checks).
|
|
|
|
## What schema changes are supported?
|
|
|
|
Support differs by destination. Check the [schema-change guides](/docs/guides/database/replication/pipelines#schema-change-support) before changing column types, defaults, keys, or constraints.
|
|
|
|
## Can data be processed more than once?
|
|
|
|
Yes. Pipelines provides at-least-once processing: recovery can replay acknowledged data, and consumers of append-only histories must tolerate repeated events. Destination deduplication does not provide an exactly-once processing guarantee.
|
|
|
|
See the data models for [BigQuery](/docs/guides/database/replication/pipelines/bigquery#how-it-works), [ClickHouse](/docs/guides/database/replication/pipelines/clickhouse#choose-a-table-engine), [DuckLake](/docs/guides/database/replication/pipelines/ducklake#how-replication-works), and [Snowflake](/docs/guides/database/replication/pipelines/snowflake#append-only-change-history), and the [billing implications](/docs/guides/platform/manage-your-usage/pipelines#how-data-processed-is-measured).
|
|
|
|
## Why is a table not being replicated?
|
|
|
|
Check that:
|
|
|
|
- The table is published. If you added it after starting the pipeline, [restart the pipeline to discover it](/docs/guides/database/replication/pipelines#adding-or-removing-tables).
|
|
- The table and published columns satisfy the destination's primary-key and replica-identity requirements.
|
|
- [Publication row filters](/docs/guides/database/replication/pipelines#filtering-rows-with-a-predicate) include the expected rows.
|
|
- Missing columns aren't generated; Pipelines skips generated columns.
|
|
|
|
If new changes arrive but existing rows are missing, check the [initial sync selection](/docs/guides/database/replication/pipelines#choosing-which-tables-to-copy). [Changing a row filter](/docs/guides/database/replication/pipelines#other-publication-changes) also does not copy newly included historical rows.
|
|
|
|
If inserts work but updates or deletes fail, review the source table requirements for [BigQuery](/docs/guides/database/replication/pipelines/bigquery#source-table-requirements), [ClickHouse](/docs/guides/database/replication/pipelines/clickhouse#source-table-requirements), [DuckLake](/docs/guides/database/replication/pipelines/ducklake#source-table-requirements), or [Snowflake](/docs/guides/database/replication/pipelines/snowflake#source-table-requirements).
|
|
|
|
## Why is a table in error state?
|
|
|
|
An initial sync or ongoing replication operation failed, for example after an unsupported schema change. Some errors retry automatically; others require restarting replication for the affected tables from scratch. See [Table errors](/docs/guides/database/replication/pipelines-monitoring#table-errors) for diagnosis and [Restarting tables](/docs/guides/database/replication/pipelines-monitoring#restarting-tables) for the procedure and its effects.
|
|
|
|
## Why is a pipeline failed or stopped?
|
|
|
|
**Failed** reports a startup or runtime error and can recover automatically. **Stopped** requires a manual start. Follow [pipeline error recovery](/docs/guides/database/replication/pipelines-monitoring#pipeline-errors) to diagnose the cause before restarting.
|
|
|
|
## Why is replication lag increasing?
|
|
|
|
Postgres is producing WAL faster than Pipelines confirms progress. Follow [Investigate the lag](/docs/guides/database/replication/pipelines-monitoring#investigate-the-lag) to distinguish source activity, destination throughput, and connectivity problems.
|
|
|
|
## What does a `Lost` slot status mean?
|
|
|
|
Required WAL is gone, so replication cannot continue from that slot. Follow [slot recovery](/docs/guides/database/replication/pipelines-monitoring#respond-based-on-the-slot-status); recovery scope depends on whether the main slot or a table-sync slot was lost.
|
|
|
|
## What happens if a table is deleted at the destination?
|
|
|
|
Deleting or modifying managed objects can stop replication and require a new initial sync. To remove one safely, [remove the source table from the publication](/docs/guides/database/replication/pipelines#removing-tables-from-replication) and restart the pipeline before deleting its destination table.
|
|
|
|
## What happens when a project becomes inactive or moves to the Free Plan?
|
|
|
|
Project inactivity stops its pipelines. Start them manually after restarting the project. Downgrading to the Free Plan deletes its pipelines.
|
|
|
|
## What happens when you disable Pipelines?
|
|
|
|
Disabling Pipelines removes its database event trigger and the entire Pipelines-managed `etl` schema, including its tables and helper functions. Your source application tables and existing destination data remain.
|
|
|
|
Delete all pipelines first, then follow [Disabling Pipelines](/docs/guides/database/replication/pipelines#disabling-pipelines). Deleting the pipelines alone does not remove the shared database installation.
|