diff --git a/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts b/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts index 3e72297149e..dfbd3ef9d71 100644 --- a/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts +++ b/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts @@ -2480,7 +2480,17 @@ export const platform: NavMenuConstant = { { name: 'Custom Domains', url: '/guides/platform/custom-domains' }, { name: 'Database Backups', url: '/guides/platform/backups' }, { name: 'IPv4 Address', url: '/guides/platform/ipv4-address' }, - { name: 'Read Replicas', url: '/guides/platform/read-replicas' }, + { + name: 'Read Replicas', + url: '/guides/platform/read-replicas', + items: [ + { name: 'Overview', url: '/guides/platform/read-replicas' as `/${string}` }, + { + name: 'Getting started', + url: '/guides/platform/read-replicas/getting-started' as `/${string}`, + }, + ], + }, ], }, { diff --git a/apps/docs/content/guides/platform/manage-your-usage/read-replicas.mdx b/apps/docs/content/guides/platform/manage-your-usage/read-replicas.mdx index 503f697ac01..b62147c9ed3 100644 --- a/apps/docs/content/guides/platform/manage-your-usage/read-replicas.mdx +++ b/apps/docs/content/guides/platform/manage-your-usage/read-replicas.mdx @@ -5,7 +5,13 @@ title: 'Manage Read Replica usage' ## What you are charged for -Each [Read Replica](/docs/guides/platform/read-replicas) is a dedicated database. You are charged for its resources: [Compute](/docs/guides/platform/compute-and-disk#compute), [Disk Size](/docs/guides/platform/database-size#disk-size), provisioned [Disk IOPS](/docs/guides/platform/compute-and-disk#provisioned-disk-throughput-and-iops), provisioned [Disk Throughput](/docs/guides/platform/compute-and-disk#provisioned-disk-throughput-and-iops), and [IPv4](/docs/guides/platform/ipv4-address). +Each [Read Replica](/docs/guides/platform/read-replicas) is a dedicated database. You are charged for its resources, which are the following, and mirrored from the primary database: + +- [Compute](/docs/guides/platform/compute-and-disk#compute) +- [Disk Size](/docs/guides/platform/database-size#disk-size) +- Provisioned [Disk IOPS](/docs/guides/platform/compute-and-disk#provisioned-disk-throughput-and-iops) +- Provisioned [Disk Throughput](/docs/guides/platform/compute-and-disk#provisioned-disk-throughput-and-iops) +- [IPv4](/docs/guides/platform/ipv4-address). @@ -13,26 +19,37 @@ Read Replicas are **not** covered by the [Spend Cap](/docs/guides/platform/cost- -## How charges are calculated +## How we calculate charges Read Replica charges are the total of the charges listed below. -**Compute** +### Compute + Compute is charged by the hour, meaning you are charged for the exact number of hours that a Read Replica is running and, therefore, incurring Compute usage. If a Read Replica runs for part of an hour, you are still charged for the full hour. Read Replicas run on the same Compute size as the primary database. -**Disk Size** -Refer to [Manage Disk Size usage](/docs/guides/platform/manage-your-usage/disk-size) for details on how charges are calculated. The disk size of a Read Replica is 1.25x the size of the primary disk to account for WAL archives. With a Read Replica you go beyond your subscription plan's quota for Disk Size. +### Disk size -**Provisioned Disk IOPS (optional)** -Read Replicas inherit any additional provisioned Disk IOPS from the primary database. Refer to [Manage Disk IOPS usage](/docs/guides/platform/manage-your-usage/disk-iops) for details on how charges are calculated. +Read [the Manage Disk Size usage guide](/docs/guides/platform/manage-your-usage/disk-size) for details on how we calculate charges. The disk size of a Read Replica is 1.25x the size of the primary disk to account for WAL archives. With a Read Replica you go beyond your subscription plan's quota for Disk Size. -**Provisioned Disk Throughput (optional)** -Read Replicas inherit any additional provisioned Disk Throughput from the primary database. Refer to [Manage Disk Throughput usage](/docs/guides/platform/manage-your-usage/disk-throughput) for details on how charges are calculated. +{/* supa-mdx-lint-disable-next-line Rule001HeadingCase */} -**IPv4 (optional)** -If the primary database has a configured IPv4 address, its Read Replicas are also assigned one, with charges for each. Refer to [Manage IPv4 usage](/docs/guides/platform/manage-your-usage/ipv4) for details on how charges are calculated. +### Provisioned Disk IOPS (optional) + +Read Replicas inherit any additional provisioned Disk IOPS from the primary database. Read the [Manage Disk IOPS usage guide](/docs/guides/platform/manage-your-usage/disk-iops) for details on how we calculate charges. + +{/* supa-mdx-lint-disable-next-line Rule001HeadingCase */} + +### Provisioned Disk Throughput (optional) + +Read Replicas inherit any additional provisioned Disk Throughput from the primary database. Read the [Manage Disk Throughput usage guide](/docs/guides/platform/manage-your-usage/disk-throughput) for details on how we calculate charges. + +{/* supa-mdx-lint-enable-next-line Rule001HeadingCase */} + +### IPv4 (optional) + +If the primary database has configured an IPv4 address add-on, its Read Replicas are also assigned one, with charges for each. Read the [Manage IPv4 usage guide](/docs/guides/platform/manage-your-usage/ipv4) for details on how we calculate charges. ### Usage on your invoice @@ -42,7 +59,7 @@ Compute incurred by Read Replicas is shown as "Replica Compute Hours" on your in ### No additional resources configured -The project has one Read Replica and no IPv4 and no additional Disk IOPS and Disk Throughput configured. +The project has one Read Replica, no IPv4, and no additional Disk IOPS and Disk Throughput configured. | Line Item | Units | Costs | | ----------------------------- | --------- | --------------------------- | @@ -60,7 +77,7 @@ The project has one Read Replica and no IPv4 and no additional Disk IOPS and Dis ### Additional resources configured -The project has two Read Replicas and IPv4 and additional Disk IOPS and Disk Throughput configured. +The project has two Read Replicas, IPv4, and additional Disk IOPS and Disk Throughput configured. | Line Item | Units | Costs | | ----------------------------- | --------- | ---------------------------- | diff --git a/apps/docs/content/guides/platform/read-replicas.mdx b/apps/docs/content/guides/platform/read-replicas.mdx index 3c2eefe23b2..93764c73784 100644 --- a/apps/docs/content/guides/platform/read-replicas.mdx +++ b/apps/docs/content/guides/platform/read-replicas.mdx @@ -4,7 +4,7 @@ description: 'Deploy read-only databases across multiple regions, for lower late subtitle: 'Deploy read-only databases across multiple regions, for lower latency and better resource management.' --- -Read Replicas are additional databases that are kept in sync with your Primary database. You can read your data from a Read Replica, which helps with: +Read Replicas are additional databases kept in sync with your Primary database. You can read your data from a Read Replica, which helps with: - **Load balancing:** Read Replicas reduce load on the Primary database. For example, you can use a Read Replica for complex analytical queries and reserve the Primary for user-facing create, update, and delete operations. - **Improved latency:** For projects with a global user base, additional databases can be deployed closer to users to reduce latency. @@ -19,7 +19,7 @@ Read Replicas are additional databases that are kept in sync with your Primary d ## About Read Replicas -The database you start with when launching a Supabase project is your Primary database. Read Replicas are kept in sync with the Primary through a process called "replication." Replication is asynchronous to ensure that transactions on the Primary aren't blocked. There is a delay between an update on the Primary and the time that a Read Replica receives the change. This delay is called "replication lag." +The database you start with when launching a Supabase project is your Primary database. A process called "replication" keeps Read Replicas in sync with the Primary. Replication is asynchronous to ensure that transactions on the Primary aren't blocked. There is a delay between an update on the Primary and the time that a Read Replica receives the change. This delay is called "replication lag." You can only read data from a Read Replica. This is in contrast to a Primary database, where you can both read and write: @@ -28,73 +28,33 @@ You can only read data from a Read Replica. This is in contrast to a Primary dat | Primary | ✅ | ✅ | ✅ | ✅ | | Read Replica | ✅ | - | - | - | -## Prerequisites + +
+ - + When your database starts slowing down, you face a choice: make your existing database bigger (scale vertically), or spread the load across multiple databases (scale horizontally). Both approaches work. Neither is universally correct. The right answer depends on your workload, your budget, and where the bottleneck actually is. -Read Replicas are available for all projects on the Pro, Team and Enterprise plans. Spin one up now over at the [Infrastructure Settings page](/dashboard/project/_/settings/infrastructure). + Read Replicas decision flowchart - + -Projects must meet these requirements to use Read Replicas: +
-1. Running on AWS. -1. Running on at least a [Small compute add-on](/docs/guides/platform/compute-add-ons). - - Read Replicas are started on the same compute instance as the Primary to keep up with changes. -1. Running on Postgres 15+. - - For projects running on older versions of Postgres, you will need to [upgrade to the latest platform version](/docs/guides/platform/migrating-and-upgrading-projects#pgupgrade). -1. Using [physical backups](/docs/guides/platform/backups#point-in-time-recovery) - - Physical backups are automatically enabled if using [PITR](/docs/guides/platform/backups#point-in-time-recovery) - - If you're not using PITR, you'll be able to switch to physical backups as part of the Read Replica setup process. Note that physical backups can't be downloaded from the dashboard in the way logical backups can. - -## Getting started - -To add a Read Replica, go to the [Infrastructure Settings page](/dashboard/project/_/settings/infrastructure) in your dashboard. - -You can also manage Read Replicas using the Management API (beta functionality): - -```bash -# Get your access token from https://supabase.com/dashboard/account/tokens -export SUPABASE_ACCESS_TOKEN="your-access-token" -export PROJECT_REF="your-project-ref" - -# Create a new Read Replica -curl -X POST "https://api.supabase.com/v1/projects/$PROJECT_REF/read-replicas/setup" \ - -H "Authorization: Bearer $SUPABASE_ACCESS_TOKEN" \ - -H "Content-Type: application/json" \ - -d '{ - "region": "us-east-1" - }' - -# Delete a Read Replica -curl -X POST "https://api.supabase.com/v1/projects/$PROJECT_REF/read-replicas/remove" \ - -H "Authorization: Bearer $SUPABASE_ACCESS_TOKEN" \ - -H "Content-Type: application/json" \ - -d '{ - "database_identifier": "abcdefghijklmnopqrst" - }' -``` - -Projects on an XL compute add-on or larger can create up to five Read Replicas. Projects on compute add-ons smaller than XL can create up to two Read Replicas. All Read Replicas inherit the compute size of their Primary database. - -### Deploying a Read Replica - -A Read Replica is deployed by using a physical backup as a starting point, and a combination of WAL file archives and direct replication from the Primary database to catch up. Both components may take significant time to complete. The duration of restoring from a physical backup is roughly dependent and directly related to the database size of your project. The time taken to catch up to the primary using WAL archives and direct replication is dependent on the level of activity on the Primary database; a more active database will produce a larger number of WAL files that will need to be processed. - -Along with the progress of the deployment, the dashboard displays rough estimates for each component. - -{/* supa-mdx-lint-disable-next-line Rule001HeadingCase */} - -### What does it mean when "Init failed" is observed? - -The status `Init failed` indicates that the Read Replica has failed to deploy. Some possible scenarios as to why a Read Replica may have failed to be deployed: - -- Underlying instance failed to come up. -- Network issue leading to inability to connect to the Primary database. -- Possible incompatible database settings between the Primary and Read Replica databases. -- Platform issues. - -It is safe to drop this failed Read Replica, and in the event of a transient issue, attempt to spin up another one. If however spinning up Read Replicas for your project consistently fails, do check out our [status page](https://status.supabase.com) for any ongoing incidents, or open a support ticket [here](/dashboard/support/new). To aid the investigation, do not bring down the recently failed Read Replica. + ## Features @@ -104,14 +64,18 @@ Read Replicas offer the following features: Each Read Replica has its own dedicated database and API endpoints. -- Find the database endpoint on the projects [**Connect** panel](/dashboard/project/_?showConnect=true) -- Find the API endpoint on the [API Settings page](/dashboard/project/_/settings/api) under **Project URL** +- Find the database endpoint on the project's [**Connect** panel](/dashboard/project/_?showConnect=true). Toggle between Primary and Read Replicas using the **Source** dropdown. +- Find the API endpoint on the [API Settings page](/dashboard/project/_/settings/api) under **Project URL**. Toggle between Primary and Read Replicas using the **Source** dropdown. + +If you use an [IPv4 add-on](/docs/guides/platform/ipv4-address#read-replicas), the database endpoints for your Read Replicas also use an IPv4 add-on. Read Replicas only support `GET` requests from the [REST API](/docs/guides/api). If you are calling a read-only Postgres function through the REST API, make sure to set the `get: true` [option](/docs/reference/javascript/rpc?queryGroups=example&example=call-a-read-only-postgres-function). + + Requests to other Supabase products, such as Auth, Storage, and Realtime, aren't able to use a Read Replica or its API endpoint. Support for more products will be added in the future. -If you're using an [IPv4 add-on](/docs/guides/platform/ipv4-address#read-replicas), the database endpoints for your Read Replicas will also use an IPv4 add-on. + ### Dedicated connection pool @@ -119,11 +83,15 @@ A connection pool through Supavisor is also available for each Read Replica. Fin ### API load balancer -A load balancer is deployed to automatically balance requests between your Primary database and Read Replicas. Find its endpoint on the [API Settings page](/dashboard/project/_/settings/api). +A load balancer automatically balances requests between your Primary database and Read Replicas. Find its endpoint on the [**API Settings page**](/dashboard/project/_/settings/api). -The load balancer enables geo-routing for Data API requests so that `GET` requests will automatically be routed to the database that is closest to your user ensuring the lowest latency. Non-`GET` requests can also be sent through this endpoint, and will be routed to the Primary database. +The load balancer enables geo-routing for Data API requests to automatically route `GET` requests to the database closest to your user ensuring the lowest latency. You can also send Non-`GET` requests through this endpoint, and they are routed to the Primary database automatically. -You can also interact with Supabase services (Auth, Edge Functions, Realtime, and Storage) through this load balancer so there's no need to worry about which endpoint to use and in which situations. However, geo-routing for these services are not yet available but is coming soon. + + +You can also interact with other Supabase services (Auth, Edge Functions, Realtime, and Storage) through this load balancer so there's no need to worry about which endpoint to use and in which situations. Geo-routing for Auth, Realtime, and Storage aren't yet available but are coming soon. + + @@ -137,10 +105,10 @@ If you remove all Read Replicas from your project, the load balancer and its end -Starting on April 4th, 2025, we will be changing the routing behavior for eligible Data API requests: +From April 4th, 2025, the routing behavior for eligible Data API requests changed: -- Old behavior: Round-Robin distribution among all databases (all read replicas + primary) of your project, regardless of location -- New behavior: Geo-routing, that directs requests to the closest available database (all read replicas + primary) +- **Old behavior**: Round-Robin distribution among all databases (all read replicas + primary) of your project, regardless of location +- **New behavior**: Geo-routing, that directs requests to the closest available database (all read replicas + primary) The new behavior delivers a better experience for your users by minimizing the latency to your project. You can take full advantage of this by placing Read Replicas close to your major customer bases. @@ -186,74 +154,6 @@ We recommend ingesting your [project's metrics](/docs/guides/platform/metrics#ac All settings configured through the dashboard will be propagated across all databases of a project. This ensures that no Read Replica get out of sync with the Primary database or with other Read Replicas. -## Operations blocked by Read Replicas - -### Project upgrades and data restorations - -The following procedures require all Read Replicas for a project to be brought down before they can be performed: - -1. [Project upgrades](/docs/guides/platform/migrating-and-upgrading-projects#pgupgrade) -1. [Data restorations](/docs/guides/platform/backups#pitr-restoration-process) - -These operations need to be completed before Read Replicas can be re-deployed. - -## About replication - -We use a hybrid approach to replicate data from a Primary to its Read Replicas, combining the native methods of streaming replication and file-based log shipping. - -### Streaming replication - -Postgres generates a Write Ahead Log (WAL) as database changes occur. With streaming replication, these changes stream from the Primary to the Read Replica server. The WAL alone is sufficient to reconstruct the database to its current state. - -This replication method is fast, since changes are streamed directly from the Primary to the Read Replica. On the other hand, it faces challenges when the Read Replica can't keep up with the WAL changes from its Primary. This can happen when the Read Replica is too small, running on degraded hardware, or has a heavier workload running. - -To address this, Postgres does provide tunable configuration, like `wal_keep_size`, to adjust the WAL retained by the Primary. If the Read Replica fails to “catch up” before the WAL surpasses the `wal_keep_size` setting, the replication is terminated. Tuning is a bit of an art - the amount of WAL required is variable for every situation. - -### File-based log shipping - -In this replication method, the Primary continuously buffers WAL changes to a local file and then sends the file to the Read Replica. If multiple Read Replicas are present, files could also be sent to an intermediary location accessible by all. The Read Replica then reads the WAL files and applies those changes. There is higher replication lag than streaming replication since the Primary buffers the changes locally first. It also means there is a small chance that WAL changes do not reach Read Replicas if the Primary goes down before the file is transferred. In these cases, if the Primary fails a Replica using streaming replication would (in most cases) be more up-to-date than a Replica using file-based log shipping. - -### File-based log shipping 🤝 streaming replication - -Map view of Primary and Read Replica databases - -We bring these two methods together to achieve quick, stable, and reliable replication. Each method addresses the limitations of the other. Streaming replication minimizes replication lag, while file-based log shipping provides a fallback. For file-based log shipping, we use our existing Point In Time Recovery (PITR) infrastructure. We regularly archive files from the Primary using [WAL-G](https://github.com/wal-g/wal-g), an open source archival and restoration tool, and ship the WAL files to S3. - -We combine it with streaming replication to reduce replication lag. Once WAL-G files have been synced from S3, Read Replicas connect to the Primary and stream the WAL directly. - -### Monitoring replication lag - -Replication lag for a specific Read Replica can be monitored through the dashboard. On the [Database Reports page](/dashboard/project/_/observability/database) Read Replicas will have an additional chart under `Replica Information` displaying historical replication lag in seconds. Realtime replication lag in seconds can be observed on the [Infrastructure Settings page](/dashboard/project/_/settings/infrastructure). This is the value on top of the Read Replica. Do note that there is no single threshold to indicate when replication lag should be addressed. It would be fully dependent on the requirements of your project. - -If you are already ingesting your [project's metrics](/docs/guides/platform/metrics#accessing-the-metrics-endpoint) into your own environment, you can also keep track of replication lag and set alarms accordingly with the metric: `physical_replication_lag_physical_replica_lag_seconds`. - -Some common sources of high replication lag include: - -1. Exclusive locks on tables on the Primary. - Operations such as `drop table`, `reindex` (amongst others) take an Access Exclusive lock on the table. This can result in increasing replication lag for the duration of the lock. -1. Resource Constraints on the database - Heavy utilization on the primary or the replica, if run on an under-resourced project, can result in high replication lag. This includes the characteristics of the disk being utilized (IOPS, Throughput). -1. Long-running transactions on the Primary. - Transactions that run for a long-time on the primary can also result in high replication lag. You can use the `pg_stat_activity` view to identify and terminate such transactions if needed. `pg_stat_activity` is a live view, and does not offer historical data on transactions that might have been active for a long time in the past. - -High replication lag can result in stale data being returned for queries being executed against the affected read replicas. - -You can [consult](https://cloud.google.com/sql/docs/postgres/replication/replication-lag) [additional](https://repost.aws/knowledge-center/rds-postgresql-replication-lag) [resources](https://severalnines.com/blog/what-look-if-your-postgresql-replication-lagging/) on the subject as well. - -## Misc - -### Restart or compute add-on change behaviour - -When a project that utilizes Read Replicas is restarted, or the compute add-on size is changed, the Primary database gets restarted first. During this period, the Read Replicas remain available. - -Once the Primary database has completed restarting (or resizing, in case of a compute add-on change) and become available for usage, all the Read Replicas are restarted (and resized, if needed) concurrently. - ## Pricing -For a detailed breakdown of how charges are calculated, refer to [Manage Read Replica usage](/docs/guides/platform/manage-your-usage/read-replicas). +For a detailed breakdown of how we calculate charges, read the [Manage Read Replica usage guide](/docs/guides/platform/manage-your-usage/read-replicas). diff --git a/apps/docs/content/guides/platform/read-replicas/getting-started.mdx b/apps/docs/content/guides/platform/read-replicas/getting-started.mdx new file mode 100644 index 00000000000..42dac545148 --- /dev/null +++ b/apps/docs/content/guides/platform/read-replicas/getting-started.mdx @@ -0,0 +1,170 @@ +--- +title: 'Getting started with Read Replicas' +description: 'Deploy read-only databases across multiple regions, for lower latency.' +subtitle: 'Deploy read-only databases across multiple regions, for lower latency and better resource management.' +--- + +## Prerequisites + + + +Read Replicas are available for all projects on the Pro, Team and Enterprise plans. Spin one up now over at the [Infrastructure Settings page](/dashboard/project/_/settings/infrastructure). + + + +Projects must meet these requirements to use Read Replicas: + +1. Running on AWS. +2. Running on at least a [Small compute add-on](/docs/guides/platform/compute-add-ons). + + - Read Replicas are started on the same compute instance as the Primary to keep up with changes. + +3. Running on Postgres 15+. + + - For projects running on older versions of Postgres, you need to [upgrade to the latest platform version](/docs/guides/platform/migrating-and-upgrading-projects#pgupgrade). + +4. Not using [legacy logical backups](/docs/guides/platform/backups#point-in-time-recovery) + + - Physical backups are automatically enabled if using [Point in time recovery (PITR)](/docs/guides/platform/backups#point-in-time-recovery) + +## Creating a Read Replica + +To add a Read Replica, go to the [Database Replication page](/dashboard/project/_/database/replication) in your project dashboard. + +You can also manage Read Replicas using the Management API (beta functionality): + +```bash +# Get your access token from https://supabase.com/dashboard/account/tokens +export SUPABASE_ACCESS_TOKEN="your-access-token" +export PROJECT_REF="your-project-ref" + +# Create a new Read Replica +curl -X POST "https://api.supabase.com/v1/projects/$PROJECT_REF/read-replicas/setup" \ + -H "Authorization: Bearer $SUPABASE_ACCESS_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "region": "us-east-1" + }' + +# Delete a Read Replica +curl -X POST "https://api.supabase.com/v1/projects/$PROJECT_REF/read-replicas/remove" \ + -H "Authorization: Bearer $SUPABASE_ACCESS_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "database_identifier": "abcdefghijklmnopqrst" + }' +``` + + + +Projects on an XL compute add-on or larger can create up to five Read Replicas. Projects on compute add-ons smaller than XL can create up to two Read Replicas. All Read Replicas inherit the compute size of their Primary database. + + + +### Deploying a Read Replica + +We deploy a Read Replica using a physical backup as a starting point, and a combination of write ahead logging (WAL) file archives and direct replication from the Primary database to catch up. Both components may take significant time to complete, depending on your specific workload. + +The time to restore from a physical backup is dependent and directly related to the database size of your project. The time taken to catch up to the primary using WAL archives and direct replication is dependent on the level of activity on the Primary database. A more active database produces a larger number of WAL files that need to be processed. + +Along with the progress of the deployment, the dashboard displays rough estimates for each component. + +## Replication method details + +We use a hybrid approach to replicate data from a Primary to its Read Replicas, combining the native methods of streaming replication and file-based log shipping. + +### Streaming replication + +Postgres generates a Write Ahead Log (WAL) as database changes occur. With streaming replication, these changes stream from the Primary to the Read Replica server. The WAL alone is sufficient to reconstruct the database to its current state. + +This replication method is fast, since the Primary streams changes directly to the Read Replica. However, it faces challenges when the Read Replica can't keep up with the WAL changes from its Primary. This can happen when the Read Replica is too small, running on degraded hardware, or has a heavier workload running. + +To address this, Postgres provides tunable configuration, like `wal_keep_size`, to adjust the WAL retained by the Primary. If the Read Replica fails to "catch up" before the WAL surpasses the `wal_keep_size` setting, it terminates the replication. Tuning is an art - the amount of WAL required varies for every situation. + +### File-based log shipping + +In this replication method, the Primary continuously buffers WAL changes to a local file and then sends the file to the Read Replica. If multiple Read Replicas are present, files could also be sent to an intermediary location accessible by all replicas. + +The Read Replica then reads the WAL files and applies those changes. There is higher replication lag than streaming replication since the Primary buffers the changes locally first. It also means there is a small chance that WAL changes do not reach Read Replicas if the Primary goes down before the file is transferred. In these cases, if the Primary fails a Replica using streaming replication would (in most cases) be more up-to-date than a Replica using file-based log shipping. + +### File-based log shipping meets streaming replication + +Map view of Primary and Read Replica databases + +We bring these two methods together to achieve quick, stable, and reliable replication. Each method addresses the limitations of the other. Streaming replication minimizes replication lag, while file-based log shipping provides a fallback. For file-based log shipping, we use our existing Point In Time Recovery (PITR) infrastructure. We regularly archive files from the Primary using [WAL-G](https://github.com/wal-g/wal-g), an open source archival and restoration tool, and ship the WAL files to off-site, durable cloud storage, such as S3. + +We combine it with streaming replication to reduce replication lag. Once WAL-G files have been synced from S3, Read Replicas connect to the Primary and stream the WAL directly. + +### Restart or compute add-on change behaviour + +When you restart a project that utilizes Read Replicas, or change the compute add-on size, the Primary database gets restarted first. During this period, the Read Replicas remain available. + +Once the Primary database has completed restarting (or resizing, in case of a compute add-on change) and become available for usage, all the Read Replicas are restarted (and resized, if needed) concurrently. + +## Operations blocked by Read Replicas + +### Project upgrades and data restorations + +The following procedures require all Read Replicas for a project to be brought down before performing them: + +1. [Project upgrades](/docs/guides/platform/migrating-and-upgrading-projects#pgupgrade) +2. [Data restorations](/docs/guides/platform/backups#pitr-restoration-process) + +These operations need to complete before you can re-deploy Read Replicas. + +### Monitoring replication lag + +You can monitor replication lag for a specific Read Replica through a project dashboard on the [**Database Reports page**](/dashboard/project/_/observability/database). Read Replicas have an additional chart under **Replica Information** displaying historical replication lag in seconds. + +You can see realtime replication lag in seconds on the [**Infrastructure Settings** page](/dashboard/project/_/settings/infrastructure). This is the value on top of the Read Replica. + + + +There is no single threshold to indicate when you should address replication lag. It is dependent on the requirements of your project. + + + + + +If you are already ingesting your [project's metrics](/docs/guides/platform/metrics#accessing-the-metrics-endpoint) into your own environment, you can also keep track of replication lag and set alarms with the `physical_replication_lag_physical_replica_lag_seconds` metric. + + + +### Addressing high replication lag + +Some common sources of high replication lag include: + +1. **Exclusive locks on tables on the Primary**: Operations such as `drop table` and `reindex` take an access-exclusive lock on the table. This can result in increasing replication lag for the duration of the lock. +2. **Resource Constraints on the database**: Heavy utilization on the primary or the replica, if run on an under-resourced project, can result in high replication lag. This includes the characteristics of the disk being utilized (IOPS, Throughput). +3. **Long-running transactions on the Primary**: Transactions that run for a long-time on the primary can also result in high replication lag. You can use the `pg_stat_activity` view to identify and terminate such transactions if needed. `pg_stat_activity` is a live view, and does not offer historical data on transactions that might have been active for a long time in the past. + High replication lag can result in stale data returned for queries executed against the affected read replicas. + + + +You can find additional resources on replication lag in [the Google documentation](https://cloud.google.com/sql/docs/postgres/replication/replication-lag), [the AWS documentation](https://repost.aws/knowledge-center/rds-postgresql-replication-lag), and [the several nines blog](https://severalnines.com/blog/what-look-if-your-postgresql-replication-lagging/). + + + +## Troubleshooting + +{/* supa-mdx-lint-disable-next-line Rule001HeadingCase */} + +### An "Init failed" status + +The replica status "Init failed" in the dashboard indicates that the Read Replica has failed to deploy. Some possible scenarios as to why a Read Replica deployment may have failed are the following: + +- An underlying instance failed to come up. +- A network issue leading to inability to connect to the Primary database. +- A possible incompatible database settings between the Primary and Read Replica databases. +- Platform issues. +- Very high active workloads combined with large (50+ GB) database sizes + +It is safe to drop this failed Read Replica, and in the event of a transient issue, attempt to spin up another one. If spinning up Read Replicas for your project consistently fails, check the[status page](https://status.supabase.com) for any ongoing incidents, or [open a support ticket](/dashboard/support/new). To aid the investigation, do not bring down the recently failed Read Replica. + +{/* supa-mdx-lint-enable-next-line Rule001HeadingCase */} diff --git a/apps/docs/public/img/guides/platform/read-replicas/read-replicas-flow.svg b/apps/docs/public/img/guides/platform/read-replicas/read-replicas-flow.svg new file mode 100644 index 00000000000..a49c06eb103 --- /dev/null +++ b/apps/docs/public/img/guides/platform/read-replicas/read-replicas-flow.svg @@ -0,0 +1 @@ +

No

Yes

No

Yes

No

Yes

Yes

No

Yes

No

Database slowing down

CPU above 70% sustained?

Monitor, do not scale yet

Queries optimized? Indexes in place?

Run EXPLAIN ANALYZE
Add missing indexes
Optimize first

Workload 80%+ reads?

Upgrade compute
Replicas will not help writes

Already at 16XL?

Read Replicas
Only horizontal option left

Need workload isolation
or geo-distribution?

Read Replicas

Either works
Compute is simpler
Replicas scale further

 \ No newline at end of file