From b79f05df902c6d53caebb08b40bdee334b08da35 Mon Sep 17 00:00:00 2001 From: Charis <26616127+charislam@users.noreply.github.com> Date: Fri, 17 Jan 2025 17:17:10 -0500 Subject: [PATCH] migrate(docs): troubleshooting 23 (#30946) * migrate(docs): troubleshooting 23 postgres performance * Update how-postgres-chooses-which-index-to-use-_JHrf4.mdx * Update memory-and-swap-usage-explained-aPNgm0.mdx --------- Co-authored-by: TheOtherBrian1 <91111415+TheOtherBrian1@users.noreply.github.com> --- ...gres-chooses-which-index-to-use-_JHrf4.mdx | 161 ++++++++++++++++++ ...-being-blocked-by-other-queries-NSKtR1.mdx | 41 +++++ ...ow-to-delete-a-role-in-postgres-8-AvxY.mdx | 40 +++++ ...peeds-by-applying-an-hsnw-index-ohLHUM.mdx | 112 ++++++++++++ ...memory-and-swap-usage-explained-aPNgm0.mdx | 55 ++++++ ...table-when-changing-column-type-qmZRpZ.mdx | 65 +++++++ 6 files changed, 474 insertions(+) create mode 100644 apps/docs/content/troubleshooting/how-postgres-chooses-which-index-to-use-_JHrf4.mdx create mode 100644 apps/docs/content/troubleshooting/how-to-check-if-my-queries-are-being-blocked-by-other-queries-NSKtR1.mdx create mode 100644 apps/docs/content/troubleshooting/how-to-delete-a-role-in-postgres-8-AvxY.mdx create mode 100644 apps/docs/content/troubleshooting/increase-vector-lookup-speeds-by-applying-an-hsnw-index-ohLHUM.mdx create mode 100644 apps/docs/content/troubleshooting/memory-and-swap-usage-explained-aPNgm0.mdx create mode 100644 apps/docs/content/troubleshooting/slow-execution-of-alter-table-on-large-table-when-changing-column-type-qmZRpZ.mdx diff --git a/apps/docs/content/troubleshooting/how-postgres-chooses-which-index-to-use-_JHrf4.mdx b/apps/docs/content/troubleshooting/how-postgres-chooses-which-index-to-use-_JHrf4.mdx new file mode 100644 index 00000000000..e51b262a5ae --- /dev/null +++ b/apps/docs/content/troubleshooting/how-postgres-chooses-which-index-to-use-_JHrf4.mdx @@ -0,0 +1,161 @@ +--- +title = "How Postgres chooses which index to use" +github_url = "https://github.com/orgs/supabase/discussions/26959" +date_created = "2024-06-03T05:53:16+00:00" +topics = ["database"] +keywords = ["index", "performance", "optimization"] +--- + +> For the curious: [here is a list of all built-in indexes in Postgres](https://www.postgresql.org/docs/current/indexes-types.html) + +### Postgres Internals + +#### How an index is chosen + +PostgreSQL, internally, contains a few components that manage query execution: + +| Module | Description | +| ----------------- | ------------------------------------------------------------------------------------------------------------- | +| Parser | Converts SQL into an easily traversable query tree | +| Planner/Optimizer | Takes the query tree and uses rules and database statistics to find the optimal strategy for getting the data | +| Executor | Executes the plan created by the planner | + +The planner will consider using an index when an indexed column is present in a filter statement, such as: + +- `WHERE` +- `LIKE` +- `ILIKE` +- `DISTINCT` +- `SIMILAR TO` +- `JOIN` +- `ORDER BY` + +Otherwise, it will likely perform a full table scan (sequential scan). + +In the majority of cases, the indexed column must not only be present but also must be filtered by a comparison operator (`=`, `>`, `<>`) that is compatible with the index. + +As an example, one can create the following table: + +| Column Name | Data Type | +| ----------- | --------- | +| id | INT | +| data | JSONB | + +On the data column, a GIN index can be applied, which is excellent for filtering JSONB datatypes: + +```SQL +CREATE INDEX some_arbitary_index_name ON some_table USING gin (data); +``` + +Here's a [link](https://www.postgresql.org/docs/current/gist-builtin-opclasses.html) to the list operators supported by the GIN index; notably, it does not support greater than `>`: + +```sql +-- GIN index will never be used +select * +from some_table +where data -> val > 5; +``` + +GIN does support the `@>` operator: + +```sql +--GIN will be considered +SELECT id FROM some_table +WHERE data @> '[ { "itemId": "p11" } ]'; +``` + +In most cases, developers work with the default BTREE index. It is the most practical and performant in the majority of cases and is compatible with the following filter [operators](https://www.postgresql.org/docs/current/btree-behavior.html): + +| Comparison Operator | +| ------------------- | +| `<` | +| `<=` | +| `=` | +| `>=` | +| `>` | + +An operator's functional equivalents, such as `IN`, `BETWEEN`, and `ANY`, are also valid. + +However, just because the base requirements (relevant column, filter, and operators) are present, doesn't mean that an index will be used. + +Indexes have a startup cost, so for small tables, Postgres might use a sequential scan if it believes that it will take less time. The database keeps statistics about each table that it uses to inform these choices. + +In very rare cases, these statistics can become stale, and Postgres may opt to use a slower index or sequential scan when a better option is available. + +You can see the query plan with the `EXPLAIN` keyword: + +``` +EXPLAIN +``` + +To understand how to interpret its output, you can check out this [explainer](https://github.com/orgs/supabase/discussions/22839). + +To reset statistics within the database, you can use the following query: + +```sql +-- use judiciously +select pg_stat_reset(); +``` + +### Complex or Composite indexes + +> For a more complete rundown, check the [Postgres Official Docs](https://www.postgresql.org/docs/current/indexes-multicolumn.html) + +#### Multi-column indexes + +If you make independent indexes on multiple columns, Postgres will likely use each of them independently to find the relevant rows and then combine the results together. + +It is possible to make [multi-column indexes](https://www.postgresql.org/docs/current/indexes-multicolumn.html). If you are regularly filtering against multiple columns, there can be performance benefits using them instead of several independent indexes. + +```sql +-- multi-column index +create index test2_mm_idx on test2 (major, minor); + +-- multi-column comparison: +select name +from test2 +where major = constant and minor = constant; +``` + +#### Ordered indexes + +If you're using an ORDER BY clause, [indexes can also be pre-sorted by DESC/ASC](https://www.postgresql.org/docs/current/indexes-ordering.html) for better performance. + +``` +-- organizes the index in a DESC order, places NULL values at the end +CREATE INDEX test3_desc_index ON test3 (id DESC NULLS LAST); +``` + +#### Functional indexes + +Although not as common, indexes can also be leveraged against modified values, such as when using a LOWER function: + +```sql +-- Index on modified column through function +create index test1_lower_col1_idx on test1 (lower(col1)); + +-- Index will be considered for the following query: +select * from test1 where lower(col1) = 'value'; +``` + +#### Covering indexes + +Indexes contain pointers to a specific row, but you could instruct an index to actually hold a copy of a column's value for even faster retrieval. These are known as `covering` indexes. Because maintaining a copy is storage intensive, you should avoid using it for values with large data footprints.[ FULL VIDEO ON TOPIC](https://www.youtube.com/watch?v=bBu_V8CfWgM) + +```sql +CREATE INDEX a_b_idx ON x (a,b) INCLUDE (c); +``` + +#### Indexes on JSONB + +Although a GIN/GIST index can be used to index entire JSONB bodies, you can also target just specific Key-values with standard BTREE indexes: + +```sql +-- Example table +create table person ( + id serial primary key, + data jsonb +); + +create index index_name on person ((data ->> 'name')); +``` diff --git a/apps/docs/content/troubleshooting/how-to-check-if-my-queries-are-being-blocked-by-other-queries-NSKtR1.mdx b/apps/docs/content/troubleshooting/how-to-check-if-my-queries-are-being-blocked-by-other-queries-NSKtR1.mdx new file mode 100644 index 00000000000..0fe37a4f132 --- /dev/null +++ b/apps/docs/content/troubleshooting/how-to-check-if-my-queries-are-being-blocked-by-other-queries-NSKtR1.mdx @@ -0,0 +1,41 @@ +--- +title = "How to check if my queries are being blocked by other queries?" +github_url = "https://github.com/orgs/supabase/discussions/19681" +date_created = "2023-12-13T15:56:12+00:00" +topics = ["database"] +--- + +## You can set a lock monitor view to help investigate these. + +Once you run the query that takes a long time to complete, you can go in the dashboard (or select from this view below) to check what are the blocks. + +```sql +create view + public.lock_monitor as +select + coalesce( + blockingl.relation::regclass::text, + blockingl.locktype + ) as locked_item, + now() - blockeda.query_start as waiting_duration, + blockeda.pid as blocked_pid, + blockeda.query as blocked_query, + blockedl.mode as blocked_mode, + blockinga.pid as blocking_pid, + blockinga.query as blocking_query, + blockingl.mode as blocking_mode +from + pg_locks blockedl + join pg_stat_activity blockeda on blockedl.pid = blockeda.pid + join pg_locks blockingl on ( + blockingl.transactionid = blockedl.transactionid + or blockingl.relation = blockedl.relation + and blockingl.locktype = blockedl.locktype + ) + and blockedl.pid <> blockingl.pid + join pg_stat_activity blockinga on blockingl.pid = blockinga.pid + and blockinga.datid = blockeda.datid +where + not blockedl.granted + and blockinga.datname = current_database(); +``` diff --git a/apps/docs/content/troubleshooting/how-to-delete-a-role-in-postgres-8-AvxY.mdx b/apps/docs/content/troubleshooting/how-to-delete-a-role-in-postgres-8-AvxY.mdx new file mode 100644 index 00000000000..19f276a187d --- /dev/null +++ b/apps/docs/content/troubleshooting/how-to-delete-a-role-in-postgres-8-AvxY.mdx @@ -0,0 +1,40 @@ +--- +title = "How to delete a role in Postgres" +github_url = "https://github.com/orgs/supabase/discussions/27427" +date_created = "2024-06-20T19:21:11+00:00" +topics = ["database"] +keywords = ["role", "delete", "postgres"] + +[[errors]] +message = "a role cannot be removed while it is still referenced in any database of the cluster" +--- + +[Quote from postgres docs:](https://www.postgresql.org/docs/current/sql-droprole.html#:~:text=A%20role%20cannot%20be%20removed,been%20granted%20on%20other%20objects.) + +> A role cannot be removed if it is still referenced in any database of the cluster; an error will be raised if so. Before dropping the role, you must drop all the objects it owns (or reassign their ownership) and revoke any privileges the role has been granted on other objects. + +First make sure that Postgres has ownership over the role: + +```sql +GRANT TO "postgres"; +``` + +Then you must reassign any objects owned by role: + +```sql +REASSIGN OWNED BY TO postgres; +``` + +Once ownership is transferred, you can run the following query: + +```sql +DROP OWNED BY ; +``` + +[DROP OWNED BY](https://www.postgresql.org/docs/current/sql-drop-owned.html) does delete all objects owned by the role, which should be none. However, it also revokes the role's privileges. Once this is done, you should be able to run: + +```sql +DROP role ; +``` + +If you encounter any issues, please create a [support ticket](https://supabase.com/dashboard/support/new) diff --git a/apps/docs/content/troubleshooting/increase-vector-lookup-speeds-by-applying-an-hsnw-index-ohLHUM.mdx b/apps/docs/content/troubleshooting/increase-vector-lookup-speeds-by-applying-an-hsnw-index-ohLHUM.mdx new file mode 100644 index 00000000000..c0987999fa2 --- /dev/null +++ b/apps/docs/content/troubleshooting/increase-vector-lookup-speeds-by-applying-an-hsnw-index-ohLHUM.mdx @@ -0,0 +1,112 @@ +--- +title = "Increase vector lookup speeds by applying an HSNW index" +github_url = "https://github.com/orgs/supabase/discussions/21379" +date_created = "2024-02-20T05:32:05+00:00" +topics = ["database", "ai", "platform"] +keywords = ["index", "hnsw", "vector", "performance", "speed"] +--- + +> Although this guide is specifically for HSNW indexes, it can be generalized to work for any index type + +> Building an index without the `CONCURRENTLY` modifier will lock the table, but it will also increase build times. For general advice about indexes, check out this [guide](https://github.com/orgs/supabase/discussions/22449). + +### **To speed up queries, it is ideal to create an HSNW index on your embedded column** + +The general structure for creating an hsnw index follows this pattern: + +```sql +CREATE INDEX ON USING hnsw ( ); +``` + +Search can be one of three types: +operator | description | search type +-- | -- | -- +`<->` | Euclidean distance | vector_l2_ops +`<#>` | negative inner product | vector_ip_ops +`<=>` | cosine distance | vector_cosine_ops + +Queries can only utilize the index if it matches the search type used. If you are unsure which search type to prioritize, vector_cosine_ops is the most commonly used. You can checkout our [guide](https://supabase.com/docs/guides/ai/vector-indexes/hnsw-indexes) for more info. The folks at Crunchy Data also wrote an [explainer](https://www.crunchydata.com/blog/hnsw-indexes-with-postgres-and-pgvector) that you may find useful. + +Applying an index can be slow and computationally expensive, so there are a few preparations that should be made beforehand: + +**1. Make sure your pgvector is the latest available version on Supabase.** + +Versions 0.6 and later have accelerated HNSW build speeds. You can observe your current version in the [Dashboard's Extensions Page](https://supabase.com/dashboard/project/_/database/extensions). You can perform a software upgrade in the [Infrastructure Settings ](https://supabase.com/dashboard/project/_/settings/infrastructure)if necessary. + +**2. Setting up an external connection** + +The Dashboard has an internal time limit of ~2 minutes for queries. Indexing a large table will almost always take more time, so it is necessary to execute your code through an external interface, such as PSQL. + +You can install PSQL in [macOS](https://stackoverflow.com/a/49689589/2188186) and [Windows](https://www.postgresql.org/download/windows/) by following these links and instructions. +For Linux (Debian) you can run the following: + +```bash +sudo apt-get update +sudo apt-get install postgresql-client +``` + +Once installed, you can find your PSQL string from the [Database Settings](https://supabase.com/dashboard/project/_/settings/database), which can be executed in the terminal to create a psql session. + +If your network can use IPv6, consider using the direct connection string instead of Supavisor. It's not mandatory, but for tasks that run a long time, it's best to reduce network complexity. To check if your network is compatible, use this cURL command to request your IPv6 address: + +```bash +curl -6 https://ifconfig.co/ip +``` + +If an address is returned, you should be able to use your direct connection string found in the [Database Settings](https://supabase.com/dashboard/project/_/settings/database): + +**3. Increase memory for index creation (optional)** + +The `maintance_work_mem` variable limits the maximum amount of memory that can be used by maintenance operations, such as vacuuming, altering, and indexing tables. In your session you should try to set it to a reasonably high value: + +```sql +set maintenance_work_mem to ; -- '#GB' +``` + +Inspect value to make sure it has been set: + +```sql +show maintenance_work_mem; +``` + +**4. Increase cores for index creation (optional)** + +The `max_parallel_maintenance_workers` variable limits the amount of cores that can be used by maintenance operations, including indexing tables. In your session, you should try to set it to a value roughly 1/2 to 2/3s of your [compute core amount](https://supabase.com/docs/guides/platform/compute-add-ons): + +```sql +set max_parallel_maintenance_workers to ; +``` + +Inspect value to make sure it has been set: + +```sql +show max_parallel_maintenance_workers; +``` + +**5. Setting a custom timeout** + +Disable query timeout for your connection: + +```sql +set statement_timeout = '0'; +``` + +Inspect value to make sure it has been set: + +```sql +show statement_timeout; +``` + +**6. Consider temporarily upgrading your compute size (optional)** + +If your task is particularly long, you can speed it up by boosting your computing power temporarily. Compute size is charged by the hour, so you can increase it for an hour or two to finish your task faster, then scale it back afterward. Here is a list of [compute add-ons](https://supabase.com/docs/guides/platform/compute-add-ons). If you want to temporarily upgrade, you can find the add-ons for your project in your [Dashboard's Add-Ons Settings.](https://supabase.green/dashboard/project/_/settings/addons) + +**7. Consider increasing disk size (optional)** + +HSNW indexes can produce temporary files during their construction that may consume a few GBs worth of disk. Consider increasing the disk size in the [Database Settings](https://supabase.com/dashboard/project/_/settings/database) to accommodate for short-term disk stress. + +Screenshot 2024-06-10 at 8 00 28 PM diff --git a/apps/docs/content/troubleshooting/memory-and-swap-usage-explained-aPNgm0.mdx b/apps/docs/content/troubleshooting/memory-and-swap-usage-explained-aPNgm0.mdx new file mode 100644 index 00000000000..e1083888d58 --- /dev/null +++ b/apps/docs/content/troubleshooting/memory-and-swap-usage-explained-aPNgm0.mdx @@ -0,0 +1,55 @@ +--- +title = "Memory and Swap usage explained" +github_url = "https://github.com/orgs/supabase/discussions/21460" +date_created = "2024-02-22T19:01:27+00:00" +topics = ["database", "platform"] +keywords = ["memory", "swap", "ram", "disk", "IO"] +--- + +## Understanding Swap and Memory Usage + +- Swap is part of Linux's tiered memory system +- Serves as a backup memory space when RAM is limited. +- System prioritizes keeping actively used data in RAM +- Data is temporarily cached in memory for quick access +- Swap is used for less frequently accessed or non-critical data +- Default swapiness value of 60 indicates the system's preference to fill up swap for performance benefits + +## Compute Instance Configuration + +- Default swapiness: 60 (out of 100) + - Determines how aggressively the system moves data from RAM to swap + - Default value in Linux distributions + - Changing the swapiness value significantly impacts system behavior, as it is a non-linear value. A small adjustment of 10 can result in a very different system behavior +- Swap provision: 1024MB on every compute instance + +## Issues with High Swap Usage + +- When swap reaches its limits, the system faces decisions: + 1. Evict cached memory from swap, which affects disk performance + 2. Evict cached memory from the system's memory cache, leading to increased disk accesses + 3. Access the disk directly without caching, resulting in more frequent disk reads +- Swap usage reaching limits indicates running out of free memory + +## Disk Balance and AWS Limits + +- Each project has a baseline 'balance replenishment rate' +- The replenishment rate is higher on greater AWS instances +- Disk IO usage below replenishment rate keeps balance at 99% +- Disk IO usage above replenishment rate decreases balance +- Balance increases when IO usage is below replenishment rate +- AWS does not export relevant metrics for tracking IO usage + +## Monitoring Recommendations + +- Monitor memory usage: + - If usage is over 85% and swap usage is over 90% for an extended period, optimize database access or add more resources +- Monitor CPU iowait usage and disk IO metrics: + - Exported metrics here: **https://github.com/supabase/grafana-agent-fly-example/blob/main/metrics.md** + - Key metrics to track excessive disk IO usage include: + - `node_disk_reads_completed_total`: Tracks the total number of completed disk reads. + - `node_disk_io_time_seconds_total`: Measures the total time spent on disk IO operations. + - `node_disk_io_now`: Indicates the current disk IO operations. + - Differentiate system and data disks using the 'device' label + - System disk: `{device="nvme0n1"}` (contains swap) + - Data disk: `{device="nvme1n1"}` (dedicated to Postgres data) diff --git a/apps/docs/content/troubleshooting/slow-execution-of-alter-table-on-large-table-when-changing-column-type-qmZRpZ.mdx b/apps/docs/content/troubleshooting/slow-execution-of-alter-table-on-large-table-when-changing-column-type-qmZRpZ.mdx new file mode 100644 index 00000000000..04882c4019f --- /dev/null +++ b/apps/docs/content/troubleshooting/slow-execution-of-alter-table-on-large-table-when-changing-column-type-qmZRpZ.mdx @@ -0,0 +1,65 @@ +--- +title = "Slow Execution of ALTER TABLE on Large Table when changing column type" +github_url = "https://github.com/orgs/supabase/discussions/19747" +date_created = "2023-12-14T17:54:46+00:00" +topics = ["database"] +keywords = ["alter", "table", "column", "slow", "performance"] +--- + +If you encounter slow execution of the ALTER TABLE operation on a large table when changing a column data type, consider the following alternative approach. + +**Alternative Approach:** + +1. Add a New Column with the New Type: + `ALTER TABLE "table_name" ADD COLUMN "new_column_name" new_data_type;` + +2. Copy Values from the First Column to the Second: + `UPDATE "table_name" SET "old_column_name" = "new_column_name"::new_data_type;` + +3. Drop the Old Column: + `ALTER TABLE "table_name" DROP COLUMN "old_column_name";` + +**Why Use This Approach?** +The long execution time for the ALTER TABLE operation can be attributed to the large size of the table. This segmented approach helps in: + +- Efficiency: The process is more efficient as it avoids prolonged transactions. +- Minimizing Disruption: Migrating data systematically minimizes the impact on other operations and users. + +**Additional Recommendations:** + +- Set session statement_timeout to 0 to prevent potential transaction [timeouts](https://supabase.com/docs/guides/database/postgres/configuration). + +- Monitor currently blocked database transactions during the process using the provided script: + +``` +create view public.lock_monitor as +select + coalesce( + blockingl.relation::regclass::text, + blockingl.locktype + ) as locked_item, + now() - blockeda.query_start as waiting_duration, + [blockeda.pid](http://blockeda.pid/) as blocked_pid, + blockeda.query as blocked_query, + blockedl.mode as blocked_mode, + [blockinga.pid](http://blockinga.pid/) as blocking_pid, + blockinga.query as blocking_query, + blockingl.mode as blocking_mode +from + pg_locks blockedl + join pg_stat_activity blockeda on [blockedl.pid](http://blockedl.pid/) = [blockeda.pid](http://blockeda.pid/) + join pg_locks blockingl on ( + blockingl.transactionid = blockedl.transactionid + or blockingl.relation = blockedl.relation + and blockingl.locktype = blockedl.locktype + ) + and [blockedl.pid](http://blockedl.pid/) <> [blockingl.pid](http://blockingl.pid/) + join pg_stat_activity blockinga on [blockingl.pid](http://blockingl.pid/) = [blockinga.pid](http://blockinga.pid/) + and blockinga.datid = blockeda.datid +where + not blockedl.granted + and blockinga.datname = current_database(); +``` + +**Notes:** +Execute these steps during a maintenance window or low-traffic period to minimize disruption.