Merge pull request #16238 from supabase/or/index-advisor

Docs: Add indexing guide and index advisor extension

apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts

@@ -568,6 +568,7 @@ export const database: NavMenuConstant = {
         { name: 'Implementing Cascade Deletes', url: '/guides/database/postgres/cascade-deletes' },
         { name: 'Implementing column encryption', url: '/guides/database/column-encryption' },
         { name: 'Partitioning your tables', url: '/guides/database/partitions' },
+        { name: 'Query Optimization', url: '/guides/database/query-optimization' },
         { name: 'Testing your database', url: '/guides/database/testing' },
         { name: 'Managing Timeouts', url: '/guides/database/timeouts' },
         { name: 'Managing Passwords', url: '/guides/database/managing-passwords' },
@@ -588,6 +589,10 @@ export const database: NavMenuConstant = {
           url: '/guides/database/extensions/plv8',
         },
         { name: 'http: RESTful Client', url: '/guides/database/extensions/http' },
+        {
+          name: 'index_advisor: Query optimization',
+          url: '/guides/database/extensions/index_advisor',
+        },
         {
           name: 'PGAudit: Postgres Auditing',
           url: '/guides/database/extensions/pgaudit',

apps/docs/data/extensions.json

@@ -113,6 +113,12 @@
     "tags": ["Admin", "Utility"],
     "link": "https://www.postgresql.org/docs/current/oldsnapshot.html"
   },
+  {
+    "name": "index_advisor",
+    "comment": "optimize query performance with automatic index recommendation",
+    "tags": ["Utility"],
+    "link": "/guides/database/extensions/index_advisor"
+  },
   {
     "name": "intarray",
     "comment": "functions, operators, and index support for 1-D arrays of integers",

apps/docs/pages/guides/database/extensions/index_advisor.mdx

@@ -0,0 +1,161 @@
+import Layout from '~/layouts/DefaultGuideLayout'
+
+export const meta = {
+  id: 'index-advisor',
+  title: 'index_advisor: query optimization',
+  description: 'Automatically optimize SQL queries',
+}
+
+[Index advisor](https://github.com/supabase/index_advisor) is a Postgres extension for recommending indexes to improve query performance.
+
+For example:
+
+```sql
+select
+    *
+from
+  index_advisor('select book.id from book where title = $1');
+
+ startup_cost_before | startup_cost_after | total_cost_before | total_cost_after |                  index_statements                   | errors
+---------------------+--------------------+-------------------+------------------+-----------------------------------------------------+--------
+ 0.00                | 1.17               | 25.88             | 6.40             | {"CREATE INDEX ON public.book USING btree (title)"},| {}
+(1 row)
+```
+
+Features:
+
+- Supports generic parameters e.g. `$1`, `$2`
+- Supports materialized views
+- Identifies tables/columns obfuscaed by views
+- Skips duplicate indexes
+
+## Installation
+
+index_advisor is a trusted language extension, which means it is directly installable by users from the [database.dev](database.dev) SQL package repository.
+
+To get started, enable the dbdev client by executing the [setup SQL script](https://database.dev/installer).
+
+Then, install index_advisor by running
+
+```sql
+select dbdev.install('olirice-index_advisor');
+create extension if not exists hypopg;
+create extension "olirice-index_advisor";
+```
+
+## API
+
+Index advisor exposes a single function `index_advisor(query text)` that accepts a query and searches for a set of SQL DDL `create index` statements that improve the query's execution time.
+
+The function's signature is:
+
+```sql
+index_advisor(query text)
+returns
+    table  (
+        startup_cost_before jsonb,
+        startup_cost_after jsonb,
+        total_cost_before jsonb,
+        total_cost_after jsonb,
+        index_statements text[],
+        errors text[]
+    )
+```
+
+## Usage
+
+As a minimal example, the `index_advisor` function can be given a single table query with a filter on an unindexed column.
+
+```sql
+create extension if not exists index_advisor cascade;
+
+create table book(
+  id int primary key,
+  title text not null
+);
+
+select
+  *
+from
+  index_advisor('select book.id from book where title = $1');
+
+ startup_cost_before | startup_cost_after | total_cost_before | total_cost_after |                  index_statements                   | errors
+---------------------+--------------------+-------------------+------------------+-----------------------------------------------------+--------
+ 0.00                | 1.17               | 25.88             | 6.40             | {"CREATE INDEX ON public.book USING btree (title)"},| {}
+(1 row)
+```
+
+and will return a row recommending an index on the unindexed column.
+
+More complex queries may generate additional suggested indexes:
+
+```sql
+create extension if not exists index_advisor cascade;
+
+create table author(
+    id serial primary key,
+    name text not null
+);
+
+create table publisher(
+    id serial primary key,
+    name text not null,
+    corporate_address text
+);
+
+create table book(
+    id serial primary key,
+    author_id int not null references author(id),
+    publisher_id int not null references publisher(id),
+    title text
+);
+
+create table review(
+    id serial primary key,
+    book_id int references book(id),
+    body text not null
+);
+
+select
+    *
+from
+    index_advisor('
+        select
+            book.id,
+            book.title,
+            publisher.name as publisher_name,
+            author.name as author_name,
+            review.body review_body
+        from
+            book
+            join publisher
+                on book.publisher_id = publisher.id
+            join author
+                on book.author_id = author.id
+            join review
+                on book.id = review.book_id
+        where
+            author.id = $1
+            and publisher.id = $2
+    ');
+
+ startup_cost_before | startup_cost_after | total_cost_before | total_cost_after |                  index_statements                         | errors
+---------------------+--------------------+-------------------+------------------+-----------------------------------------------------------+--------
+ 27.26               | 12.77              | 68.48             | 42.37            | {"CREATE INDEX ON public.book USING btree (author_id)",   | {}
+                                                                                    "CREATE INDEX ON public.book USING btree (publisher_id)",
+                                                                                    "CREATE INDEX ON public.review USING btree (book_id)"}
+(3 rows)
+```
+
+## Limitations
+
+- index_advisor will only recommend single column, B-tree indexes. More complex indexes will be supported in future releases.
+- when a generic argument's type is not discernible from context, an error is returned in the `errors` field. To resolve those errors, add explicit type casting to the argument. e.g. `$1::int`.
+
+## Resources
+
+- Official [`index_advisor` GitHub Repository](https://github.com/supabase/index_advisor)
+
+export const Page = ({ children }) => <Layout meta={meta} children={children} />
+
+export default Page

apps/docs/pages/guides/database/query-optimization.mdx

@@ -0,0 +1,134 @@
+import Layout from '~/layouts/DefaultGuideLayout'
+
+export const meta = {
+  title: 'Query Optimization',
+  description: 'Choosing indexes',
+  footerHelpType: 'postgres',
+}
+
+When working with PostgreSQL, or any relational database, indexing is key to improving query performance.
+Aligning indexes with common query patterns can speed up data retrieval by an order of magnitude.
+
+This guide is intended to:
+
+- help identify parts of a query that have the potential to be improved by indexes
+- introduce tooling to help identify useful indexes
+
+This is not a comprehensive resource, but rather a helpful starting point for your optimization journey.
+
+If you're new to query optimization, you may be interested in [index_advisor](/docs/guides/database/extensions/index_advisor), our tool for automatically detecting indexes that improve performance on a given query.
+
+## Example Query
+
+Consider the following example query that retrieves customer names and purchase dates from two tables:
+
+```sql
+select
+  a.name,
+  b.date_of_purchase
+from
+  customers a
+  join orders b
+    on a.id = b.customer_id
+where
+  a.sign_up_date > '2023-01-01'
+  and b.status = 'shipped'
+order by
+  b.date_of_purchase;
+limit 10
+```
+
+In this query, there are several parts that indexes could likely help in optimizing the performance:
+
+### `where` Clause:
+
+The `where` clause filters rows based on certain conditions, and indexing the columns involved can improve this process:
+
+- `a.sign_up_date`: If filtering by `sign_up_date` is common, indexing this column can speed up the query.
+- `b.status`: Indexing the status may be beneficial if the column has diverse values.
+
+```sql
+create index idx_customers_sign_up_date on customers (sign_up_date);
+
+create index idx_orders_status on orders (status);
+```
+
+### `join` Columns
+
+Indexes on the columns used for joining tables can help Postgres avoid scanning tables in their entirety when connecting tables.
+
+- Indexing and `b.customer_id` would likely improve the performance of the join in this query.
+- Note that if `a.id` is the primary key of the `customers` table it is already indexed
+
+```sql
+create index idx_orders_customer_id on orders (customer_id);
+```
+
+### `order by` Clause
+
+Sorting can also be optimized by indexing:
+
+- An index on `b.date_of_purchase` wcan improve the sorting process, and is particularly beneficial when a subset of rows is being returned with a `limit` clause.
+
+```sql
+create index idx_orders_date_of_purchase on orders (date_of_purchase);
+```
+
+## Key Concepts
+
+Here are some concepts and tools to keep in mind to help you identify the best index for the job, and measure the impact that your index had:
+
+### Analyze the Query Plan
+
+Use the `explain` command to understand the query's execution. Look for slow parts, such as Sequential Scans or high cost numbers. If creating an index does not reduce the cost of the query plan, remove it.
+
+For example:
+
+```sql
+explain select * from customers where sign_up_date > 25;
+```
+
+### Use Appropriate Index Types
+
+PostgreSQL offers various index types like [B-tree, Hash, GIN, etc](https://www.postgresql.org/docs/current/indexes-types.html). Select the type that best suits your data and query pattern. Using the right index type can make a significant difference. For example, using a BRIN index on a field that always increases and lives within a table that updates infrequently - like `created_at` on an `orders` table - routinely results in indexes that are +10x smaller than the equivalent default B-tree index. That translates into better scalability.
+
+```sql
+create index idx_orders_created_at ON customers using brin(created_at);
+```
+
+### Partial Indexes
+
+For queries that frequently target a subset of data, a partial index could be faster and smaller than indexing the entire column. A partial index contains a `where` clause to filter the values included in the index. Note that a query's `where` clause must match the index for it to be used.
+
+```sql
+create index idx_orders_status on orders (status)
+where status = 'shipped';
+```
+
+### Composite Indexes
+
+If filtering or joining on multiple columns, a composite index prevents Postgres from referring to multiple indexes when identifying the relevant rows.
+
+```sql
+create index idx_customers_sign_up_date_priority on customers (sign_up_date, priority);
+```
+
+### Over-Indexing
+
+Avoid the urge to index columns you operate on infrequently. While indexes can speed up reads, they also slow down writes, so it's important to balance those factors when making indexing decisions.
+
+### Statistics
+
+Postgres maintains a set of statistics about the contents of your tables. Those statistics are used by the query planner to decide when it's is more efficient to use an index vs scanning the entire table. If the collected statistics drift too far from reality, the query planner may make poor decisions. To avoid this risk, you can periodically `analyze` tables.
+
+```sql
+analyze customers;
+```
+
+---
+
+By following this guide, you'll be able to discern where indexes can optimize queries and enhance your PostgreSQL performance. Remember that each database is unique, so always consider the specific context and use case of your queries.
+
+export const Page = ({ children }) => <Layout meta={meta} children={children} />
+
+export default Page