Files
supabase/packages/pg-meta/test/table-privileges.test.ts
Andrew ValleteauandClaude Fable 5 6c6a721cb7 fix(pg-meta): scope remaining O(catalog) introspection queries behind pgMetaScopedIntrospection (#48148)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Bug fix (performance), follow-up to #47894, plus regression-guard tests.

## What is the current behavior?

#47894 scoped the Table Editor and entity-definition introspection
queries, but four more `@supabase/pg-meta` query families still do
O(catalog) work per request. On a production project with a very large
catalog (hundreds of schemas, ~465K `pg_constraint` rows) they run 5 to
55 seconds each, trip the 58s `statement_timeout`, and spill sorts to
temp files. During a recent "DB CPU > 85%" incident on such a project,
24 of 27 active backends were running these queries concurrently.

1. **`tables.retrieve()` (single-table lookup by name+schema or id)**:
the `tables`/`columns` CTEs scan the whole catalog (`pg_class`,
`pg_constraint`, `pg_index`, all of `pg_attribute`, per-table sizes) and
the one-table predicate is applied only on the outer select. Same bug
class #47894 fixed for the OID-based table editor query; this sibling
path never got the treatment. It accounted for 94 of the 96
statement-timeout cancellations in the incident.
2. **Types listing**: the `t_enums` and `t_attributes` subqueries
aggregate the entire `pg_enum` and every composite relation before the
wrapper's schema filter applies.
3. **Table privileges**: `aclexplode` + double `pg_roles` join + GROUP
BY over every relation in the database; schema/OID filters applied only
after aggregation, in both `list()` and `retrieve()`.
4. **Row counts**: `getTableRowsCountSql` treats `reltuples = -1`
(never-analyzed table) as "small table, run exact count(*)". A freshly
bulk-loaded multi-million-row table times out on every Table Editor
pagination render.

Two Studio-side amplifiers turned one slow query into a sustained load
storm:

- `useTableQuery` (behind `tables.retrieve()`) mounts once per visible
foreign-key grid cell via `ForeignKeyFormatter`, so a single Table
Editor view fires ~20 concurrent copies against the FK target table. A
timed-out query caches nothing, and TanStack retries errored no-data
queries on every observer mount by default, so scrolling kept re-issuing
the 58s scan.
- `useTableApiAccessQuery` fetched table privileges for the entire
database and filtered down to one schema client-side.

## What is the new behavior?

**pg-meta (all behind the existing `pgMetaScopedIntrospection` flag,
same rollout mechanism as #47894; `scoped: false` keeps serving the
current SQL):**

- `tables.retrieve()`: the identifier is resolved to a scalar
`targetOid` init-plan and pushed into the base scan, primary-key,
relationships (both FK directions kept: `conrelid` or `confrelid`) and
columns CTEs. A materialized `target` CTE was deliberately avoided: it
acts as an optimization barrier and forces the very seq scans being
removed.
- Types: filter `pg_type`/`pg_namespace` first, then compute
enums/attributes per surviving row via correlated index-scan subqueries
(`pg_enum(enumtypid, enumsortorder)`, `pg_attribute(attrelid, attnum)`).
- Table privileges: schema/OID predicates injected into the base WHERE
before `aclexplode`/GROUP BY for `list()` and `retrieve()`.
- Row counts: `reltuples = -1` is treated as "unknown" and gated on
physical size via `pg_relation_size` (a cheap stat call; `relpages` is
equally stale pre-vacuum). At or below `THRESHOLD_ESTIMATE_BYTES`
(~10MB, derived from `THRESHOLD_COUNT` at a conservative ~200 bytes/row)
the exact count runs as before: fast by construction, and it avoids
bogus estimates since Postgres floors never-vacuumed heaps at 10 pages,
so an empty table would otherwise report ~2K estimated rows. Above the
gate the count routes through the EXPLAIN-based
`pg_temp.count_estimate`, or returns `-1`/`is_estimate = true` in
read-only contexts where the temp function cannot be created. The scoped
branch embeds the estimated select via `literal()` instead of legacy's
apostrophe-only escaping, so it stays correct under
`standard_conforming_strings = off`. `enforceExactCount` unchanged.

**Studio:**

- The flag decision is contained in the data layer instead of
prop-drilled: a small imperative accessor
(`apps/studio/data/scoped-introspection.ts`) is hydrated from `useFlag`
via a one-line `useSyncScopedIntrospection()` call in `DefaultLayout`,
and the query functions read it internally when building the pg-meta
SQL. `DefaultLayout` is shared by both the Next and TanStack router
trees; hydrating from `_app.tsx` alone would leave TanStack-served pages
permanently unscoped since `routes/__root.tsx` mounts its own flag
provider. Cold loads cannot race the flag: the query functions await a
readiness promise that resolves only after the sync hook has hydrated
the accessor with a loaded flag store (immediately on self-hosted where
flags are disabled; a 5s safety net armed lazily on the first `ready()`
call - not at module import, which would let the timer expire before a
project page ever mounts - bounds genuine ConfigCat outages). No
component threading, no query-key changes (remaining tradeoff,
documented in the module: a mid-session flag flip can serve stale-keyed
caches until refetch, fine for a session-stable rollout flag). #47894's
existing threading is left as-is and gets deleted together with the flag
in the cleanup PR. Also fixes the previously-missing `scoped`
pass-through in `getTableRowsCount`.
- Flag-independent hardening: `useTableQuery` now sets `retryOnMount:
false`, `refetchOnWindowFocus: false` and `staleTime: 5min`. Errored
(timed-out) queries no longer refire on every grid cell remount, while
stale successful metadata still revalidates on mount after `staleTime`.
- `useTableApiAccessQuery` now passes `includedSchemas: [schemaName]`;
the client-side filter stays as a safety net.
- The rows-count query is `enabled`-gated on the permission check
settling, so a transiently-false `canSQLAdminWrite` can no longer cache
a read-only `-1` count for a writable user (read replicas short-circuit
synchronously as before).

**Regression guards (extending the #47894 infrastructure):**

- Execution-based scoped-vs-legacy equivalence tests for all four
queries: both variants run against the test database and are compared
with raw `toEqual` - no normalization, ids included (types across 6
option combos, privileges incl. multi-grantee + PUBLIC,
`tables.retrieve` for both identifier branches, row counts for every
case where the two paths must agree). Two documented exceptions where
only the LEGACY side is sorted, because a de-normalized diagnostic run
proved legacy emits genuinely plan-dependent order there (an
adversarial-FK fixture shows it is neither oid, name, nor creation
order): the `types.list` outer row order (scoped adds `order by t.oid`;
legacy has no ORDER BY) and the `tables.retrieve` relationships array
(scoped orders by `constraint_name` + column-name tie-breakers - a
composite two-column FK expands to 4 entries sharing one
constraint_name). Everything else (privileges via `aclexplode` over the
same relacl, columns by `ordinal_position`, primary keys by `indkey`
order, enums by `enumsortorder`) is byte-identical between the two paths
with no test-side help. The one intentional value divergence,
never-analyzed tables above the size gate where legacy's exact count is
the timeout bug itself, is asserted explicitly as a divergence.
- Plan-guard budgets for every scoped query against the stress catalog
(extended with 200 enums + 200 composite types). Residual seq scans are
justified in-budget: `pg_constraint` max 2 (no index on `confrelid`),
`pg_attrdef` max 1, `pg_authid` max 2 (scales with role count, not
schema count).
- Legacy templates carry a FROZEN do-not-edit marker (they must keep
matching production behavior until the flag cleanup deletes them); the
ordinary test suite runs against the legacy default, so behavioral drift
there fails regular tests.

### Validation

- pg-meta: typecheck clean; the affected suites (types,
table-privileges, tables, rows-count, catalog-plan-guard) pass in full.
- Cross-version: the scoped-vs-legacy equivalence and rows-count
behavioral suites were validated on PostgreSQL 14, 15, and 17 (identical
results on all three). Two version-marginal planner choices surfaced on
17 (`pg_type` / `pg_class` seq scan vs full-index bitmap for per-schema
listings, both structurally unavoidable without an index leading on the
namespace column) and are carried as justified plan-guard budget
entries. A full 468-test suite run sequentially: 452 passed, 16 failures
verified environmental (13 timeouts in an untouched file that passes
27/27 in isolation on the marathon-run cluster, 3 cluster-global role
collisions from container reuse).
- Studio: `pnpm --filter studio typecheck` clean; 39/39 tests across the
touched data hooks; eslint clean on touched files.

### Rollout

Same staged ConfigCat rollout as #47894 via `pgMetaScopedIntrospection`
(user-email targeting first, then percentage, then 100%). The
`useTableQuery` hardening and the API-access schema scoping ship
unflagged (behavior-safe). Gate before percentage rollout: functionally
verify the FK popover/selector UX under the new
`staleTime`/`retryOnMount` settings (a just-edited FK target must not
look stale anywhere Studio does not already refetch on save). Once fully
rolled out, the legacy templates and flag get deleted together with
#47894's in one cleanup PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 08:07:08 +02:00

456 lines
13 KiB
TypeScript

import { afterAll, beforeAll, expect, test } from 'vitest'
import pgMeta from '../src/index'
import { cleanupRoot, createTestDatabase } from './db/utils'
beforeAll(async () => {
// Any global setup if needed
})
afterAll(async () => {
await cleanupRoot()
})
const withTestDatabase = (
name: string,
fn: (db: Awaited<ReturnType<typeof createTestDatabase>>) => Promise<void>
) => {
test(name, async () => {
const db = await createTestDatabase()
try {
await fn(db)
} finally {
await db.cleanup()
}
})
}
withTestDatabase('list table privileges', async ({ executeQuery }) => {
const { sql, zod } = await pgMeta.tablePrivileges.list()
const res = zod.parse(await executeQuery(sql))
expect(
res.find(({ schema, name }) => schema === 'public' && name === 'todos')
).toMatchInlineSnapshot(
{ relation_id: expect.any(Number) },
`
{
"kind": "table",
"name": "todos",
"privileges": [
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "INSERT",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "SELECT",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "UPDATE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "DELETE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "TRUNCATE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "REFERENCES",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "TRIGGER",
},
],
"relation_id": Any<Number>,
"schema": "public",
}
`
)
})
type Priv = { grantor: string; grantee: string; privilege_type: string; is_grantable: boolean }
type PrivRow = { relation_id: number; schema: string; name: string; privileges: Priv[] }
withTestDatabase(
'scoped tablePrivileges.list/retrieve matches legacy (multiple grantees incl PUBLIC)',
async ({ executeQuery }) => {
// Multiple grantees including PUBLIC to exercise the PUBLIC (oid 0) branch of
// the grantee union and multi-grantee aggregation. Roles are cluster-global
// (not dropped by the per-test database cleanup), so guard the creation to
// stay idempotent when the test cluster is reused across runs.
await executeQuery(`
drop role if exists grantee_a;
drop role if exists grantee_b;
create role grantee_a;
create role grantee_b;
create table public.priv_demo (id int primary key, data text);
grant select, insert on public.priv_demo to grantee_a;
grant update (data) on public.priv_demo to grantee_b with grant option;
grant select on public.priv_demo to public;
`)
// list(): default and schema-filtered combos. RAW comparison -- no
// normalization, no sorting of rows or the privileges array.
for (const options of [{}, { includedSchemas: ['public'] }, { excludedSchemas: ['public'] }]) {
const legacy = await pgMeta.tablePrivileges.list(options)
const scoped = await pgMeta.tablePrivileges.list({ ...options, scoped: true })
const legacyRes = legacy.zod.parse(await executeQuery(legacy.sql)) as PrivRow[]
const scopedRes = scoped.zod.parse(await executeQuery(scoped.sql)) as PrivRow[]
expect(scopedRes, `list options: ${JSON.stringify(options)}`).toEqual(legacyRes)
}
// The scoped path must surface the PUBLIC grantee for priv_demo.
const { sql, zod } = await pgMeta.tablePrivileges.list({
includedSchemas: ['public'],
scoped: true,
})
const rows = zod.parse(await executeQuery(sql)) as PrivRow[]
const demo = rows.find((r) => r.name === 'priv_demo')!
expect(demo.privileges.some((p) => p.grantee === 'PUBLIC')).toBe(true)
// retrieve() by id and by schema+name.
const legacyById = await pgMeta.tablePrivileges.retrieve({ id: demo.relation_id })
const scopedById = await pgMeta.tablePrivileges.retrieve({
id: demo.relation_id,
scoped: true,
})
expect(scopedById.zod.parse((await executeQuery(scopedById.sql))[0])).toEqual(
legacyById.zod.parse((await executeQuery(legacyById.sql))[0])
)
const legacyByName = await pgMeta.tablePrivileges.retrieve({
name: 'priv_demo',
schema: 'public',
})
const scopedByName = await pgMeta.tablePrivileges.retrieve({
name: 'priv_demo',
schema: 'public',
scoped: true,
})
expect(scopedByName.zod.parse((await executeQuery(scopedByName.sql))[0])).toEqual(
legacyByName.zod.parse((await executeQuery(legacyByName.sql))[0])
)
}
)
withTestDatabase('revoke & grant table privileges', async ({ executeQuery }) => {
// Get initial table privileges
const { sql: listSql, zod: listZod } = await pgMeta.tablePrivileges.list()
const listRes = listZod.parse(await executeQuery(listSql))
const { relation_id } = listRes.find(
({ schema, name }) => schema === 'public' && name === 'todos'
)!
// Revoke all privileges
const { sql: revokeSql } = pgMeta.tablePrivileges.revoke([
{
relationId: relation_id,
grantee: 'postgres',
privilegeType: 'ALL',
},
])
await executeQuery(revokeSql)
// Verify privileges were revoked
const { sql: retrieveSql, zod: retrieveZod } = await pgMeta.tablePrivileges.retrieve({
id: relation_id,
})
let privs = retrieveZod.parse((await executeQuery(retrieveSql))[0])
expect(privs).toMatchInlineSnapshot(
{ relation_id: expect.any(Number) },
`
{
"kind": "table",
"name": "todos",
"privileges": [],
"relation_id": Any<Number>,
"schema": "public",
}
`
)
// Grant all privileges back
const { sql: grantSql } = pgMeta.tablePrivileges.grant([
{
relationId: relation_id,
grantee: 'postgres',
privilegeType: 'ALL',
},
])
await executeQuery(grantSql)
// Verify privileges were granted
const { sql: verifyGrantSql } = await pgMeta.tablePrivileges.retrieve({ id: relation_id })
privs = retrieveZod.parse((await executeQuery(verifyGrantSql))[0])
expect(privs).toMatchInlineSnapshot(
{ relation_id: expect.any(Number) },
`
{
"kind": "table",
"name": "todos",
"privileges": [
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "TRIGGER",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "REFERENCES",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "TRUNCATE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "DELETE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "UPDATE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "SELECT",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "INSERT",
},
],
"relation_id": Any<Number>,
"schema": "public",
}
`
)
})
withTestDatabase(
'revoke & grant table privileges w/ quoted table name',
async ({ executeQuery }) => {
// Create test role and schema
await executeQuery(`create role r; create schema "s 1"; create table "s 1"."t 1"();`)
// Get table privileges
const { sql: listSql, zod: listZod } = await pgMeta.tablePrivileges.list()
const listRes = listZod.parse(await executeQuery(listSql))
const { relation_id } = listRes.find(({ schema, name }) => schema === 's 1' && name === 't 1')!
// Grant all privileges
const { sql: grantSql } = pgMeta.tablePrivileges.grant([
{
relationId: relation_id,
grantee: 'r',
privilegeType: 'ALL',
},
])
await executeQuery(grantSql)
// Verify privileges were granted
const { sql: retrieveSql, zod: retrieveZod } = await pgMeta.tablePrivileges.retrieve({
id: relation_id,
})
let privs = retrieveZod.parse((await executeQuery(retrieveSql))[0])
expect(privs).toMatchInlineSnapshot(
{ relation_id: expect.any(Number) },
`
{
"kind": "table",
"name": "t 1",
"privileges": [
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "TRIGGER",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "REFERENCES",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "TRUNCATE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "DELETE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "UPDATE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "SELECT",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "INSERT",
},
{
"grantee": "r",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "TRIGGER",
},
{
"grantee": "r",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "REFERENCES",
},
{
"grantee": "r",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "TRUNCATE",
},
{
"grantee": "r",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "DELETE",
},
{
"grantee": "r",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "UPDATE",
},
{
"grantee": "r",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "SELECT",
},
{
"grantee": "r",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "INSERT",
},
],
"relation_id": Any<Number>,
"schema": "s 1",
}
`
)
// Revoke all privileges
const { sql: revokeSql } = pgMeta.tablePrivileges.revoke([
{
relationId: relation_id,
grantee: 'r',
privilegeType: 'ALL',
},
])
await executeQuery(revokeSql)
// Verify privileges were revoked
const { sql: verifyRevokeSql } = await pgMeta.tablePrivileges.retrieve({ id: relation_id })
privs = retrieveZod.parse((await executeQuery(verifyRevokeSql))[0])
expect(privs).toMatchInlineSnapshot(
{ relation_id: expect.any(Number) },
`
{
"kind": "table",
"name": "t 1",
"privileges": [
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "TRIGGER",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "REFERENCES",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "TRUNCATE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "DELETE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "UPDATE",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "SELECT",
},
{
"grantee": "postgres",
"grantor": "postgres",
"is_grantable": false,
"privilege_type": "INSERT",
},
],
"relation_id": Any<Number>,
"schema": "s 1",
}
`
)
}
)