mirror of
https://github.com/supabase/supabase.git
synced 2026-10-12 21:05:06 +03:00
## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Bug fix (performance), follow-up to #47894, plus regression-guard tests. ## What is the current behavior? #47894 scoped the Table Editor and entity-definition introspection queries, but four more `@supabase/pg-meta` query families still do O(catalog) work per request. On a production project with a very large catalog (hundreds of schemas, ~465K `pg_constraint` rows) they run 5 to 55 seconds each, trip the 58s `statement_timeout`, and spill sorts to temp files. During a recent "DB CPU > 85%" incident on such a project, 24 of 27 active backends were running these queries concurrently. 1. **`tables.retrieve()` (single-table lookup by name+schema or id)**: the `tables`/`columns` CTEs scan the whole catalog (`pg_class`, `pg_constraint`, `pg_index`, all of `pg_attribute`, per-table sizes) and the one-table predicate is applied only on the outer select. Same bug class #47894 fixed for the OID-based table editor query; this sibling path never got the treatment. It accounted for 94 of the 96 statement-timeout cancellations in the incident. 2. **Types listing**: the `t_enums` and `t_attributes` subqueries aggregate the entire `pg_enum` and every composite relation before the wrapper's schema filter applies. 3. **Table privileges**: `aclexplode` + double `pg_roles` join + GROUP BY over every relation in the database; schema/OID filters applied only after aggregation, in both `list()` and `retrieve()`. 4. **Row counts**: `getTableRowsCountSql` treats `reltuples = -1` (never-analyzed table) as "small table, run exact count(*)". A freshly bulk-loaded multi-million-row table times out on every Table Editor pagination render. Two Studio-side amplifiers turned one slow query into a sustained load storm: - `useTableQuery` (behind `tables.retrieve()`) mounts once per visible foreign-key grid cell via `ForeignKeyFormatter`, so a single Table Editor view fires ~20 concurrent copies against the FK target table. A timed-out query caches nothing, and TanStack retries errored no-data queries on every observer mount by default, so scrolling kept re-issuing the 58s scan. - `useTableApiAccessQuery` fetched table privileges for the entire database and filtered down to one schema client-side. ## What is the new behavior? **pg-meta (all behind the existing `pgMetaScopedIntrospection` flag, same rollout mechanism as #47894; `scoped: false` keeps serving the current SQL):** - `tables.retrieve()`: the identifier is resolved to a scalar `targetOid` init-plan and pushed into the base scan, primary-key, relationships (both FK directions kept: `conrelid` or `confrelid`) and columns CTEs. A materialized `target` CTE was deliberately avoided: it acts as an optimization barrier and forces the very seq scans being removed. - Types: filter `pg_type`/`pg_namespace` first, then compute enums/attributes per surviving row via correlated index-scan subqueries (`pg_enum(enumtypid, enumsortorder)`, `pg_attribute(attrelid, attnum)`). - Table privileges: schema/OID predicates injected into the base WHERE before `aclexplode`/GROUP BY for `list()` and `retrieve()`. - Row counts: `reltuples = -1` is treated as "unknown" and gated on physical size via `pg_relation_size` (a cheap stat call; `relpages` is equally stale pre-vacuum). At or below `THRESHOLD_ESTIMATE_BYTES` (~10MB, derived from `THRESHOLD_COUNT` at a conservative ~200 bytes/row) the exact count runs as before: fast by construction, and it avoids bogus estimates since Postgres floors never-vacuumed heaps at 10 pages, so an empty table would otherwise report ~2K estimated rows. Above the gate the count routes through the EXPLAIN-based `pg_temp.count_estimate`, or returns `-1`/`is_estimate = true` in read-only contexts where the temp function cannot be created. The scoped branch embeds the estimated select via `literal()` instead of legacy's apostrophe-only escaping, so it stays correct under `standard_conforming_strings = off`. `enforceExactCount` unchanged. **Studio:** - The flag decision is contained in the data layer instead of prop-drilled: a small imperative accessor (`apps/studio/data/scoped-introspection.ts`) is hydrated from `useFlag` via a one-line `useSyncScopedIntrospection()` call in `DefaultLayout`, and the query functions read it internally when building the pg-meta SQL. `DefaultLayout` is shared by both the Next and TanStack router trees; hydrating from `_app.tsx` alone would leave TanStack-served pages permanently unscoped since `routes/__root.tsx` mounts its own flag provider. Cold loads cannot race the flag: the query functions await a readiness promise that resolves only after the sync hook has hydrated the accessor with a loaded flag store (immediately on self-hosted where flags are disabled; a 5s safety net armed lazily on the first `ready()` call - not at module import, which would let the timer expire before a project page ever mounts - bounds genuine ConfigCat outages). No component threading, no query-key changes (remaining tradeoff, documented in the module: a mid-session flag flip can serve stale-keyed caches until refetch, fine for a session-stable rollout flag). #47894's existing threading is left as-is and gets deleted together with the flag in the cleanup PR. Also fixes the previously-missing `scoped` pass-through in `getTableRowsCount`. - Flag-independent hardening: `useTableQuery` now sets `retryOnMount: false`, `refetchOnWindowFocus: false` and `staleTime: 5min`. Errored (timed-out) queries no longer refire on every grid cell remount, while stale successful metadata still revalidates on mount after `staleTime`. - `useTableApiAccessQuery` now passes `includedSchemas: [schemaName]`; the client-side filter stays as a safety net. - The rows-count query is `enabled`-gated on the permission check settling, so a transiently-false `canSQLAdminWrite` can no longer cache a read-only `-1` count for a writable user (read replicas short-circuit synchronously as before). **Regression guards (extending the #47894 infrastructure):** - Execution-based scoped-vs-legacy equivalence tests for all four queries: both variants run against the test database and are compared with raw `toEqual` - no normalization, ids included (types across 6 option combos, privileges incl. multi-grantee + PUBLIC, `tables.retrieve` for both identifier branches, row counts for every case where the two paths must agree). Two documented exceptions where only the LEGACY side is sorted, because a de-normalized diagnostic run proved legacy emits genuinely plan-dependent order there (an adversarial-FK fixture shows it is neither oid, name, nor creation order): the `types.list` outer row order (scoped adds `order by t.oid`; legacy has no ORDER BY) and the `tables.retrieve` relationships array (scoped orders by `constraint_name` + column-name tie-breakers - a composite two-column FK expands to 4 entries sharing one constraint_name). Everything else (privileges via `aclexplode` over the same relacl, columns by `ordinal_position`, primary keys by `indkey` order, enums by `enumsortorder`) is byte-identical between the two paths with no test-side help. The one intentional value divergence, never-analyzed tables above the size gate where legacy's exact count is the timeout bug itself, is asserted explicitly as a divergence. - Plan-guard budgets for every scoped query against the stress catalog (extended with 200 enums + 200 composite types). Residual seq scans are justified in-budget: `pg_constraint` max 2 (no index on `confrelid`), `pg_attrdef` max 1, `pg_authid` max 2 (scales with role count, not schema count). - Legacy templates carry a FROZEN do-not-edit marker (they must keep matching production behavior until the flag cleanup deletes them); the ordinary test suite runs against the legacy default, so behavioral drift there fails regular tests. ### Validation - pg-meta: typecheck clean; the affected suites (types, table-privileges, tables, rows-count, catalog-plan-guard) pass in full. - Cross-version: the scoped-vs-legacy equivalence and rows-count behavioral suites were validated on PostgreSQL 14, 15, and 17 (identical results on all three). Two version-marginal planner choices surfaced on 17 (`pg_type` / `pg_class` seq scan vs full-index bitmap for per-schema listings, both structurally unavoidable without an index leading on the namespace column) and are carried as justified plan-guard budget entries. A full 468-test suite run sequentially: 452 passed, 16 failures verified environmental (13 timeouts in an untouched file that passes 27/27 in isolation on the marathon-run cluster, 3 cluster-global role collisions from container reuse). - Studio: `pnpm --filter studio typecheck` clean; 39/39 tests across the touched data hooks; eslint clean on touched files. ### Rollout Same staged ConfigCat rollout as #47894 via `pgMetaScopedIntrospection` (user-email targeting first, then percentage, then 100%). The `useTableQuery` hardening and the API-access schema scoping ship unflagged (behavior-safe). Gate before percentage rollout: functionally verify the FK popover/selector UX under the new `staleTime`/`retryOnMount` settings (a just-edited FK target must not look stale anywhere Studio does not already refetch on save). Once fully rolled out, the legacy templates and flag get deleted together with #47894's in one cleanup PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
456 lines
13 KiB
TypeScript
456 lines
13 KiB
TypeScript
import { afterAll, beforeAll, expect, test } from 'vitest'
|
|
|
|
import pgMeta from '../src/index'
|
|
import { cleanupRoot, createTestDatabase } from './db/utils'
|
|
|
|
beforeAll(async () => {
|
|
// Any global setup if needed
|
|
})
|
|
|
|
afterAll(async () => {
|
|
await cleanupRoot()
|
|
})
|
|
|
|
const withTestDatabase = (
|
|
name: string,
|
|
fn: (db: Awaited<ReturnType<typeof createTestDatabase>>) => Promise<void>
|
|
) => {
|
|
test(name, async () => {
|
|
const db = await createTestDatabase()
|
|
try {
|
|
await fn(db)
|
|
} finally {
|
|
await db.cleanup()
|
|
}
|
|
})
|
|
}
|
|
|
|
withTestDatabase('list table privileges', async ({ executeQuery }) => {
|
|
const { sql, zod } = await pgMeta.tablePrivileges.list()
|
|
const res = zod.parse(await executeQuery(sql))
|
|
|
|
expect(
|
|
res.find(({ schema, name }) => schema === 'public' && name === 'todos')
|
|
).toMatchInlineSnapshot(
|
|
{ relation_id: expect.any(Number) },
|
|
`
|
|
{
|
|
"kind": "table",
|
|
"name": "todos",
|
|
"privileges": [
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "INSERT",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "SELECT",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "UPDATE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "DELETE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "TRUNCATE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "REFERENCES",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "TRIGGER",
|
|
},
|
|
],
|
|
"relation_id": Any<Number>,
|
|
"schema": "public",
|
|
}
|
|
`
|
|
)
|
|
})
|
|
|
|
type Priv = { grantor: string; grantee: string; privilege_type: string; is_grantable: boolean }
|
|
type PrivRow = { relation_id: number; schema: string; name: string; privileges: Priv[] }
|
|
|
|
withTestDatabase(
|
|
'scoped tablePrivileges.list/retrieve matches legacy (multiple grantees incl PUBLIC)',
|
|
async ({ executeQuery }) => {
|
|
// Multiple grantees including PUBLIC to exercise the PUBLIC (oid 0) branch of
|
|
// the grantee union and multi-grantee aggregation. Roles are cluster-global
|
|
// (not dropped by the per-test database cleanup), so guard the creation to
|
|
// stay idempotent when the test cluster is reused across runs.
|
|
await executeQuery(`
|
|
drop role if exists grantee_a;
|
|
drop role if exists grantee_b;
|
|
create role grantee_a;
|
|
create role grantee_b;
|
|
create table public.priv_demo (id int primary key, data text);
|
|
grant select, insert on public.priv_demo to grantee_a;
|
|
grant update (data) on public.priv_demo to grantee_b with grant option;
|
|
grant select on public.priv_demo to public;
|
|
`)
|
|
|
|
// list(): default and schema-filtered combos. RAW comparison -- no
|
|
// normalization, no sorting of rows or the privileges array.
|
|
for (const options of [{}, { includedSchemas: ['public'] }, { excludedSchemas: ['public'] }]) {
|
|
const legacy = await pgMeta.tablePrivileges.list(options)
|
|
const scoped = await pgMeta.tablePrivileges.list({ ...options, scoped: true })
|
|
const legacyRes = legacy.zod.parse(await executeQuery(legacy.sql)) as PrivRow[]
|
|
const scopedRes = scoped.zod.parse(await executeQuery(scoped.sql)) as PrivRow[]
|
|
expect(scopedRes, `list options: ${JSON.stringify(options)}`).toEqual(legacyRes)
|
|
}
|
|
|
|
// The scoped path must surface the PUBLIC grantee for priv_demo.
|
|
const { sql, zod } = await pgMeta.tablePrivileges.list({
|
|
includedSchemas: ['public'],
|
|
scoped: true,
|
|
})
|
|
const rows = zod.parse(await executeQuery(sql)) as PrivRow[]
|
|
const demo = rows.find((r) => r.name === 'priv_demo')!
|
|
expect(demo.privileges.some((p) => p.grantee === 'PUBLIC')).toBe(true)
|
|
|
|
// retrieve() by id and by schema+name.
|
|
const legacyById = await pgMeta.tablePrivileges.retrieve({ id: demo.relation_id })
|
|
const scopedById = await pgMeta.tablePrivileges.retrieve({
|
|
id: demo.relation_id,
|
|
scoped: true,
|
|
})
|
|
expect(scopedById.zod.parse((await executeQuery(scopedById.sql))[0])).toEqual(
|
|
legacyById.zod.parse((await executeQuery(legacyById.sql))[0])
|
|
)
|
|
|
|
const legacyByName = await pgMeta.tablePrivileges.retrieve({
|
|
name: 'priv_demo',
|
|
schema: 'public',
|
|
})
|
|
const scopedByName = await pgMeta.tablePrivileges.retrieve({
|
|
name: 'priv_demo',
|
|
schema: 'public',
|
|
scoped: true,
|
|
})
|
|
expect(scopedByName.zod.parse((await executeQuery(scopedByName.sql))[0])).toEqual(
|
|
legacyByName.zod.parse((await executeQuery(legacyByName.sql))[0])
|
|
)
|
|
}
|
|
)
|
|
|
|
withTestDatabase('revoke & grant table privileges', async ({ executeQuery }) => {
|
|
// Get initial table privileges
|
|
const { sql: listSql, zod: listZod } = await pgMeta.tablePrivileges.list()
|
|
const listRes = listZod.parse(await executeQuery(listSql))
|
|
const { relation_id } = listRes.find(
|
|
({ schema, name }) => schema === 'public' && name === 'todos'
|
|
)!
|
|
|
|
// Revoke all privileges
|
|
const { sql: revokeSql } = pgMeta.tablePrivileges.revoke([
|
|
{
|
|
relationId: relation_id,
|
|
grantee: 'postgres',
|
|
privilegeType: 'ALL',
|
|
},
|
|
])
|
|
await executeQuery(revokeSql)
|
|
|
|
// Verify privileges were revoked
|
|
const { sql: retrieveSql, zod: retrieveZod } = await pgMeta.tablePrivileges.retrieve({
|
|
id: relation_id,
|
|
})
|
|
let privs = retrieveZod.parse((await executeQuery(retrieveSql))[0])
|
|
expect(privs).toMatchInlineSnapshot(
|
|
{ relation_id: expect.any(Number) },
|
|
`
|
|
{
|
|
"kind": "table",
|
|
"name": "todos",
|
|
"privileges": [],
|
|
"relation_id": Any<Number>,
|
|
"schema": "public",
|
|
}
|
|
`
|
|
)
|
|
|
|
// Grant all privileges back
|
|
const { sql: grantSql } = pgMeta.tablePrivileges.grant([
|
|
{
|
|
relationId: relation_id,
|
|
grantee: 'postgres',
|
|
privilegeType: 'ALL',
|
|
},
|
|
])
|
|
await executeQuery(grantSql)
|
|
|
|
// Verify privileges were granted
|
|
const { sql: verifyGrantSql } = await pgMeta.tablePrivileges.retrieve({ id: relation_id })
|
|
privs = retrieveZod.parse((await executeQuery(verifyGrantSql))[0])
|
|
expect(privs).toMatchInlineSnapshot(
|
|
{ relation_id: expect.any(Number) },
|
|
`
|
|
{
|
|
"kind": "table",
|
|
"name": "todos",
|
|
"privileges": [
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "TRIGGER",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "REFERENCES",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "TRUNCATE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "DELETE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "UPDATE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "SELECT",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "INSERT",
|
|
},
|
|
],
|
|
"relation_id": Any<Number>,
|
|
"schema": "public",
|
|
}
|
|
`
|
|
)
|
|
})
|
|
|
|
withTestDatabase(
|
|
'revoke & grant table privileges w/ quoted table name',
|
|
async ({ executeQuery }) => {
|
|
// Create test role and schema
|
|
await executeQuery(`create role r; create schema "s 1"; create table "s 1"."t 1"();`)
|
|
|
|
// Get table privileges
|
|
const { sql: listSql, zod: listZod } = await pgMeta.tablePrivileges.list()
|
|
const listRes = listZod.parse(await executeQuery(listSql))
|
|
const { relation_id } = listRes.find(({ schema, name }) => schema === 's 1' && name === 't 1')!
|
|
|
|
// Grant all privileges
|
|
const { sql: grantSql } = pgMeta.tablePrivileges.grant([
|
|
{
|
|
relationId: relation_id,
|
|
grantee: 'r',
|
|
privilegeType: 'ALL',
|
|
},
|
|
])
|
|
await executeQuery(grantSql)
|
|
|
|
// Verify privileges were granted
|
|
const { sql: retrieveSql, zod: retrieveZod } = await pgMeta.tablePrivileges.retrieve({
|
|
id: relation_id,
|
|
})
|
|
let privs = retrieveZod.parse((await executeQuery(retrieveSql))[0])
|
|
expect(privs).toMatchInlineSnapshot(
|
|
{ relation_id: expect.any(Number) },
|
|
`
|
|
{
|
|
"kind": "table",
|
|
"name": "t 1",
|
|
"privileges": [
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "TRIGGER",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "REFERENCES",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "TRUNCATE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "DELETE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "UPDATE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "SELECT",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "INSERT",
|
|
},
|
|
{
|
|
"grantee": "r",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "TRIGGER",
|
|
},
|
|
{
|
|
"grantee": "r",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "REFERENCES",
|
|
},
|
|
{
|
|
"grantee": "r",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "TRUNCATE",
|
|
},
|
|
{
|
|
"grantee": "r",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "DELETE",
|
|
},
|
|
{
|
|
"grantee": "r",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "UPDATE",
|
|
},
|
|
{
|
|
"grantee": "r",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "SELECT",
|
|
},
|
|
{
|
|
"grantee": "r",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "INSERT",
|
|
},
|
|
],
|
|
"relation_id": Any<Number>,
|
|
"schema": "s 1",
|
|
}
|
|
`
|
|
)
|
|
|
|
// Revoke all privileges
|
|
const { sql: revokeSql } = pgMeta.tablePrivileges.revoke([
|
|
{
|
|
relationId: relation_id,
|
|
grantee: 'r',
|
|
privilegeType: 'ALL',
|
|
},
|
|
])
|
|
await executeQuery(revokeSql)
|
|
|
|
// Verify privileges were revoked
|
|
const { sql: verifyRevokeSql } = await pgMeta.tablePrivileges.retrieve({ id: relation_id })
|
|
privs = retrieveZod.parse((await executeQuery(verifyRevokeSql))[0])
|
|
expect(privs).toMatchInlineSnapshot(
|
|
{ relation_id: expect.any(Number) },
|
|
`
|
|
{
|
|
"kind": "table",
|
|
"name": "t 1",
|
|
"privileges": [
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "TRIGGER",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "REFERENCES",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "TRUNCATE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "DELETE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "UPDATE",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "SELECT",
|
|
},
|
|
{
|
|
"grantee": "postgres",
|
|
"grantor": "postgres",
|
|
"is_grantable": false,
|
|
"privilege_type": "INSERT",
|
|
},
|
|
],
|
|
"relation_id": Any<Number>,
|
|
"schema": "s 1",
|
|
}
|
|
`
|
|
)
|
|
}
|
|
)
|