mirror of
https://github.com/supabase/supabase.git
synced 2026-10-05 09:25:06 +03:00
The "Hire an agent" and "monitor" wording oversold the feature: it is just a prompt you give an agent to check health, security, performance, or resources, optionally on a schedule. Rename the group to "Agent prompts", drop "monitor" from the four child pages (Health, Security, Performance, Resources), and use plainer framing across the landing page and observability hub. URL slugs are unchanged. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Renamed the Observability section to “Agent prompts” and updated its page titles and descriptions to describe project checks. * Clarified that prompts read project data without changing it, and that findings can be sent through existing harness connections. * Updated setup guidance to refer to running prompts and using “checks” in task names. * Renamed the Health, Security, Performance, and Resources entries. The related links, schedules, and reporting details remain unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
217 lines
12 KiB
Plaintext
217 lines
12 KiB
Plaintext
---
|
|
id: 'detecting'
|
|
title: 'Detection checks'
|
|
description: 'Repeatable health, security, performance, and capacity checks with explicit inputs and outcomes'
|
|
---
|
|
|
|
Use these checks to identify evidence worth investigating. A finding does not establish a cause. The [agent prompts](/docs/guides/observability/automate-with-agents) use these same checks.
|
|
|
|
## Before running checks
|
|
|
|
- Identify the project and database instance. Use project-scoped [Supabase MCP](/docs/guides/ai-tools/mcp) with `read_only=true`.
|
|
- Run ClickHouse SQL with `query_logs`; supply an explicit UTC time range using the tool's input schema. Run Postgres SQL with `execute_sql`. In [Explorer](/dashboard/project/_/explorer), select **Run SQL**, then query source **Logs** or **Database**, respectively.
|
|
- Record observation time, windows, thresholds, and saved baseline. Defaults below are starting alert policies, not Supabase service guarantees. Record operator overrides before running.
|
|
- Failed tools, missing permissions or required fields, incomplete windows, and unavailable history make the affected check **unable to assess**. Continue independent checks. Zero recorded events alone does not prove service health.
|
|
|
|
Each check returns **finding**, **clear** (completed, no threshold crossed), or **unable to assess** with the missing input. Preserve this result even when a clear run sends no notification.
|
|
|
|
## Health
|
|
|
|
### Measure API and Auth server errors
|
|
|
|
**Input:** the last complete UTC hour and preceding complete hour, queried separately. Evaluate each source separately; API Gateway and Auth events are different observations, not unique requests to add together.
|
|
|
|
```sql
|
|
select source,
|
|
count() as events,
|
|
countIf(status between 100 and 599) as responses,
|
|
countIf(status between 500 and 599) as server_errors,
|
|
countIf(status in (401, 403)) as access_failures,
|
|
countIf(status is null or status < 100 or status > 599) as unknown_status
|
|
from (
|
|
select source,
|
|
toInt32OrNull(if(source = 'edge_logs',
|
|
log_attributes['response.status_code'], log_attributes['status'])) as status
|
|
from logs
|
|
where source in ('edge_logs', 'auth_logs')
|
|
)
|
|
group by source
|
|
order by source
|
|
limit 2;
|
|
```
|
|
|
|
**Signal:** compute `100 * server_errors / responses` per source. Report at least 20 server errors, a rate of at least 1%, and at least twice the preceding rate. When the preceding rate is zero, use the count and 1% conditions. Both windows need at least 100 responses; otherwise the comparison is unable to assess.
|
|
|
|
Rates use valid statuses only. Report `unknown_status` separately; no valid statuses makes the check unable to assess. Auth events without response statuses are not successful requests. A missing source row requires a capture/traffic check, not an assumed zero error rate.
|
|
|
|
**Next:** narrow to the source and hour. Collect at most five event IDs with timestamps and status, then follow [API error troubleshooting](/docs/guides/troubleshooting/discovering-and-interpreting-api-errors-in-the-logs-7xREI9). Redact paths and messages. After a fix, rerun on a comparable window.
|
|
|
|
### Check connection pressure
|
|
|
|
**Input:** a current Postgres snapshot with permission to read all sessions.
|
|
|
|
```sql
|
|
select
|
|
count(*) filter (where backend_type = 'client backend') as client_connections,
|
|
count(*) filter (where backend_type = 'client backend' and state = 'active') as active_connections,
|
|
current_setting('max_connections')::int as max_connections
|
|
from pg_stat_activity;
|
|
```
|
|
|
|
**Signal:** report client connections at 80% of `max_connections`. This is an instance-wide pressure indicator. Reserved slots, role limits, and pooler limits can constrain a client sooner; this does not measure slots available to an application.
|
|
|
|
**Next:** inspect [connection management](/docs/guides/database/connection-management) and [role counts](#collect-size-and-connection-measurements). Rerun after the workload or pooling change.
|
|
|
|
## Security
|
|
|
|
### Review advisor findings
|
|
|
|
**Action:** call `get_advisors` with `type: "security"`, using the tool's project scope. Report `WARN` and `ERROR` findings with the lint name, affected object, and documentation link. Keep `INFO` as context without alerting by default.
|
|
|
|
**Next:** follow the check documentation and verify the intended access model before proposing a change. Rerun the advisor after a fix. No findings does not prove the project is secure. See [Advisors](/docs/guides/observability/advisors) for other execution paths.
|
|
|
|
### Measure authentication and authorization failures
|
|
|
|
**Input/action:** run the [status-count query](#measure-api-and-auth-server-errors) for the last complete UTC day and preceding complete day, in separate requests of at most 24 hours. Evaluate each source separately.
|
|
|
|
**Signal:** compute `100 * access_failures / responses`. Apply the Health minimum of 100 responses in both windows. Report at least 20 failures, a rate of at least 1%, and at least twice the preceding rate. When the preceding rate is zero, use the count and 1% conditions. Apply the same unknown-status and missing-source rules.
|
|
|
|
**Next:** group failures by status and sanitized path, not by user, email, or IP. Investigate the client flow and [Auth error codes](/docs/guides/auth/debugging/error-codes). A spike is a review signal, not proof of an attack. Verify against a comparable window.
|
|
|
|
## Performance
|
|
|
|
### Find long-running sessions and blockers
|
|
|
|
**Input:** a current Postgres snapshot with permission to read all sessions. This cannot reconstruct sessions that ended between scheduled runs.
|
|
|
|
```sql
|
|
select pid, usename as role, state,
|
|
now() - query_start as query_age,
|
|
now() - xact_start as transaction_age,
|
|
wait_event_type, wait_event,
|
|
pg_blocking_pids(pid) as blocking_pids
|
|
from pg_stat_activity
|
|
where datname = current_database()
|
|
and pid <> pg_backend_pid()
|
|
and (
|
|
(state = 'active' and now() - query_start > interval '30 seconds')
|
|
or (state like 'idle in transaction%' and now() - xact_start > interval '30 seconds')
|
|
or cardinality(pg_blocking_pids(pid)) > 0
|
|
)
|
|
order by query_start
|
|
limit 20;
|
|
```
|
|
|
|
**Signal:** each row needs review. Nonempty `blocking_pids` identifies blockers; a long query or wait event alone does not. Query age is not lock-wait duration. Twenty returned rows may indicate truncation.
|
|
|
|
**Next:** inspect the PIDs using [database inspection](/docs/guides/observability/inspect#using-sql) and establish the transaction's purpose and impact. Do not recommend cancellation from age alone. Rerun to verify resolution.
|
|
|
|
### Compare query execution time
|
|
|
|
**Input:** enabled [pg_stat_statements](/docs/guides/database/extensions/pg_stat_statements), query-identifier visibility, and three saved snapshots spaced one hour apart. They define the preceding and current hour.
|
|
|
|
```sql
|
|
select
|
|
now() as observed_at,
|
|
s.dbid,
|
|
s.userid,
|
|
s.queryid,
|
|
s.toplevel,
|
|
s.calls,
|
|
s.total_exec_time,
|
|
i.stats_reset,
|
|
i.dealloc,
|
|
to_jsonb(s) ->> 'stats_since' as statement_stats_since
|
|
from
|
|
pg_stat_statements as s
|
|
cross join pg_stat_statements_info as i
|
|
where s.dbid = (select oid from pg_database where datname = current_database())
|
|
order by s.total_exec_time desc
|
|
limit 100;
|
|
```
|
|
|
|
**Signal:** match `(dbid, userid, queryid, toplevel)` within the same project instance. For each interval, compute `delta(total_exec_time) / delta(calls)` in milliseconds. Report a current mean of at least 100 ms and twice the preceding mean, with at least 20 calls in each interval.
|
|
|
|
Compare rows present in all snapshots with unchanged reset/start markers and counters that have not decreased. Discard comparisons after an upgrade, reset, or change to `dealloc` (entry eviction). If `statement_stats_since` is unavailable, require confirmation that no per-statement reset occurred. Missing history or reset provenance means unable to assess; start collecting snapshots. The top 100 rows are a sample, not full query coverage. Do not reset statistics to collect a baseline. See [Postgres statistics semantics](https://www.postgresql.org/docs/current/pgstatstatements.html).
|
|
|
|
**Next:** inspect the statement and its [query plan](/docs/guides/database/query-optimization#analyze-the-query-plan). Preserve a comparison window to verify any change.
|
|
|
|
### Review performance advisors
|
|
|
|
Call `get_advisors` with `type: "performance"`. Apply the Security severity policy: report `WARN` and `ERROR`; retain `INFO` as context. Follow the returned documentation, verify relevance to the workload, and rerun after a fix.
|
|
|
|
### Inspect cache misses
|
|
|
|
This optional diagnostic is cumulative, not an hourly alert or a measurement of physical disk reads:
|
|
|
|
```sql
|
|
select
|
|
sum(heap_blks_hit) as heap_hits,
|
|
sum(heap_blks_read) as heap_reads,
|
|
round(
|
|
100.0 * sum(heap_blks_hit) / nullif(sum(heap_blks_hit) + sum(heap_blks_read), 0),
|
|
2
|
|
) as heap_hit_percent
|
|
from pg_statio_user_tables;
|
|
```
|
|
|
|
Use a workload-specific baseline before alerting. A null ratio means no observed accesses. The operating system cache may serve a Postgres buffer miss. See [cache inspection](/docs/reference/cli/supabase-inspect-db-cache-hit).
|
|
|
|
## Capacity [#usage]
|
|
|
|
### Collect size and connection measurements
|
|
|
|
**Input/action:** read the same database instance daily at the same UTC time. Save numeric values and timestamps in authorized persistent harness state, or use an authorized historical metrics source. Do not create monitoring tables in the project.
|
|
|
|
```sql
|
|
select
|
|
now() as observed_at,
|
|
current_database() as database_name,
|
|
pg_database_size(current_database()) as database_bytes;
|
|
```
|
|
|
|
```sql
|
|
select
|
|
now() as observed_at,
|
|
schemaname,
|
|
relname as table_name,
|
|
pg_total_relation_size(relid) as total_bytes
|
|
from pg_catalog.pg_statio_user_tables
|
|
order by total_bytes desc
|
|
limit 20;
|
|
```
|
|
|
|
```sql
|
|
select now() as observed_at, usename as role, state, count(*) as connections
|
|
from pg_stat_activity
|
|
where datname = current_database() and backend_type = 'client backend'
|
|
group by role, state
|
|
order by connections desc
|
|
limit 100;
|
|
```
|
|
|
|
**Interpretation:** sizes are bytes, connections are a snapshot count, and table totals include indexes. A relation missing from the top 20 has not necessarily shrunk. Snapshots do not establish peak connection demand; use the [Metrics API](/docs/guides/observability/metrics) for a time series.
|
|
|
|
### Forecast a resource limit
|
|
|
|
**Input:** at least seven daily measurements of the same metric and scope, plus a confirmed limit in the same units. Record the limit's source and retrieval time. Database size is not total disk usage: a disk forecast needs disk-used bytes and disk capacity. Never compare table bytes or request counts with an unrelated plan limit.
|
|
|
|
**Signal:** when growth is positive, calculate:
|
|
|
|
```text
|
|
growth_per_day = (latest_value - earliest_value) / elapsed_days
|
|
days_remaining = (confirmed_limit - latest_value) / growth_per_day
|
|
```
|
|
|
|
Report when the current value already meets the confirmed limit, regardless of history. Otherwise, report a supported projection at most 14 days away, labeled as a linear estimate. Missing history, unknown limits, changed scope, or discontinuous measurements make the forecast unable to assess. Flat or falling values do not support an exhaustion date.
|
|
|
|
**Next:** carry the metric, units, history, limit source, and calculation to [compute and disk guidance](/docs/guides/platform/compute-and-disk). Measure again after a capacity change and update the stored limit.
|
|
|
|
### Compare request volume
|
|
|
|
Run the Health query for two separate complete UTC days. Compare API Gateway `events`; report at least 1,000 events and twice the preceding count. If the preceding count is zero, report new observed traffic without a growth percentage. Apply the missing-source rules. Request growth is workload context, not a capacity limit or billing total.
|
|
|
|
## Turn a detection into a diagnosis
|
|
|
|
Report the check, outcome, project, observation time, window or snapshot, threshold, measured values and units, and an evidence identifier. Include one investigation link and a verification step. Separate observations from hypotheses; do not invent a cause or remediation SQL. Use the [troubleshooting guides](/docs/guides/troubleshooting) to investigate the evidence.
|