migrate(docs): troubleshooting 19 (#30945)
* migrate(docs): troubleshooting 19 grafana, metrics * Update grafana-not-displaying-data-sXJrMj.mdx * Update grafana-not-displaying-data-sXJrMj.mdx * Update how-to-view-database-metrics-uqf2z_.mdx * Update interpreting-supabase-grafana-io-charts-MUynDR.mdx * added keywords * Apply suggestions from code review --------- Co-authored-by: TheOtherBrian1 <91111415+TheOtherBrian1@users.noreply.github.com>
No files matched your search
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title = "Grafana not displaying data"
|
||||
github_url = "https://github.com/orgs/supabase/discussions/30579"
|
||||
date_created = "2024-11-21T03:45:05+00:00"
|
||||
topics = ["database"]
|
||||
keywords = ["grafana", "docker", "metrics", "configuration"]
|
||||
---
|
||||
|
||||
This guide is for identifying configuration mistakes in [self-hosted Supabase Grafana installations](https://supabase.com/docs/guides/monitoring-troubleshooting/metrics#deploying-supabase-grafana)
|
||||
|
||||
## Step 1: Ping your Grafana Endpoint
|
||||
|
||||
Use the below cURL command to make sure your metrics endpoint returns data:
|
||||
|
||||
```sh
|
||||
curl https://<YOUR_PROJECT_REF>.supabase.co/customer/v1/privileged/metrics --user 'service_role:<SERVICE_ROLE_KEY>'
|
||||
```
|
||||
|
||||
## Step 2: Set your Grafana Dashboard to auto-refresh in the top right corner
|
||||
|
||||

|
||||
|
||||
## Step 3: Make sure your docker container has the default configurations
|
||||
|
||||
Run the following command in the terminal:
|
||||
|
||||
```sh
|
||||
docker ps -f name=supabase-grafana
|
||||
```
|
||||
|
||||
The output should look something like this:
|
||||
|
||||

|
||||
|
||||
Here it is in an easier to read format
|
||||
|
||||
- CONTAINER ID: < container id >
|
||||
- IMAGE: supabase-grafana-supabase-grafana
|
||||
- COMMAND: /entrypoint.sh
|
||||
- CREATED: < time >
|
||||
- STATUS: Up < unit of time > ago
|
||||
- PORTS: 3000/tcp, 0.0.0.0:8000 → 8080/tcp
|
||||
- NAMES: supabase-grafana-supabase-grafana-1
|
||||
|
||||
## Step 4: Enter the container
|
||||
|
||||
Try running the following terminal command:
|
||||
|
||||
```sh
|
||||
docker exec -it <container id> bash
|
||||
```
|
||||
|
||||
## Step 5: Check the environment variables for errors
|
||||
|
||||
Run the following in the docker container:
|
||||
|
||||
```sh
|
||||
printenv | egrep 'GRAFANA_PASSWORD|SUPABASE_PROJECT_REF|SUPABASE_SERVICE_ROLE_KEY'
|
||||
```
|
||||
|
||||
Ensure the values are correct by comparing them with those in the Dashboard. Users have previously encountered issues by accidentally omitting the last character of their strings, so a thorough check is essential.
|
||||
|
||||
## Step 6: Go to the root folder and check permissions on the `entrypoint.sh` file
|
||||
|
||||
Run the following terminal commands:
|
||||
|
||||
```sh
|
||||
cd /
|
||||
ls -l | grep entrypoint.sh
|
||||
```
|
||||
|
||||
`entrypoint.sh` should have the following permissions:
|
||||
|
||||
```sh
|
||||
-rwxr-xr-x
|
||||
```
|
||||
|
||||
If off, update the values
|
||||
|
||||
```sh
|
||||
chmod +x entrypoint.sh
|
||||
```
|
||||
@@ -0,0 +1,11 @@
|
||||
---
|
||||
title = "How to View Database Metrics"
|
||||
github_url = "https://github.com/orgs/supabase/discussions/20938"
|
||||
date_created = "2024-02-01T17:48:11+00:00"
|
||||
topics = ["database", "platform"]
|
||||
keywords = ["metrics", "grafana", "monitoring"]
|
||||
---
|
||||
|
||||
To monitor real-time metrics of your database, like CPU, EBS, active database connections, and memory usage, you can deploy a Grafana Dashboard. Check our [GitHub repo](https://github.com/supabase/supabase-grafana) for setup instructions for local or free [Fly.io](http://fly.io/) deployments. Refer to our concise [documentation](https://supabase.com/docs/guides/platform/metrics) to learn more about the metrics endpoint.
|
||||
|
||||
While the [Dashboard's Reports Page](https://supabase.com/dashboard/project/_/reports) displays some metric data, it provides hourly averages, not real-time by the second data. However, it offers query metrics, which the Grafana Dashboard does not include.
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
title = "Interpreting Supabase Grafana CPU charts"
|
||||
github_url = "https://github.com/orgs/supabase/discussions/27022"
|
||||
date_created = "2024-06-05T04:46:27+00:00"
|
||||
topics = ["platform", "database"]
|
||||
keywords = ["cpu", "grafana", "metrics"]
|
||||
---
|
||||
|
||||
> [Guide](https://supabase.com/docs/guides/monitoring-troubleshooting/metrics#deploying-supabase-grafana) for setting up Supabase Grafana
|
||||
|
||||
# CPU
|
||||
|
||||
Here are examples of unhealthy CPU utilization:
|
||||
|
||||

|
||||

|
||||
|
||||
The CPU chart shows 4 distinct metrics of interest:
|
||||
|
||||
- **Yellow**: It represents CPU utilized in [kernel space](https://en.wikipedia.org/wiki/User_space_and_kernel_space) (privileged OS operations). If this is high, it may be a sign that your app is connecting/disconnecting too aggressively. It could also be symptomatic of extension-related errors.
|
||||
- **Blue**: It represents requests in user space and mainly reflects the CPU usage from regular queries. For optimization tips, check out the links at the end of the page.
|
||||
- **Red**: It represents cycles the CPU spent idle because it was waiting on IO tasks. Any amount of red often implies [disk](https://github.com/orgs/supabase/discussions/27003) or, indirectly, [memory](https://github.com/orgs/supabase/discussions/27021) problems. The links outline how to address this type of issue.
|
||||
- **Green**: These are CPU cycles spent idle.
|
||||
|
||||
As the CPU peaks towards 100%, queries and database tasks will begin to throttle, as they won't have enough time or access to the CPU.
|
||||
|
||||
### Other useful Supabase Grafana guides:
|
||||
|
||||
- [Connections](https://github.com/orgs/supabase/discussions/27141)
|
||||
- [Disk](https://github.com/orgs/supabase/discussions/27003)
|
||||
- [Memory](https://github.com/orgs/supabase/discussions/27021)
|
||||
|
||||
### Optimizing:
|
||||
|
||||
1. [Optimize your queries](https://supabase.com/docs/guides/database/query-optimization).
|
||||
2. [Add indexes](https://github.com/orgs/supabase/discussions/22449) if possible.
|
||||
3. [Increasing the compute size](https://supabase.com/docs/guides/platform/compute-add-ons)
|
||||
4. [Distribute load by using read-replicas](https://supabase.com/dashboard/project/_/settings/infrastructure)
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
title = "Interpreting Supabase Grafana IO charts"
|
||||
github_url = "https://github.com/orgs/supabase/discussions/27003"
|
||||
date_created = "2024-06-04T15:32:14+00:00"
|
||||
topics = ["database", "platform"]
|
||||
keywords = ["io", "disk", "database", "grafana"]
|
||||
---
|
||||
|
||||
> [Supabase Grafana Installation Guide](https://supabase.com/docs/guides/platform/metrics#deploying-supabase-grafana)
|
||||
|
||||
There are two primary values that matter for IO:
|
||||
|
||||
- **Disk Throughput**: how much data can be moved to and from disk per second
|
||||
- **IOPS(Input/Output per second)**: how many read/write requests can be performed against your disk per second
|
||||
|
||||
Each compute instance has unique IO settings. The settings at the time this was written are listed below.
|
||||
| Plan | Baseline Disk Throughput | Baseline IOPS |
|
||||
|---------------|--------------------------|---------------|
|
||||
| Nano (free) | 43 Mbps | 250 IOPS |
|
||||
| Micro | 87 Mbps | 500 IOPS |
|
||||
| Small | 174 Mbps | 1,000 IOPS |
|
||||
| Medium | 347 Mbps | 2,000 IOPS |
|
||||
| Large | 630 Mbps | 3,000 IOPS |
|
||||
| XL | 1,048 Mbps | 3,000 IOPS |
|
||||
| 2XL | 1,048 Mbps | 3,000 IOPS |
|
||||
| 4XL | 1,048 Mbps | 3,000 IOPS |
|
||||
| 8XL | 1,048 Mbps | 3,000 IOPS |
|
||||
| 12XL | 1,048 Mbps | 3,000 IOPS |
|
||||
| 16XL | 1,048 Mbps | 3,000 IOPS |
|
||||
|
||||
Compute sizes below XL, to accommodate their limitations, have burst budgets. This is when instances are allowed to utilize 1048 Mbps of disk throughput and 3,000 IOPS for 30 minutes before returning back to their baseline behavior.
|
||||
|
||||
There are other metrics that indicate IO strain.
|
||||
|
||||
This example shows a 16XL database exhibiting severe IO strain:
|
||||
|
||||

|
||||
|
||||
Its Disk IOPS is constantly near peak capacity:
|
||||
|
||||

|
||||
|
||||
Its throughput is also high:
|
||||
|
||||

|
||||
|
||||
As a side-effect, its CPU is encumbered by heavy Busy IOWait activity:
|
||||
|
||||

|
||||
|
||||
Excessive IO usage is highly problematic as it clarifies that your database is expending more IO than it normally is intended to manage. This can be caused by
|
||||
|
||||
- **Excessive and needless sequential scans:** poorly indexed tables force requests to scan disk ([guide to resolve](https://github.com/orgs/supabase/discussions/22449))
|
||||
- **Too little cache**: There is not enough memory, so instead of reading data from the memory cache, it is accessed from disk ([guide to inspect](https://github.com/orgs/supabase/discussions/22449))
|
||||
- **Poorly optimized RLS policies**: RLS that rely heavily on joins are more likely to hit disk. If possible, they should optimized ([RLS best practice guide](https://supabase.com/docs/guides/database/postgres/row-level-security#rls-performance-recommendations))
|
||||
- **Excessive bloat**: This is the least likely to cause major issues, but bloat can take up space, preventing data on disk from being placed in the same locality. This can force the database to scan more pages than necessary. ([explainer guide](https://supabase.com/blog/postgres-bloat))
|
||||
- **Uploading high amounts of data:** temporarily increase compute add-on size for the duration of the uploads
|
||||
- **Insufficient memory**: Sometimes an inadequate amount of memory forces queries to hit disk instead of the memory cache. Address memory issues ([guide](https://github.com/orgs/supabase/discussions/27021)) can reduce disk strain.
|
||||
|
||||
If a database exhibited some of these metrics for prolonged periods, then there are a few primary approaches:
|
||||
|
||||
- Scale the database to get more IO if possible
|
||||
- Optimize queries/tables or refactor database/app to reduce IO
|
||||
- Spin-up a [read-replica](https://supabase.com/dashboard/project/_/settings/infrastructure)
|
||||
- Modify IO configs in the [Database Settings](https://supabase.com/dashboard/project/_/settings/database)
|
||||
- [Partitions](https://supabase.com/docs/guides/database/partitions): Generally should be used on very large tables to minimize data pulled from disk
|
||||
|
||||
Other useful Supabase Grafana guides:
|
||||
|
||||
- [Connections](https://github.com/orgs/supabase/discussions/27141)
|
||||
- [Memory](https://github.com/orgs/supabase/discussions/27021)
|
||||
- [CPU](https://github.com/orgs/supabase/discussions/27022)
|
||||
|
||||
### Esoteric factors
|
||||
|
||||
**Webhooks**:
|
||||
Supabase webhooks use the pg_net extension to handle requests. The `net.http_request_queue` table isn't indexed to keep write costs low. However, if you upload millions of rows to a webhook-enabled table too quickly, it can significantly increase the read costs for the extension.
|
||||
|
||||
To check if reads are becoming expensive, run:
|
||||
|
||||
```sql
|
||||
select count(*) as exact_count from net.http_request_queue;
|
||||
-- the number should be relatively low <20,000
|
||||
```
|
||||
|
||||
If you encounter this issue, you can either:
|
||||
|
||||
1. Increase your compute size to help handle the large volume of requests.
|
||||
|
||||
2. Truncate the table to clear the queue:
|
||||
|
||||
```sql
|
||||
TRUNCATE net.http_request_queue;
|
||||
```
|
||||
@@ -2,8 +2,8 @@
|
||||
title = "Why is my camelCase name not working in Postgres functions or RLS policies?"
|
||||
github_url = "https://github.com/orgs/supabase/discussions/12893"
|
||||
date_created = "2023-03-08T16:26:51+00:00"
|
||||
topics = [ "database", "API" ]
|
||||
keywords = [ "camelCase", "Postgres", "functions", "RLS" ]
|
||||
topics = [ "database" ]
|
||||
keywords = ["rls", "rpc"]
|
||||
database_id = "9dbcf760-7b1f-4cc4-8653-4e884fd02dad"
|
||||
---
|
||||
|
||||
|
||||
|
After Width: | Height: | Size: 227 KiB |
|
After Width: | Height: | Size: 92 KiB |
|
After Width: | Height: | Size: 17 KiB |
|
After Width: | Height: | Size: 136 KiB |
|
After Width: | Height: | Size: 63 KiB |
|
After Width: | Height: | Size: 51 KiB |
|
After Width: | Height: | Size: 125 KiB |
|
After Width: | Height: | Size: 113 KiB |