From 8fed0fd760f27b96569e0046919f8dfeb3f9fda2 Mon Sep 17 00:00:00 2001 From: TzeYiing Date: Fri, 7 Apr 2023 00:21:51 +0800 Subject: [PATCH 1/3] docs: add IAM and service account information --- .../self-hosting-analytics/introduction.mdx | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/apps/docs/docs/ref/self-hosting-analytics/introduction.mdx b/apps/docs/docs/ref/self-hosting-analytics/introduction.mdx index 5d6efe41eda..39261bedebd 100644 --- a/apps/docs/docs/ref/self-hosting-analytics/introduction.mdx +++ b/apps/docs/docs/ref/self-hosting-analytics/introduction.mdx @@ -26,8 +26,55 @@ However, it's important to note that certain [differences](#differences) may ari ## Getting Started +### Pre-requisites + +Logflare currently requires BigQuery usage. You will need to create a Google Cloud project with billing enabled. + +The requirements are as follows after creating the project: + +- Project ID +- Project number +- A service account key + +#### Setting up BigQuery Service Account + +To ensure that you have sufficient permissions to insert into your Google Cloud BigQuery, ensure that you have created a service account with either: + +- BigQuery Admin role; or +- The following permissions: + - bigquery.datasets.create + - bigquery.datasets.get + - bigquery.datasets.getIamPolicy + - bigquery.datasets.update + - bigquery.jobs.create + - bigquery.routines.create + - bigquery.routines.update + - bigquery.tables.create + - bigquery.tables.delete + - bigquery.tables.get + - bigquery.tables.getData + - bigquery.tables.update + - bigquery.tables.updateData + +We recommend setting the BigQuery Admin role, as it simplifies permissions setup. + +#### Downloading the Service Account key + +After the service account is created, you will need to create a key for the service account. This key will sign the JWTs for API requests that the Analytics server makes with BigQuery. + +### Docker Compose + Using the example [self-hosting stack based on docker-compose](https://github.com/supabase/supabase/tree/master/docker), you include the logging related services using the following command +You will first need to update the `.env.example` file with the necessary environment variables. + +- `GOOGLE_PROJECT_ID` +- `GOOGLE_PROJECT_NUMBER` + +You will need to place your Service Account key in your present working directory with the filename `gcloud.json`. + +Thereafter, you can run the docker-compose commands and include the logging-specific compose file. + ```bash # assuming you clone the supabase/supabase repo. cd docker @@ -41,6 +88,18 @@ This would include two additional docker services to your compose stack: Logflar server. +#### Vector Usage + +In the Docker Compose example, we utilize vector to coordinate the logging pipeline between the services. However, if you peer into the [vector configuration file](https://github.com/supabase/supabase/blob/master/docker/volumes/logs/vector.yml), you will be able to see that Vector sends logs to the Analytics ingestion endpoint. + +If you need to customize the logging pipeline for your own needs, you must ensure that the payloads matches the expected event schema structure. Without the correct structure, it would cause the Studio Logs UI features to break. + +### Standalone Docker Container + +If desired, you can utilize the standalone docker-container. Please refer to the [docker-compose file](https://github.com/supabase/supabase/tree/master/docker/docker-compose-logging.yml) for required docker configuration. + +Additional supplementary technical documentation on self-hosting and using the `supabase/logflare` image for a full Logflare experience is available at the [official Logflare documentation](https://docs.logflare.app/self-hosting/). + ## Differences API logs rely on Kong instead of the Supabase Cloud API Gateway. Logs from Kong are not enriched with platform-only data. From 346150f02deb49cb766ba5683fb18eacd35b2874 Mon Sep 17 00:00:00 2001 From: TzeYiing Date: Fri, 7 Apr 2023 00:25:27 +0800 Subject: [PATCH 2/3] docs: add link to gcp docs --- apps/docs/docs/ref/self-hosting-analytics/introduction.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/apps/docs/docs/ref/self-hosting-analytics/introduction.mdx b/apps/docs/docs/ref/self-hosting-analytics/introduction.mdx index 39261bedebd..ea4297b6c87 100644 --- a/apps/docs/docs/ref/self-hosting-analytics/introduction.mdx +++ b/apps/docs/docs/ref/self-hosting-analytics/introduction.mdx @@ -56,6 +56,8 @@ To ensure that you have sufficient permissions to insert into your Google Cloud - bigquery.tables.update - bigquery.tables.updateData +You can create the service account via the web console or `gcloud`, as per the [Google Cloud documentation](https://cloud.google.com/iam/docs/keys-create-delete). In the web console, you can create the key by navigating to IAM > Service Accounts > Actions (dropdown) > Manage Keys + We recommend setting the BigQuery Admin role, as it simplifies permissions setup. #### Downloading the Service Account key From ba4981eea41406df1fb5405f165e0ac1cc61fe7d Mon Sep 17 00:00:00 2001 From: TzeYiing Date: Fri, 7 Apr 2023 00:29:38 +0800 Subject: [PATCH 3/3] chore: update local dev docs --- apps/docs/pages/guides/cli/local-development.mdx | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/apps/docs/pages/guides/cli/local-development.mdx b/apps/docs/pages/guides/cli/local-development.mdx index f6fe7c6c637..ca38587e786 100644 --- a/apps/docs/pages/guides/cli/local-development.mdx +++ b/apps/docs/pages/guides/cli/local-development.mdx @@ -412,14 +412,15 @@ supabase functions deploy Local logs rely on the Supabase Analytics Server. This can be enabled via the CLI configuration, and requires a Google Cloud project and BigQuery access. -The Google Cloud project must have billing enabled. + The Google Cloud project must have billing enabled. Read more about this requirement + [here](https://supabase.com/docs/reference/self-hosting-analytics/introduction#bigquery). Requirements: -1. Google Cloud project number (must be paid) +1. Google Cloud project number 2. Google Cloud project ID -3. Google Cloud JWT, obtained through [Google Cloud IAM dashboard](https://console.cloud.google.com/iam-admin/iam) +3. Google Cloud Service Account Key, obtained through [Google Cloud IAM dashboard](https://console.cloud.google.com/iam-admin/iam), with BigQuery Admin role assigned to the service account. Once you have these 3 items, follow these steps: @@ -433,7 +434,7 @@ gcp_project_id = "my-project-id" gcp_jwt_path = "supabase/gcloud.json" ``` -2. Place your JWT file at the corresponding path and ensure that it is correctly named. +2. Place your service account key (a JSON file) at the corresponding path and ensure that it is correctly named. 3. Start your local stack using `supabase start`