diff --git a/apps/kb/package.json b/apps/kb/package.json index c352bcedb80..57c20827124 100644 --- a/apps/kb/package.json +++ b/apps/kb/package.json @@ -25,6 +25,7 @@ "remark-gfm": "^4.0.1", "remark-parse": "^11.0.0", "remark-stringify": "^11.0.0", + "sharp": "^0.35.4", "ui": "workspace:*", "ui-patterns": "workspace:*", "unified": "^11.0.5", diff --git a/apps/kb/src/assets/guides/migrate-from-amazon-rds/amazon-rds_credentials.png b/apps/kb/src/assets/guides/migrate-from-amazon-rds/amazon-rds_credentials.png new file mode 100644 index 00000000000..9a692329b15 Binary files /dev/null and b/apps/kb/src/assets/guides/migrate-from-amazon-rds/amazon-rds_credentials.png differ diff --git a/apps/kb/src/assets/guides/migrate-from-amazon-rds/database-settings-host.png b/apps/kb/src/assets/guides/migrate-from-amazon-rds/database-settings-host.png new file mode 100644 index 00000000000..1206b19d479 Binary files /dev/null and b/apps/kb/src/assets/guides/migrate-from-amazon-rds/database-settings-host.png differ diff --git a/apps/kb/src/assets/guides/migrate-from-mssql/database-settings-host.png b/apps/kb/src/assets/guides/migrate-from-mssql/database-settings-host.png new file mode 100644 index 00000000000..6e912bc8359 Binary files /dev/null and b/apps/kb/src/assets/guides/migrate-from-mssql/database-settings-host.png differ diff --git a/apps/kb/src/assets/guides/migrate-from-mysql/database-settings-host.png b/apps/kb/src/assets/guides/migrate-from-mysql/database-settings-host.png new file mode 100644 index 00000000000..67a653b47b0 Binary files /dev/null and b/apps/kb/src/assets/guides/migrate-from-mysql/database-settings-host.png differ diff --git a/apps/kb/src/assets/guides/migrate-from-render/render_dashboard.png b/apps/kb/src/assets/guides/migrate-from-render/render_dashboard.png new file mode 100644 index 00000000000..1645901dc23 Binary files /dev/null and b/apps/kb/src/assets/guides/migrate-from-render/render_dashboard.png differ diff --git a/apps/kb/src/content/guides/migrate-from-amazon-rds.mdx b/apps/kb/src/content/guides/migrate-from-amazon-rds.mdx new file mode 100644 index 00000000000..d64468566d8 --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-amazon-rds.mdx @@ -0,0 +1,92 @@ +--- +title: 'Migrate from Amazon RDS to Supabase' +description: 'Migrate your Amazon RDS MySQL or MS SQL database to Supabase.' +topics: ['Migration', 'Database'] +--- + +This guide aims to exhibit the process of transferring your Amazon RDS database from any of these engines Postgres, MySQL or MS SQL to Supabase's Postgres database. Although Amazon RDS is a favored managed database service provided by AWS, it may not suffice for all use cases. Supabase, on the other hand, provides an excellent free and open source option that encompasses all the necessary backend features to develop a product: a Postgres database, authentication, instant APIs, edge functions, real-time subscriptions, and storage. + +Supabase's core is Postgres, enabling the use of row-level security and providing access to over 40 Postgres extensions. By migrating from Amazon RDS to Supabase, you can leverage Postgres to its fullest potential and acquire all the features you need to complete your project. + +## Retrieve your Amazon RDS database credentials + +1. Sign in to your [Amazon RDS account](https://aws.amazon.com/rds/). +1. Select the region where your RDS database is located. +1. Navigate to the **Databases** tab. +1. Select the database that you want to migrate. +1. In the **Connectivity & Security** tab, note down the Endpoint and the port number. +1. In the **Configuration** tab, note down the Database name and the Username. +1. If you do not have the password, create a new one and note it down. + +![Copying RDS credentials from AWS Management Console](../../assets/guides/migrate-from-amazon-rds/amazon-rds_credentials.png) + +## Retrieve your Supabase host + +1. If you're new to Supabase, [create a project](https://database.new). Make a note of your password, you will need this later. If you forget it, you can [reset it here](/dashboard/project/_/database/settings). +1. On your project dashboard, click [Connect](/dashboard/project/_?showConnect=true&method=session) +1. Under the Session pooler, click on the View parameters under the connect string. Note your Host (`$SUPABASE_HOST`). + +![Finding Supabase host address](../../assets/guides/migrate-from-amazon-rds/database-settings-host.png) + +## Migrate the database + +The fastest way to migrate your database is with the Supabase migration tool on +[Google Colab](https://colab.research.google.com/github/mansueli/Supa-Migrate/blob/main/Amazon_RDS_to_Supabase.ipynb). + +Alternatively, you can use [pgloader](https://github.com/dimitri/pgloader), a flexible and powerful data migration tool that supports a wide range of source database engines, including MySQL and MS SQL, and migrates the data to a Postgres database. For databases using the Postgres engine, we recommend using the [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html) and [psql](https://www.postgresql.org/docs/current/app-psql.html) command line tools, which are included in a full Postgres installation. + +### Migrate using Colab + +1. Select the Database Engine from the Source database in the dropdown +1. Set the environment variables (`HOST`, `USER`, `SOURCE_DB`,`PASSWORD`, `SUPABASE_URL`, and `SUPABASE_PASSWORD`) in the Colab notebook. +1. Run the first two steps in [the notebook](https://colab.research.google.com/github/mansueli/Supa-Migrate/blob/main/Amazon_RDS_to_Supabase.ipynb) in order. The first sets engine and installs the necessary files. +1. Run the third step to start the migration. This will take a few minutes. + +### Migrate from MySQL with pgloader + +1. Install pgloader. +2. Create a configuration file (e.g., config.load). + + For your destination, use your Supabase connection string with `Use connection pooling` enabled, and the mode set to `Session`. You can get the string from your [`Database Settings`](/dashboard/project/_/settings/general). + + ```sql + load database + from mysql://user:password@host/source_db + into postgres://postgres.xxxx:password@xxxx.pooler.supabase.com:5432/postgres + alter schema 'public' owner to 'postgres'; + set wal_buffers = '64MB', max_wal_senders = 0, statement_timeout = 0, work_mem to '2GB'; + ``` + +3. Run the migration with pgloader + + ```bash + pgloader config.load + ``` + +### Migrate from MSSQL + +1. Install pgloader. +2. Create a configuration file (e.g., config.load). + + ```sql + LOAD DATABASE + FROM mssql://USER:PASSWORD@HOST/SOURCE_DB + INTO postgres://postgres.xxxx:password@xxxx.pooler.supabase.com:6543/postgres + ALTER SCHEMA 'public' OWNER TO 'postgres'; + set wal_buffers = '64MB', max_wal_senders = 0, statement_timeout = 0, work_mem to '2GB'; + ``` + +3. Run the migration with pgloader + + ```bash + pgloader config.load + ``` + +> [!CAUTION] +> +> - If you're planning to migrate a database larger than 6 GB, we recommend [upgrading to at least a Large compute add-on](/docs/guides/platform/compute-and-disk). This will ensure you have the necessary resources to handle the migration efficiently. +> - We strongly advise you to pre-provision the disk space you will need for your migration. On paid projects, you can do this by navigating to the [Infrastructure settings](/dashboard/project/_/settings/infrastructure) page. For more information on disk scaling and disk limits, check out our [disk settings](/docs/guides/platform/compute-and-disk#disk) documentation. + +## Enterprise + +[Contact us](https://forms.supabase.com/enterprise) if you need more help migrating your project. diff --git a/apps/kb/src/content/guides/migrate-from-auth0.mdx b/apps/kb/src/content/guides/migrate-from-auth0.mdx new file mode 100644 index 00000000000..f31f9ffd673 --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-auth0.mdx @@ -0,0 +1,233 @@ +--- +title: 'Migrate from Auth0 to Supabase Auth' +description: 'Migrate your users from Auth0 to Supabase Auth.' +topics: ['Migration', 'Auth'] +--- + +You can migrate your users from Auth0 to Supabase Auth. + +Changing authentication providers for a production app is an important operation. It can affect most aspects of your application. Prepare in advance by reading this guide, and develop a plan for handling the key migration steps and possible problems. + +With advance planning, a smooth and safe Auth migration is possible. + +## Before you begin + +Before beginning, consider the answers to the following questions. They will help you need decide if you need to migrate, and which strategy to use: + +- How do Auth provider costs scale as your user base grows? +- Does the new Auth provider provide all needed features? (for example, OAuth, password logins, Security Assertion Markup Language (SAML), Multi-Factor Authentication (MFA)) +- Is downtime acceptable during the migration? +- What is your timeline to migrate before terminating the old Auth provider? + +## Migration strategies + +Depending on your evaluation, you may choose to go with one of the following strategies: + +1. Rolling migration +2. One-off migration + +| Strategy | Advantages | Disadvantages | +| -------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Rolling | | | +| One-off | | | + +## Migration steps + +Auth provider migrations require 2 main steps: + +1. Export your user data from the old provider (Auth0) +1. Import the data into your new provider (Supabase Auth) + +### Step 1: Export your user data + +Auth0 provides two methods for exporting user data: + +1. Use the [Auth0 data export feature](https://auth0.com/docs/troubleshoot/customer-support/manage-subscriptions/export-data) +1. Use the [Auth0 management API](https://auth0.com/docs/api/management/v2/users/get-users). This endpoint has a rate limit, so you may need to export your users in several batches. + +To export password hashes and MFA factors, contact Auth0 support. + +### Step 2: Import your users into Supabase Auth + +The steps for importing your users depends on the sign-in methods that you support. + +See the following sections for how to import users with: + +- [Password-based sign-in](#password-based-methods) +- [Passwordless sign-in](#passwordless-methods) +- [OAuth](#oauth) + +#### Password-based methods + +For users who sign in with passwords, we recommend a hybrid approach to reduce downtime: + +1. For new users, use Supabase Auth for sign up. +1. Migrate existing users in a one-off migration. + +##### Sign up new users + +Sign up new users using Supabase Auth's [signin methods](/docs/guides/auth/passwords#signing-up-with-an-email-and-password). + +##### Migrate existing users to Supabase Auth + +Migrate existing users to Supabase Auth. This requires two main steps: first, check which users need to be migrated, then create their accounts using the Supabase admin endpoints. + +1. Get your Auth 0 user export and password hash export lists. +1. Filter for users who use password sign-in. + - Under the `identities` field in the user object, these users will have `auth0` as a provider. In the same identity object, you can find their Auth0 `user_id`. + - Check that the user has a corresponding password hash by comparing their Auth0 `user_id` to the `oid` field in the password hash export. +1. Use Supabase Auth's [admin create user](/docs/reference/javascript/auth-admin-createuser) method to recreate the user in Supabase Auth. If the user has a confirmed email address or phone number, set `email_confirm` or `phone_confirm` to `true`. + + ```ts + import { createClient } from '@supabase/supabase-js' + + const supabase = createClient('your_project_url', 'your_supabase_api_key') + + // ---cut--- + const { data, error } = await supabase.auth.admin.createUser({ + email: 'valid.email@supabase.io', + password_hash: '$2y$10$a9pghn27d7m0ltXvlX8LiOowy7XfFw0hW0G80OjKYQ1jaoejaA7NC', + email_confirm: true, + }) + ``` + + > [!NOTE] + > **Supported password hashing algorithms** + > + > Supabase supports bcrypt and Argon2 password hashes. + + If you have a plaintext password instead of a hash, you can provide that instead. Supabase Auth will handle hashing the password for you. (Passwords are **always** stored hashed.) + + ```ts + import { createClient } from '@supabase/supabase-js' + + const supabase = createClient('your_project_url', 'your_supabase_api_key') + + // ---cut--- + const { data, error } = await supabase.auth.admin.createUser({ + email: 'valid.email@supabase.io', + password: 'supersecurepassword123!', + }) + ``` + +1. To sign in your migrated users, use the Supabase Auth [sign in methods](/docs/reference/javascript/auth-signinwithpassword). + + To check for edge cases where users aren't successfully migrated, use a fallback strategy. This ensures that users can continue to sign in seamlessly: + 1. Try to sign in the user with Supabase Auth. + 1. If the signin fails, try to sign in with Auth0. + 1. If Auth0 signin succeeds, call the admin create user method again to create the user in Supabase Auth. + +#### Passwordless methods + +For passwordless signin via email or phone, check for users with verified email addresses or phone numbers. Create these users in Supabase Auth with `email_confirm` or `phone_confirm` set to `true`: + +```ts +import { createClient } from '@supabase/supabase-js' + +const supabase = createClient('your_project_url', 'your_supabase_api_key') + +// ---cut--- +const { data, error } = await supabase.auth.admin.createUser({ + email: 'valid.email@supabase.io', + email_confirm: true, +}) +``` + +Check your Supabase Auth [email configuration](/docs/guides/auth/auth-smtp) and configure your [email template](/dashboard/project/_/auth/templates) for use with magic links. See the [Email templates guide](/docs/guides/auth/auth-email-templates) to learn more. + +Once you have imported your users, you can sign them in using the [`signInWithOtp`](/docs/reference/javascript/auth-signinwithotp) method. + +#### OAuth + +Configure your OAuth providers in Supabase by following the [Social login guides](/docs/guides/auth/social-login). + +For both new and existing users, sign in the user using the [`signInWithOAuth`](/docs/reference/javascript/auth-signinwithoauth) method. This works without pre-migrating existing users, since the user always needs to sign in through the OAuth provider before being redirected to your service. + +After the user has completed the OAuth flow successfully, you can check if the user is a new or existing user in Auth0 by mapping their social provider id to Auth0. Auth0 stores the social provider ID in the user ID, which has the format `provider_name|provider_id` (for example, `github|123456`). See the [Auth0 identity docs](https://auth0.com/docs/manage-users/user-accounts/identify-users) to learn more. + +## Mapping between Auth0 and Supabase Auth + +Each Auth provider has its own schema for tracking users and user information. + +In Supabase Auth, your users are stored in your project's database under the `auth` schema. Every user has an identity (unless the user is an anonymous user), which represents the signin method they can use with Supabase. This is represented by the `auth.users` and `auth.identities` table. + +See the [Users](/docs/guides/auth/users) and [Identities](/docs/guides/auth/identities) sections to learn more. + +### Mapping user metadata and custom claims + +Supabase Auth provides 2 fields which you can use to map user-specific metadata from Auth0: + +- `auth.users.raw_user_meta_data` : For storing non-sensitive user metadata that the user can update (e.g full name, age, favorite color). +- `auth.users.raw_app_meta_data` : For storing non-sensitive user metadata that the user should not be able to update (e.g pricing plan, access control roles). + +Both columns are accessible from the admin user methods. To create a user with custom metadata, you can use the following method: + +```ts +import { createClient } from '@supabase/supabase-js' + +const supabase = createClient('your_project_url', 'your_supabase_api_key') + +// ---cut--- +const { data, error } = await supabase.auth.admin.createUser({ + email: 'valid.email@supabase.io', + user_metadata: { + full_name: 'Foo Bar', + }, + app_metadata: { + role: 'admin', + }, +}) +``` + +> [!CAUTION] +> These fields will be exposed in the user's access token JWT so it is recommended not to store excessive metadata in these fields. + +These fields are stored as columns in the `auth.users` table using the `jsonb` type. Both fields can be updated by using the admin [`updateUserById` method](/docs/reference/javascript/auth-admin-updateuserbyid). If you want to allow the user to update their own `raw_user_meta_data` , you can use the [`updateUser` method](/docs/reference/javascript/auth-updateuser). + +If you have a lot of user-specific metadata to store, it is recommended to create your own table in a private schema that uses the user id as a foreign key: + +```sql +create table private.user_metadata ( + id int generated always as identity, + user_id uuid references auth.users(id) on delete cascade, + user_metadata jsonb +); +``` + +## Frequently Asked Questions (FAQ) + +### I have IDs assigned to existing users in my database, how can I maintain these IDs? + +All users stored in Supabase Auth use the UUID V4 format as the ID. If your UUID format is identical, you can specify it in the admin create user method like this: + +> [!NOTE] +> New users in Supabase Auth will always be created with a UUID V4 ID by default. + +```ts +// specify a custom id +const { data, error } = await supabase.auth.admin.createUser({ + id: 'e7f5ae65-376e-4d05-a18c-10a91295727a', + email: 'valid.email@supabase.io', +}) +``` + +### How can I allow my users to retain their existing password? + +Supabase Auth never stores passwords as plaintext. Since Supabase Auth supports reading bcrypt and argon2 password hashes, you can import your users passwords if they use the same hashing algorithm. New users in Supabase Auth who use password-based sign-in methods will always use a bcrypt hash. Passwords are stored in the `auth.users.encrypted_password` column. + +### My users have multi-factor authentication (MFA) enabled, how do I make sure they don't have to set up MFA again? + +You can obtain an export of your users' MFA secrets by opening a support ticket with Auth0, similar to obtaining the export for password hashes. Supabase Auth only supports time-based one-time passwords (TOTP). Users who have TOTP-based factors may need to re-enroll using their choice of TOTP-based authenticator instead (e.g. 1Password / Google authenticator). + +### How do I migrate existing SAML Single Sign-On (SSO) connections? + +Customers may need to link their identity provider with Supabase Auth separately, but their users should still be able to sign in as per normal after authenticating with their identity provider. For more information about SSO with SAML 2.0, you can check out [this guide](/docs/guides/auth/enterprise-sso/auth-sso-saml). If you want to migrate your existing SAML SSO connections from Auth0 to Supabase Auth, reach out to us via support. + +### How do I migrate my Auth0 organizations to Supabase? + +This isn't supported by Supabase Auth yet. + +## Useful references + +- [Migrating 125k users from Auth0 to Supabase](https://kevcodez.medium.com/migrating-125-000-users-from-auth0-to-supabase-81c0568de307) +- [Loper to Supabase migration](https://eigen.sh/posts/auth-migration) diff --git a/apps/kb/src/content/guides/migrate-from-firebase-auth.mdx b/apps/kb/src/content/guides/migrate-from-firebase-auth.mdx new file mode 100644 index 00000000000..8e2c36fd1ab --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-firebase-auth.mdx @@ -0,0 +1,90 @@ +--- +title: 'Migrate from Firebase Auth to Supabase' +description: 'Migrate Firebase Auth users to Supabase Auth.' +topics: ['Migration', 'Auth'] +--- + +Supabase provides several [tools](https://github.com/supabase-community/firebase-to-supabase/tree/main/auth) to help migrate auth users from a Firebase project to a Supabase project. There are two parts to the migration process: + +- `firestoreusers2json` ([TypeScript](https://github.com/supabase-community/firebase-to-supabase/blob/main/auth/firestoreusers2json.ts), [JavaScript](https://github.com/supabase-community/firebase-to-supabase/blob/main/auth/firestoreusers2json.js)) exports users from an existing Firebase project to a `.json` file on your local system. +- `import_users` ([TypeScript](https://github.com/supabase-community/firebase-to-supabase/blob/main/auth/import_users.ts), [JavaScript](https://github.com/supabase-community/firebase-to-supabase/blob/main/auth/import_users.js)) imports users from a saved `.json` file into your Supabase project (inserting those users into the `auth.users` table of your `Postgres` database instance). + +## Set up the migration tool + +1. Clone the [`firebase-to-supabase`](https://github.com/supabase-community/firebase-to-supabase) repository: + + ```bash + git clone https://github.com/supabase-community/firebase-to-supabase.git + ``` + +1. In the `/auth` directory, create a file named `supabase-service.json` with the following contents: + + ```json + { + "host": "database.server.com", + "password": "secretpassword", + "user": "postgres", + "database": "postgres", + "port": 5432 + } + ``` + +1. On your project dashboard, click [Connect](/dashboard/project/_?showConnect=true&method=session) +1. Under the Session pooler, click on the View parameters under the connect string. Replace the `Host` and `User` fields with the values shown. +1. Enter the password you used when you created your Supabase project in the `password` entry in the `supabase-service.json` file. + +## Generate a Firebase private key + +1. Sign in to your [Firebase Console](https://console.firebase.google.com/project) and open your project. +1. Click the gear icon next to **Project Overview** in the sidebar and select **Project Settings**. +1. Click **Service Accounts** and select **Firebase Admin SDK**. +1. Click **Generate new private key**. +1. Rename the downloaded file to `firebase-service.json`. + +## Save your Firebase password hash parameters + +1. Sign in to your [Firebase Console](https://console.firebase.google.com/project) and open your project. +1. Select **Authentication** (Build section) in the sidebar. +1. Select **Users** in the top menu. +1. At the top right of the users list, open the menu (3 dots) and click **Password hash parameters**. +1. Copy and save the parameters for `base64_signer_key`, `base64_salt_separator`, `rounds`, and `mem_cost`. + +```text Sample +hash_config { + algorithm: SCRYPT, + base64_signer_key: XXXX/XXX+XXXXXXXXXXXXXXXXX+XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==, + base64_salt_separator: Aa==, + rounds: 8, + mem_cost: 14, +} +``` + +## Command line options + +### Dump Firestore users to a JSON file + +`node firestoreusers2json.js [] []` + +- `filename.json`: (optional) output filename (defaults to `./users.json`) +- `batchSize`: (optional) number of users to fetch in each batch (defaults to 100) + +### Import JSON users file to Supabase Auth (Postgres: `auth.users`) + +`node import_users.js []` + +- `path_to_json_file`: full local path and filename of JSON input file (of users) +- `batch_size`: (optional) number of users to process in a batch (defaults to 100) + +## Notes + +For more advanced migrations, including the use of a middleware server component for verifying a user's existing Firebase password and updating that password in your Supabase project the first time a user logs in, see the [`firebase-to-supabase` repo](https://github.com/supabase-community/firebase-to-supabase/tree/main/auth). + +## Resources + +- [Supabase vs Firebase](/alternatives/supabase-vs-firebase) +- [Firestore Data Migration](/kb/guides/platform/migrating-to-supabase/firestore-data) +- [Firestore Storage Migration](/kb/guides/platform/migrating-to-supabase/firebase-storage) + +## Migrate to Supabase + +[Contact us](https://forms.supabase.com/firebase-migration) if you need more help migrating your project. diff --git a/apps/kb/src/content/guides/migrate-from-firebase-storage.mdx b/apps/kb/src/content/guides/migrate-from-firebase-storage.mdx new file mode 100644 index 00000000000..9bcf6dec6c0 --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-firebase-storage.mdx @@ -0,0 +1,66 @@ +--- +title: 'Migrate from Firebase Storage to Supabase' +description: 'Migrate Firebase Storage files to Supabase Storage.' +topics: ['Migration', 'Storage'] +--- + +Supabase provides several [tools](https://github.com/supabase-community/firebase-to-supabase/tree/main/storage) to convert storage files from Firebase Storage to Supabase Storage. Conversion is a two-step process: + +1. Files are downloaded from a Firebase storage bucket to a local filesystem. +2. Files are uploaded from the local filesystem to a Supabase storage bucket. + +## Set up the migration tool + +1. Clone the [`firebase-to-supabase`](https://github.com/supabase-community/firebase-to-supabase) repository: + + ```bash + git clone https://github.com/supabase-community/firebase-to-supabase.git + ``` + +1. In the `/storage` directory, rename [supabase-keys-sample.js](https://github.com/supabase-community/firebase-to-supabase/blob/main/storage/supabase-keys-sample.js) to `supabase-keys.js`. +1. Go to your Supabase project's [API settings](/dashboard/project/_/settings/api) in the Dashboard. +1. Copy the **Project URL** and update the `SUPABASE_URL` value in `supabase-keys.js`. +1. Under **Project API keys**, copy the **secret** key and update the `SUPABASE_KEY` value in `supabase-keys.js`. + +## Generate a Firebase private key + +1. Sign in to your [Firebase Console](https://console.firebase.google.com/project) and open your project. +1. Click the gear icon next to **Project Overview** in the sidebar and select **Project Settings**. +1. Click **Service Accounts** and select **Firebase Admin SDK**. +1. Click **Generate new private key**. +1. Rename the downloaded file to `firebase-service.json`. + +## Command line options + +### Download Firestore Storage bucket to a local filesystem folder + +`node download.js [] [] [] []` + +- ``: The prefix of the files to download. To process the root bucket, use an empty prefix: "". +- ``: (optional) Name of subfolder for downloaded files. The selected folder is created as a subfolder of the current folder (e.g., `./downloads/`). The default is `downloads`. +- ``: (optional) The default is 100. +- ``: (optional) Stop after processing this many files. For no limit, use `0`. +- ``: (optional) Begin processing at this `pageToken`. + +To process in batches using multiple command-line executions, you must use the same parameters with a new `` on subsequent calls. Use the token displayed on the last call to continue the process at a given point. + +### Upload files to Supabase Storage bucket + +`node upload.js ` + +- ``: The prefix of the files to download. To process all files, use an empty prefix: "". +- ``: Name of subfolder of files to upload. The selected folder is read as a subfolder of the current folder (e.g., `./downloads/`). The default is `downloads`. +- ``: Name of the bucket to upload to. + +> [!NOTE] +> If the bucket doesn't exist, it's created as a `non-public` bucket. You must set permissions on this new bucket in the [Supabase Dashboard](/dashboard/project/_/storage/buckets) before users can download any files. + +## Resources + +- [Supabase vs Firebase](/alternatives/supabase-vs-firebase) +- [Firestore Data Migration](/kb/guides/platform/migrating-to-supabase/firestore-data) +- [Firebase Auth Migration](/kb/guides/platform/migrating-to-supabase/firebase-auth) + +## Migrate to Supabase + +[Contact us](https://forms.supabase.com/firebase-migration) if you need more help migrating your project. diff --git a/apps/kb/src/content/guides/migrate-from-firestore-data.mdx b/apps/kb/src/content/guides/migrate-from-firestore-data.mdx new file mode 100644 index 00000000000..bd6b69fe04b --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-firestore-data.mdx @@ -0,0 +1,213 @@ +--- +title: 'Migrate from Firebase Firestore to Supabase' +description: 'Migrate your Firebase Firestore database to a Supabase Postgres database.' +topics: ['Migration', 'Database'] +--- + +Supabase provides several [tools](https://github.com/supabase-community/firebase-to-supabase/tree/main/firestore) to convert data from a Firebase Firestore database to a Supabase Postgres database. The process copies the entire contents of a single Firestore `collection` to a single Postgres `table`. + +The Firestore `collection` is "flattened" and converted to a table with basic columns of one of the following types: `text`, `numeric`, `boolean`, or `jsonb`. If your structure is more complex, you can write a program to split the newly-created `json` file into multiple, related tables before you import your `json` file(s) to Supabase. + +## Set up the migration tool + +1. Clone the [`firebase-to-supabase`](https://github.com/supabase-community/firebase-to-supabase) repository: + + ```bash + git clone https://github.com/supabase-community/firebase-to-supabase.git + ``` + +1. In the `/firestore` directory, create a file named `supabase-service.json` with the following contents: + + ```json + { + "host": "database.server.com", + "password": "secretpassword", + "user": "postgres", + "database": "postgres", + "port": 5432 + } + ``` + +1. On your project dashboard, click [Connect](/dashboard/project/_?showConnect=true&method=session) +1. Under the Session pooler, click on the View parameters under the connect string. Replace the `Host` and `User` fields with the values shown. +1. Enter the password you used when you created your Supabase project in the `password` entry in the `supabase-service.json` file. + +## Generate a Firebase private key + +1. Sign in to your [Firebase Console](https://console.firebase.google.com/project) and open your project. +1. Click the gear icon next to **Project Overview** in the sidebar and select **Project Settings**. +1. Click **Service Accounts** and select **Firebase Admin SDK**. +1. Click **Generate new private key**. +1. Rename the downloaded file to `firebase-service.json`. + +## Command line options + +### List all Firestore collections + +`node collections.js` + +### Dump Firestore collection to JSON file + +`node firestore2json.js [] []` + +- `batchSize` (optional) defaults to 1000 +- output filename is `.json` +- `limit` (optional) defaults to 0 (no limit) + +#### Customize the JSON file with hooks + +You can customize the way your JSON file is written using a [custom hook](#custom-hooks). A common use for this is to "flatten" the JSON file, or to split nested data into separate, related database tables. For example, you could take a Firestore document that looks like this: + +```json Firestore +[{ "user": "mark", "score": 100, "items": ["hammer", "nail", "glue"] }] +``` + +And split it into two files (one table for users and one table for items): + +```json Users +[{ "user": "mark", "score": 100 }] +``` + +```json Items +[ + { "user": "mark", "item": "hammer" }, + { "user": "mark", "item": "nail" }, + { "user": "mark", "item": "glue" } +] +``` + +### Import JSON file to Supabase (Postgres) + +`node json2supabase.js [] []` + +- `` The full path of the file you created in the previous step (`Dump Firestore collection to JSON file `), such as `./my_collection.json` +- `[]` (optional) Is one of: + - `none` (default) No primary key is added to the table. + - `smallserial` Creates a key using `(id SMALLSERIAL PRIMARY KEY)` (autoincrementing 2-byte integer). + - `serial` Creates a key using `(id SERIAL PRIMARY KEY)` (autoincrementing 4-byte integer). + - `bigserial` Creates a key using `(id BIGSERIAL PRIMARY KEY)` (autoincrementing 8-byte integer). + - `uuid` Creates a key using `(id UUID PRIMARY KEY DEFAULT gen_random_uuid())` (randomly generated UUID). + - `firestore_id` Creates a key using `(id TEXT PRIMARY KEY)` (uses existing `firestore_id` random text as key). +- `[]` (optional) Name of primary key. Defaults to "id". + +## Custom hooks + +Hooks are used to customize the process of exporting a collection of Firestore documents to JSON. They can be used for: + +- Customizing or modifying keys +- Calculating data +- Flattening nested documents into related SQL tables + +### Write a custom hook + +#### Create a `.js` file for your collection + +If your Firestore collection is called `users`, create a file called `users.js` in the current folder. + +#### Construct your `.js` file + +The basic format of a hook file looks like this: + +```js +module.exports = (collectionName, doc, recordCounters, writeRecord) => { + // modify the doc here + return doc +} +``` + +##### Parameters + +- `collectionName`: The name of the collection you are processing. +- `doc`: The current document (JSON object) being processed. +- `recordCounters`: An internal object that keeps track of how many records have been processed in each collection. +- `writeRecord`: This function automatically handles the process of writing data to other JSON files (useful for "flatting" your document into separate JSON files to be written to separate database tables). `writeRecord` takes the following parameters: + - `name`: Name of the JSON file to write to. + - `doc`: The document to write to the file. + - `recordCounters`: The same `recordCounters` object that was passed to this hook (passes it on). + +### Examples + +#### Add a new (unique) numeric key to a collection + +```js +module.exports = (collectionName, doc, recordCounters, writeRecord) => { + doc.unique_key = recordCounter[collectionName] + 1 + return doc +} +``` + +#### Add a timestamp of when this record was dumped from Firestore + +```js +module.exports = (collectionName, doc, recordCounters, writeRecord) => { + doc.dump_time = new Date().toISOString() + return doc +} +``` + +#### Flatten JSON into separate files + +Flatten the `users` collection into separate files: + +```json +[ + { + "uid": "abc123", + "name": "mark", + "score": 100, + "weapons": ["toothpick", "needle", "rock"] + }, + { + "uid": "xyz789", + "name": "chuck", + "score": 9999999, + "weapons": ["hand", "foot", "head"] + } +] +``` + +The `users.js` hook file: + +```js +module.exports = (collectionName, doc, recordCounters, writeRecord) => { + for (let i = 0; i < doc.weapons.length; i++) { + const weapon = { + uid: doc.uid, + weapon: doc.weapons[i], + } + writeRecord('weapons', weapon, recordCounters) + } + delete doc.weapons // moved to separate file + return doc +} +``` + +The result is two separate JSON files: + +```json users.json +[ + { "uid": "abc123", "name": "mark", "score": 100 }, + { "uid": "xyz789", "name": "chuck", "score": 9999999 } +] +``` + +```json weapons.json +[ + { "uid": "abc123", "weapon": "toothpick" }, + { "uid": "abc123", "weapon": "needle" }, + { "uid": "abc123", "weapon": "rock" }, + { "uid": "xyz789", "weapon": "hand" }, + { "uid": "xyz789", "weapon": "foot" }, + { "uid": "xyz789", "weapon": "head" } +] +``` + +## Resources + +- [Supabase vs Firebase](/alternatives/supabase-vs-firebase) +- [Firestore Storage Migration](/kb/guides/platform/migrating-to-supabase/firebase-storage) +- [Firebase Auth Migration](/kb/guides/platform/migrating-to-supabase/firebase-auth) + +## Migrate to Supabase + +[Contact us](https://forms.supabase.com/firebase-migration) if you need more help migrating your project. diff --git a/apps/kb/src/content/guides/migrate-from-heroku.mdx b/apps/kb/src/content/guides/migrate-from-heroku.mdx new file mode 100644 index 00000000000..0c2d512471b --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-heroku.mdx @@ -0,0 +1,62 @@ +--- +title: 'Migrate from Heroku to Supabase' +description: 'Migrate your Heroku Postgres database to Supabase.' +topics: ['Migration', 'Database'] +--- + +Supabase is one of the best [free alternatives to Heroku Postgres](/alternatives/supabase-vs-heroku-postgres). This guide shows how to migrate your Heroku Postgres database to Supabase. This migration requires the [pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html) and [psql](https://www.postgresql.org/docs/current/app-psql.html) CLI tools, which are installed automatically as part of the complete Postgres installation package. + +Alternatively, use the [Heroku to Supabase migration tool](https://migrate.supabase.com/) to migrate in a few clicks. + +## Retrieve your Heroku database credentials + +1. Sign in to your [Heroku account](https://heroku.com) and select the project you want to migrate. +1. Click **Resources** in the menu and select your **Heroku Postgres** database. +1. Click **Settings** in the menu. +1. Click **View Credentials** and save the following information: + - Host (`$HEROKU_HOST`) + - Database (`$HEROKU_DATABASE`) + - User (`$HEROKU_USER`) + - Password (`$HEROKU_PASSWORD`) + +## Retrieve your Supabase connection string + +1. If you're new to Supabase, [create a project](/dashboard). +1. Get your project's Session pooler connection string from your project dashboard by clicking [Connect](/dashboard/project/_?showConnect=true&method=session). +1. Replace [YOUR-PASSWORD] in the connection string with your database password. You can reset your database password on the [Database Settings page](/dashboard/project/_/database/settings) if you do not have it. + +## Export your Heroku database to a file + +Use `pg_dump` with your Heroku credentials to export your Heroku database to a file (e.g., `heroku_dump.sql`). + +```bash +pg_dump --clean --if-exists --quote-all-identifiers \ + -h $HEROKU_HOST -U $HEROKU_USER -d $HEROKU_DATABASE \ + --no-owner --no-privileges > heroku_dump.sql +``` + +## Import the database to your Supabase project + +Use `psql` to import the Heroku database file to your Supabase project. + +```bash +psql -d "$YOUR_CONNECTION_STRING" -f heroku_dump.sql +``` + +## Additional options + +- To only migrate a single database schema, add the `--schema=PATTERN` parameter to your `pg_dump` command. +- To exclude a schema: `--exclude-schema=PATTERN`. +- To only migrate a single table: `--table=PATTERN`. +- To exclude a table: `--exclude-table=PATTERN`. + +Run `pg_dump --help` for a full list of options. + +> [!CAUTION] +> +> - If you're planning to migrate a database larger than 6 GB, we recommend [upgrading to at least a Large compute add-on](/docs/guides/platform/compute-and-disk). This will ensure you have the necessary resources to handle the migration efficiently. +> - We strongly advise you to pre-provision the disk space you will need for your migration. On paid projects, you can do this by navigating to the [Infrastructure settings](/dashboard/project/_/settings/infrastructure) page. For more information on disk scaling and disk limits, check out our [disk settings](/docs/guides/platform/compute-and-disk#disk) documentation. + +## Enterprise + +[Contact us](https://forms.supabase.com/enterprise) if you need more help migrating your project. diff --git a/apps/kb/src/content/guides/migrate-from-mssql.mdx b/apps/kb/src/content/guides/migrate-from-mssql.mdx new file mode 100644 index 00000000000..4681008682f --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-mssql.mdx @@ -0,0 +1,72 @@ +--- +title: 'Migrate from MSSQL to Supabase' +description: 'Migrate your Microsoft SQL Server database to Supabase.' +topics: ['Migration', 'Database'] +--- + +This guide aims to demonstrate the process of transferring your Microsoft SQL Server database to Supabase's Postgres database. Supabase is a powerful and open-source platform offering a wide range of backend features, including a Postgres database, authentication, instant APIs, edge functions, real-time subscriptions, and storage. Migrating your MSSQL database to Supabase's Postgres enables you to leverage Postgres's capabilities and access all the features you need for your project. + +## Retrieve your MSSQL database credentials + +Before you begin the migration, you need to collect essential information about your MSSQL database. Follow these steps: + +1. Sign in to your MSSQL database provider. +1. Locate and note the following database details: + - Hostname or IP address + - Database name + - Username + - Password + +## Retrieve your Supabase host + +1. If you're new to Supabase, [create a project](/dashboard). + Make a note of your password, you will need this later. If you forget it, you can [reset it here](/dashboard/project/_/database/settings). + +1. On your project dashboard, click [Connect](/dashboard/project/_?showConnect=true&method=session) +1. Under the Session pooler, click on the View parameters under the connect string. Note your Host (`$SUPABASE_HOST`). + +![Finding Supabase host address](../../assets/guides/migrate-from-mssql/database-settings-host.png) + +## Migrate the database + +The fastest way to migrate your database is with the Supabase migration tool on +[Google Colab](https://colab.research.google.com/github/mansueli/Supa-Migrate/blob/main/Amazon_RDS_to_Supabase.ipynb). + +Alternatively, you can use [pgloader](https://github.com/dimitri/pgloader), a flexible and powerful data migration tool that supports a wide range of source database engines, including MySQL and MS SQL, and migrates the data to a Postgres database. For databases using the Postgres engine, we recommend using the [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html) and [psql](https://www.postgresql.org/docs/current/app-psql.html) command line tools, which are included in a full Postgres installation. + +### Migrate using Colab + +1. Select the Database Engine from the Source database in the dropdown. +1. Set the environment variables (`HOST`, `USER`, `SOURCE_DB`,`PASSWORD`, `SUPABASE_URL`, and `SUPABASE_PASSWORD`) in the Colab notebook. +1. Run the first two steps in [the notebook](https://colab.research.google.com/github/mansueli/Supa-Migrate/blob/main/Amazon_RDS_to_Supabase.ipynb) in order. The first sets engine and installs the necessary files. +1. Run the third step to start the migration. This will take a few minutes. + +### Migrate from MSSQL + +1. Install pgloader. +2. Create a configuration file (e.g., config.load). + + For your destination, use your Supabase connection string with `Use connection pooling` enabled, and the mode set to `Session`. You can get the string from your [`Database Settings`](/dashboard/project/_/settings/general). + + ```sql + LOAD DATABASE + FROM mssql://USER:PASSWORD@HOST/SOURCE_DB + INTO postgres://postgres.xxxx:password@xxxx.pooler.supabase.com:5432/postgres + ALTER SCHEMA 'public' OWNER TO 'postgres'; + set wal_buffers = '64MB', max_wal_senders = 0, statement_timeout = 0, work_mem to '2GB'; + ``` + +3. Run the migration with pgloader + + ```bash + pgloader config.load + ``` + +> [!CAUTION] +> +> - If you're planning to migrate a database larger than 6 GB, we recommend [upgrading to at least a Large compute add-on](/docs/guides/platform/compute-and-disk). This will ensure you have the necessary resources to handle the migration efficiently. +> - We strongly advise you to pre-provision the disk space you will need for your migration. On paid projects, you can do this by navigating to the [Infrastructure settings](/dashboard/project/_/settings/infrastructure) page. For more information on disk scaling and disk limits, check out our [disk settings](/docs/guides/platform/compute-and-disk#disk) documentation. + +## Enterprise + +[Contact us](https://forms.supabase.com/enterprise) if you need more help migrating your project. diff --git a/apps/kb/src/content/guides/migrate-from-mysql.mdx b/apps/kb/src/content/guides/migrate-from-mysql.mdx new file mode 100644 index 00000000000..07b7245e869 --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-mysql.mdx @@ -0,0 +1,73 @@ +--- +title: 'Migrate from MySQL to Supabase' +description: 'Migrate your MySQL database to a Supabase Postgres database.' +topics: ['Migration', 'Database'] +--- + +This guide aims to exhibit the process of transferring your MySQL database to Supabase's Postgres database. Supabase is a robust and open-source platform offering a wide range of backend features, including a Postgres database, authentication, instant APIs, edge functions, real-time subscriptions, and storage. Migrating your MySQL database to Supabase's Postgres enables you to leverage Postgres's capabilities and access all the features you need for your project. + +## Retrieve your MySQL database credentials + +Before you begin the migration, you need to collect essential information about your MySQL database. Follow these steps: + +1. Sign in to your MySQL database provider. + +1. Locate and note the following database details: + - Hostname or IP address + - Database name + - Username + - Password + +## Retrieve your Supabase host + +1. If you're new to Supabase, [create a project](/dashboard). + Make a note of your password, you will need this later. If you forget it, you can [reset it here](/dashboard/project/_/database/settings). + +1. On your project dashboard, click [Connect](/dashboard/project/_?showConnect=true&method=session) +1. Under the Session pooler, click on the View parameters under the connect string. Note your Host (`$SUPABASE_HOST`). + +![Finding Supabase host address](../../assets/guides/migrate-from-mysql/database-settings-host.png) + +## Migrate the database + +The fastest way to migrate your database is with the Supabase migration tool on +[Google Colab](https://colab.research.google.com/github/mansueli/Supa-Migrate/blob/main/Amazon_RDS_to_Supabase.ipynb). + +Alternatively, you can use [pgloader](https://github.com/dimitri/pgloader), a flexible and powerful data migration tool that supports a wide range of source database engines, including MySQL and MS SQL, and migrates the data to a Postgres database. For databases using the Postgres engine, we recommend using the [`pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html) and [psql](https://www.postgresql.org/docs/current/app-psql.html) command line tools, which are included in a full Postgres installation. + +### Migrate using Colab + +1. Select the Database Engine from the Source database in the dropdown +1. Set the environment variables (`HOST`, `USER`, `SOURCE_DB`,`PASSWORD`, `SUPABASE_URL`, and `SUPABASE_PASSWORD`) in the Colab notebook. +1. Run the first two steps in [the notebook](https://colab.research.google.com/github/mansueli/Supa-Migrate/blob/main/Amazon_RDS_to_Supabase.ipynb) in order. The first sets engine and installs the necessary files. +1. Run the third step to start the migration. This will take a few minutes. + +### Migrate from MySQL with pgloader + +1. Install pgloader. +2. Create a configuration file (e.g., config.load). + + For your destination, use your Supabase connection string with `Use connection pooling` enabled, and the mode set to `Session`. You can get the string from your [`Database Settings`](/dashboard/project/_/settings/general). + + ```sql + load database + from mysql://user:password@host/source_db + into postgres://postgres.xxxx:password@xxxx.pooler.supabase.com:5432/postgres + alter schema 'public' owner to 'postgres'; + set wal_buffers = '64MB', max_wal_senders = 0, statement_timeout = 0, work_mem to '2GB'; + ``` + +3. Run the migration with pgloader + + ```bash + pgloader config.load + ``` + +> [!CAUTION] +> +> - If you're planning to migrate a database larger than 6 GB, we recommend [upgrading to at least a Large compute add-on](/docs/guides/platform/compute-and-disk). This will ensure you have the necessary resources to handle the migration efficiently. +> - We strongly advise you to pre-provision the disk space you will need for your migration. On paid projects, you can do this by navigating to the [Infrastructure settings](/dashboard/project/_/settings/infrastructure) page. For more information on disk scaling and disk limits, check out our [disk settings](/docs/guides/platform/compute-and-disk#disk) documentation. + +## Enterprise + +[Contact us](https://forms.supabase.com/enterprise) if you need more help migrating your project. diff --git a/apps/kb/src/content/guides/migrate-from-neon.mdx b/apps/kb/src/content/guides/migrate-from-neon.mdx new file mode 100644 index 00000000000..bb4f9ffb596 --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-neon.mdx @@ -0,0 +1,92 @@ +--- +title: 'Migrate from Neon to Supabase' +description: 'Migrate your existing Neon database to Supabase.' +topics: ['Migration', 'Database'] +--- + +This guide demonstrates how to migrate your Neon database to Supabase to get the most out of Postgres while gaining access to all the features you need to build a project. + +## Retrieve your Neon database credentials + +1. Sign in to your Neon Console [https://console.neon.tech/login](https://console.neon.tech/login). +1. Select **Projects** on the left. +1. Click on your project in the list. +1. From your Project Dashboard find your **Connection string** and click **Copy snippet** to copy it to the clipboard (do not check "pooled connection"). + +Example: + +```bash +postgresql://neondb_owner:xxxxxxxxxxxxxxx-random-word-yyyyyyyy.us-west-2.aws.neon.tech/neondb?sslmode=require +``` + +## Set your `OLD_DB_URL` environment variable + +Set the **OLD_DB_URL** environment variable at the command line using your Neon database credentials from the clipboard. + +Example: + +```bash +export OLD_DB_URL="postgresql://neondb_owner:xxxxxxxxxxxxxxx-random-word-yyyyyyyy.us-west-2.aws.neon.tech/neondb?sslmode=require" +``` + +## Retrieve your Supabase connection string + +1. If you're new to Supabase, [create a project](/dashboard). + Make a note of your password, you will need this later. If you forget it, you can [reset it here](/dashboard/project/_/database/settings). + +1. On your project dashboard, click [Connect](/dashboard/project/_?showConnect=true&method=session) +1. Under the Session pooler, click the **Copy** button to the right of your connection string to copy it to the clipboard. + +## Set your `NEW_DB_URL` environment variable + +Set the **NEW_DB_URL** environment variable at the command line using your Supabase connection string. You will need to replace `[YOUR-PASSWORD]` with your actual database password. + +Example: + +```bash +export NEW_DB_URL="postgresql://postgres.xxxxxxxxxxxxxxxxxxxx:[YOUR-PASSWORD]@aws-0-us-west-1.pooler.supabase.com:5432/postgres" +``` + +## Migrate the database + +You will need the [pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html) and [psql](https://www.postgresql.org/docs/current/app-psql.html) command line tools, which are included in a full [Postgres installation](https://www.postgresql.org/download). + +1. Export your database to a file in console + + Use `pg_dump` with your Postgres credentials to export your database to a file (e.g., `dump.sql`). + +```bash +pg_dump "$OLD_DB_URL" \ + --clean \ + --if-exists \ + --quote-all-identifiers \ + --no-owner \ + --no-privileges \ + > dump.sql +``` + +2. Import the database to your Supabase project + + Use `psql` to import the Postgres database file to your Supabase project. + + ```bash + psql -d "$NEW_DB_URL" -f dump.sql + ``` + +Additional options + +- To only migrate a single database schema, add the `--schema=PATTERN` parameter to your `pg_dump` command. +- To exclude a schema: `--exclude-schema=PATTERN`. +- To only migrate a single table: `--table=PATTERN`. +- To exclude a table: `--exclude-table=PATTERN`. + +Run `pg_dump --help` for a full list of options. + +> [!CAUTION] +> +> - If you're planning to migrate a database larger than 6 GB, we recommend [upgrading to at least a Large compute add-on](/docs/guides/platform/compute-and-disk). This will ensure you have the necessary resources to handle the migration efficiently. +> - We strongly advise you to pre-provision the disk space you will need for your migration. On paid projects, you can do this by navigating to the [Infrastructure settings](/dashboard/project/_/settings/infrastructure) page. For more information on disk scaling and disk limits, check out our [disk settings](/docs/guides/platform/compute-and-disk#disk) documentation. + +## Enterprise + +[Contact us](https://forms.supabase.com/enterprise) if you need more help migrating your project. diff --git a/apps/kb/src/content/guides/migrate-from-postgres.mdx b/apps/kb/src/content/guides/migrate-from-postgres.mdx new file mode 100644 index 00000000000..96b8a384477 --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-postgres.mdx @@ -0,0 +1,581 @@ +--- +title: 'Migrate from Postgres to Supabase' +description: 'Migrate your existing Postgres database to Supabase.' +topics: ['Migration', 'Database'] +--- + +This is a guide for migrating your Postgres database to [Supabase](https://supabase.com). +Supabase is a robust and open-source platform. Supabase provides all the backend features developers need to build a product: a Postgres database, authentication, instant APIs, edge functions, real-time subscriptions, and storage. Postgres is the core of Supabase—for example, you can use row-level security, and there are more than 40 Postgres extensions available. + +This guide demonstrates how to migrate your Postgres database to Supabase to get the most out of Postgres while gaining access to all the features you need to build a project. + +This guide provides three methods for migrating your Postgres database to Supabase: + +1. **Google Colab** - Guided notebook with copy-paste workflow +2. **Manual Dump/Restore** - CLI approach, works for all versions +3. **Logical Replication** - Minimal downtime, requires Postgres 10+ + +## Connection modes + +Supabase provides the following connection modes: + +- Direct connection +- Supavisor session mode +- Supavisor transaction mode + +Use Supavisor session mode for the database migration tasks (pg_dump/restore and logical replication). + +## Method 1: Google Colab (easiest) + +Supabase provides a Google Colab migration notebook for a guided migration experience: +[Supabase Migration Colab Notebook](https://colab.research.google.com/github/mansueli/Supa-Migrate/blob/main/Migrate_Postgres_Supabase.ipynb) + +This is ideal if you prefer a step-by-step, copy-paste workflow with minimal setup. + +## Method 2: Manual dump/restore + +This method works for all Postgres versions using CLI tools. + +### Prerequisites + +#### Source Postgres requirements + +- Connection string with rights to run `pg_dump` +- No special settings required for dump/restore +- Network access from migration VM + +#### Migration environment + +- Cloud VM running Ubuntu in the same region as source or target database +- Postgres client tools matching your source database version +- tmux for session persistence +- Sufficient disk space (usually ~50% of source database size is enough, but varies case by case) + +### Pre-Migration checklist + +```sql +-- Check database size +select pg_size_pretty(pg_database_size(current_database())) as size; + +-- Check Postgres version +select version(); + +-- List installed extensions +select * from pg_extension order by extname; + +-- Check active connections +select count(*) from pg_stat_activity; +``` + +#### Check available extensions in Supabase + +```sql +-- Connect to your Supabase database and check available extensions +SELECT name, comment FROM pg_available_extensions ORDER BY name; + +-- Compare with source database extensions +SELECT extname FROM pg_extension ORDER BY extname; + +-- Install needed extensions +CREATE EXTENSION IF NOT EXISTS extension_name; +``` + +### Step 1: Set up migration VM + +> [!NOTE] +> For optimal performance, run the migration from a cloud VM, not your local machine. The VM should be in the same region as either your source or target database to optimize network performance. See the Resource Requirements table in Step 2 for VM sizing recommendations. + +#### Set up Ubuntu VM + +```bash +# Install Postgres client and tools +sudo apt update +sudo apt install software-properties-common +sudo sh -c 'echo "deb http://apt.Postgres.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' +wget --quiet -O - https://www.Postgres.org/media/keys/ACCC4CF8.asc | sudo apt-key add - +sudo apt update +sudo apt install Postgres-client-17 tmux htop iotop moreutils + +# Start or attach to tmux session +tmux a -t migration || tmux new -s migration +``` + +### Step 2: Prepare Supabase project + +1. Create a Supabase project at [supabase.com/dashboard](/dashboard) +2. Note your database password +3. Install required extensions via SQL or Dashboard +4. Get your connection string: + - Go to **Project → Settings → Database → Connection Pooling** + - Select **Session pooler** (port 5432) and copy the connection string + - Connection format: `Postgres://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:5432/postgres` + +**Important Notes**: + +- **Users/roles are not migrated** - You'll need to recreate roles and privileges after import ([Supabase Roles Guide](/blog/postgres-roles-and-privileges)) +- **Row Level Security (RLS) status on tables is not migrated** - You'll need to enable RLS for tables after migration. + +**Resource Requirements**: +| Database Size | Recommended Compute | Recommended VM | Action Required | +|--------------|-------------------|----------------|-----------------| +| < 10 GB | Default | 2 vCPUs, 4 GB RAM | None | +| 10-100 GB | Default-Small | 4 vCPUs, 8 GB RAM | Consider compute upgrade | +| 100-500 GB | Large compute | 8 vCPUs, 16 GB RAM, NVMe | Upgrade compute before restore | +| 500 GB - 1 TB | XL compute | 16 vCPUs, 32 GB RAM, NVMe | Upgrade compute before restore | +| > 1 TB | Custom | Custom | [Contact support](/dashboard/support/new) first | + +Also, you can temporarily increase compute size and/or disk IOPS and throughput via Settings → Compute and Disk if you want faster database restore (you can use larger -j for pg_restore if you do so). + +### Step 3: Create database dump + +#### Set source database to read only mode for production migration + +If doing a maintenance window migration, prevent data changes: + +```sql +-- Connect to source database and run: +ALTER DATABASE your_database_name SET default_transaction_read_only = true; +``` + +For testing without a maintenance window, skip this step but use lower -j values. + +#### Dump the database + +```bash +# Determine number of parallel jobs based on: +# - Source database CPU cores (don't saturate production) +# - VM CPU cores +# - For testing without maintenance window: use lower values to be gentle +# - For production with maintenance window: can use higher values + +DUMP_JOBS=4 # Adjust based on your setup + +# Check available cores on VM +nproc + +# Create dump with progress logging +pg_dump \ + --host= \ + --port= \ + --username= \ + --dbname= \ + --jobs=$DUMP_JOBS \ + --format=directory \ + --no-owner \ + --no-privileges \ + --no-subscriptions \ + --verbose \ + --file=./db_dump 2>&1 | ts | tee -a dump.log +``` + +**Notes about dump flags**: + +- `--no-owner --no-privileges`: Applied at dump time to prevent Supabase user management conflicts. While these could be used in pg_restore instead, applying them during dump keeps the dump file cleaner and more portable. +- `--no-subscriptions`: Logical replication subscriptions won't work in the target +- The dump captures all data and schema but excludes ownership/privileges that would conflict with Supabase's managed environment +- To only migrate a single database schema, add the `--schema=PATTERN` parameter to your `pg_dump` command. +- To exclude a schema: `--exclude-schema=PATTERN`. +- To only migrate a single table: `--table=PATTERN`. +- To exclude a table: `--exclude-table=PATTERN`. + +Run `pg_dump --help` for a full list of options. + +#### Recommended parallelization (-j values) + +| Database Size | Testing (no maintenance window) | Production (with maintenance window) | Limiting Factor | +| ------------- | ------------------------------- | ------------------------------------ | --------------- | +| < 10 GB | 2 | 4 | Source CPU | +| 10-100 GB | 2-4 | 8 | Source CPU | +| 100-500 GB | 4 | 16 | Disk IOPS | +| 500 GB - 1 TB | 4-8 | 16-32 | Disk IOPS + CPU | + +**Note**: For testing without a maintenance window, use lower -j values to avoid impacting production performance. + +### Step 4: Restore to Supabase + +#### Set connection and restore + +```bash +# Set Supabase connection (Session Pooler on port 5432 or direct connection) +export SUPABASE_DB_URL="Postgres://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:5432/postgres" + +# Determine restore parallelization based on your Supabase compute size: +# Free tier: 2 cores → use -j 2 +# Small compute: 2 cores → use -j 2 +# Medium compute: 4 cores → use -j 4 +# Large compute: 8 cores → use -j 8 +# XL compute: 16 cores → use -j 16 + +RESTORE_JOBS=8 # Adjust based on your Supabase compute size + +# Restore the dump (parallel mode) +# Note: -j cannot be used with --single-transaction +pg_restore \ + --dbname="$SUPABASE_DB_URL" \ + --jobs=$RESTORE_JOBS \ + --format=directory \ + --no-owner \ + --no-privileges \ + --verbose \ + ./db_dump 2>&1 | ts | tee -a restore.log +``` + +If restore fails with extension errors, check that errors are only extension-related. + +### Step 5: Post-Migration tasks + +#### Update statistics (important) + +```bash +psql "$SUPABASE_DB_URL" -c "VACUUM VERBOSE ANALYZE;" +``` + +> [!NOTE] +> For Postgres 18+, pg_dump includes statistics with `--with-statistics`, but you should still run VACUUM for optimal performance. + +#### Verify migration + +```sql +-- Check row counts +select schemaname, tablename, n_live_tup +from pg_stat_user_tables +order by n_live_tup desc +limit 20; +-- Verify data with application-specific queries +``` + +#### Re-enable writes on source (if keeping it) + +```sql +ALTER DATABASE your_database_name SET default_transaction_read_only = false; +``` + +### Migration time estimates + +| Database Size | Dump Time | Restore Time | Total Time | +| ------------- | --------- | ------------ | ---------- | +| 10 GB | ~5 min | ~10 min | ~15 min | +| 100 GB | ~30 min | ~45 min | ~1.5 hours | +| 500 GB | ~2 hours | ~3 hours | ~5 hours | +| 1 TB | ~4 hours | ~6 hours | ~10 hours | + +_Times vary based on hardware, network, and parallelization settings_ + +### Important notes + +1. **Region proximity matters**: VM should be in the same region as the source or target for best performance +2. **Downgrade migrations**: While technically possible in some cases, highly not recommended +3. **Testing without downtime**: Use lower `-j` values for pg_dump to avoid impacting production +4. **For pg_restore**: Can use full parallelization regardless of production impact +5. **Monitor resources**: Watch CPU, disk I/O with `htop`, `iotop` +6. **Disk I/O**: Often the bottleneck before network bandwidth + +--- + +## Method 3: Logical replication + +This method allows migration with minimal downtime using Postgres's logical replication feature. Requires Postgres 10+ on both source and target. + +### When to use logical replication + +- You need minimal downtime (minutes instead of hours) +- Source database is Postgres 10 or higher +- You can configure logical replication on the source +- Database has high write activity that can't be paused for long + +### Source Postgres prerequisites + +#### Access & privileges + +- Connection string with rights to CREATE PUBLICATION and read tables +- Superuser or replication privileges recommended + +#### Required settings for logical replication + +- `wal_level = logical` +- `max_wal_senders ≥ 1` +- `max_replication_slots ≥ 1` +- Sufficient `max_connections` (current + 1 for subscription) + +#### Replica identity + +Every table receiving UPDATE/DELETE must have a replica identity (typically a PRIMARY KEY). For tables without one: + +```sql +ALTER TABLE schema.table_name REPLICA IDENTITY FULL; +``` + +#### Non-Replicated items + +- **DDL changes** (schema modifications) +- **Sequences** (need manual sync) +- **Large Objects (LOBs)** (use dump/restore or store in regular bytea columns) + +Plan a schema freeze, sequence sync before cutover, and handle LOBs separately. + +### Step 1: Configure source database + +Edit Postgres configuration files: + +#### Postgres.conf + +```bash +# Set Supabase connection (Session Pooler on port 5432 or direct connection) +export SUPABASE_DB_URL="Postgres://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:5432/postgres" + +# Set WAL level to logical +wal_level = logical + +# Ensure sufficient replication slots +max_replication_slots = 10 + +# Ensure sufficient WAL senders +max_wal_senders = 10 + +# Set appropriate max_connections (current connections + 1 for subscription) +max_connections = 200 # Adjust based on your needs + +# Optional: Enable SSL for secure replication +ssl = on + +# Allow connections from Supabase +listen_addresses = '*' # Or specific IP addresses +``` + +#### pg_hba.conf + +```bash +# Allow replication connections from Supabase +# Replace with actual Supabase IP range +host replication all md5 +host all all md5 + +# With SSL: +hostssl replication all md5 +hostssl all all md5 +``` + +Restart Postgres: + +```bash +sudo systemctl restart Postgres +sudo systemctl status Postgres +``` + +### Step 2: Verify configuration + +```sql +-- Should return 'logical' +SHOW wal_level; + +-- Check other parameters +SHOW max_replication_slots; +SHOW max_wal_senders; + +-- Check current connections +SELECT count(*) FROM pg_stat_activity; +``` + +### Step 3: Check and set replica identity + +```sql +-- Find tables without primary keys +SELECT n.nspname, c.relname +FROM pg_class c +JOIN pg_namespace n ON n.oid = c.relnamespace +LEFT JOIN pg_constraint pk ON pk.conrelid = c.oid AND pk.contype = 'p' +WHERE c.relkind = 'r' + AND pk.oid IS NULL + AND n.nspname NOT IN ('pg_catalog','information_schema'); + +-- For tables without a primary key, set REPLICA IDENTITY FULL +ALTER TABLE my_schema.my_table REPLICA IDENTITY FULL; +``` + +### Step 4: Export and restore schema only + +```bash +# Export schema from source +pg_dump \ + -h \ + -U \ + -p \ + -d \ + --schema-only \ + --no-privileges \ + --no-subscriptions \ + --format=directory \ + -f ./schema_dump + +# Restore schema to Supabase (use Session Pooler) +pg_restore \ + --dbname="$SUPABASE_DB_URL" \ + --format=directory \ + --schema-only \ + --no-privileges \ + --single-transaction \ + --verbose \ + ./schema_dump +``` + +### Step 5: Create publication on source + +```sql +-- Create publication for all tables +CREATE PUBLICATION supabase_migration FOR ALL TABLES; + +-- Or for specific tables only (doesn't require superuser) +CREATE PUBLICATION supabase_migration FOR TABLE + schema1.table1, + schema1.table2, + public.table3; + +-- Verify publication was created +SELECT * FROM pg_publication; +``` + +### Step 6: Create subscription on Supabase + +Connect to your Supabase database: + +```sql +-- Create subscription with SSL (recommended) +CREATE SUBSCRIPTION supabase_subscription +CONNECTION 'host= port= user= password= dbname= sslmode=require' +PUBLICATION supabase_migration; + +-- Or without SSL (if source doesn't support it) +CREATE SUBSCRIPTION supabase_subscription +CONNECTION 'host= port= user= password= dbname= sslmode=disable' +PUBLICATION supabase_migration; +``` + +### Step 7: Monitor replication status + +```sql +-- On Supabase (subscriber) - check subscription status +select * from pg_subscription_rel; + +-- srsubstate = 'r' means ready (synchronized) +-- srsubstate = 'i' means initializing +-- srsubstate = 'd' means data is being copied + +-- Overall subscription status +select * from pg_stat_subscription; + +-- On source database - check replication status +select * from pg_stat_replication; + +-- Check replication lag +select + slot_name, + pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) as lag_size +from pg_replication_slots; +``` + +Wait until all tables show `srsubstate = 'r'` (ready) status. + +### Step 8: Synchronize sequences + +After initial data sync is complete, but BEFORE switching to Supabase: + +```bash +# Set source to read-only +psql -h -c "ALTER DATABASE SET default_transaction_read_only = true;" + +# Export sequences from source +pg_dump \ + -h \ + -U \ + -p \ + -d \ + --data-only \ + --table='*_seq' \ + --table='*_id_seq' > sequences.sql + +# Import sequences to Supabase +psql "$SUPABASE_DB_URL" -f sequences.sql +``` + +### Step 9: Switch to Supabase + +1. Ensure replication lag is zero: + +```sql +-- On Supabase +select * from pg_stat_subscription; +-- Check that latest_end_lsn is current +``` + +2. Stop writes to the source database (if not already read-only) + +3. Drop subscription on Supabase: + +```sql +DROP SUBSCRIPTION supabase_subscription; +``` + +4. Update application connection strings to point to Supabase + +5. Verify application functionality + +### Step 10: Cleanup + +On source database (after successful migration): + +```sql +-- Remove publication +DROP PUBLICATION supabase_migration; + +-- Check and remove any remaining replication slots +SELECT * FROM pg_replication_slots; +DROP REPLICATION SLOT slot_name; -- if any remain + +-- The source database should remain read-only or be decommissioned +-- Do NOT re-enable writes to avoid a split-brain scenario! +``` + +### Troubleshooting logical replication + +| Issue | Solution | +| ------------------------------------ | ------------------------------------------------------------------- | +| "could not connect to the publisher" | Check network connectivity, firewall rules, pg_hba.conf | +| "role does not exist" | Ensure replication user exists on source with REPLICATION privilege | +| "publication does not exist" | Verify publication name and that it was created successfully | +| Replication lag growing | Check network bandwidth, source database load, add more WAL senders | +| Tables stuck in `i` state | Check for locks on source tables, verify table structure matches | +| "out of replication slots" | Increase max_replication_slots in Postgres.conf | + +### Important limitations + +- **DDL changes**: Schema modifications are not replicated - freeze schema during migration +- **Sequences**: Need manual synchronization before cutover +- **Large Objects (LOBs)**: Not replicated - use dump/restore or store in regular bytea columns +- **Custom types**: May need special handling +- **Users and roles**: Must be recreated manually on Supabase + +For detailed restrictions, see [Postgres Logical Replication Restrictions](https://www.Postgres.org/docs/current/logical-replication-restrictions.html) + +### When to use which method + +**Use Dump/Restore when:** + +- Downtime window is acceptable +- Source is Postgres < 10 +- Simpler process preferred +- Cannot configure logical replication on the source + +**Use Logical Replication when:** + +- Minimal downtime required +- Postgres 10+ on both sides +- Can modify source configuration +- Have replication privileges + +## Getting help + +- For databases > 150 GB: [Contact Supabase support](/dashboard/support/new) before starting +- [Supabase Dashboard Support](/dashboard/support/new) +- [Supabase Discord](https://discord.supabase.com) +- [Postgres Roles and Privileges Guide](/blog/postgres-roles-and-privileges) +- [Row Level Security Guide](/docs/guides/database/postgres/row-level-security) diff --git a/apps/kb/src/content/guides/migrate-from-render.mdx b/apps/kb/src/content/guides/migrate-from-render.mdx new file mode 100644 index 00000000000..cb7ce7dfd06 --- /dev/null +++ b/apps/kb/src/content/guides/migrate-from-render.mdx @@ -0,0 +1,84 @@ +--- +title: 'Migrate from Render to Supabase' +description: 'Migrate your Render Postgres database to Supabase.' +topics: ['Migration', 'Database'] +--- + +Render is a popular Web Hosting service in the online services category that also has a managed Postgres service. Render has a great developer experience, allowing users to deploy straight from GitHub or GitLab. This is the core of their product and they do it really well. However, when it comes to Postgres databases, it may not be the best option. + +Supabase is one of the best free alternative to Render Postgres. Supabase provide all the backend features developers need to build a product: a Postgres database, authentication, instant APIs, edge functions, realtime subscriptions, and storage. Postgres is the core of Supabase—for example, you can use row-level security and there are more than 40 Postgres extensions available. + +This guide demonstrates how to migrate from Render to Supabase to get the most out of Postgres while gaining access to all the features you need to build a project. + +## Retrieve your Render database credentials + +1. Sign in to your [Render account](https://render.com) and select the project you want to migrate. +1. Click **Dashboard** in the menu and click in your **Postgres** database. +1. Scroll down in the **Info** tab. +1. Click on **PSQL Command** and edit it adding the content after `PSQL_COMMAND=`. + +![Copying PSQL command from Render dashboard](../../assets/guides/migrate-from-render/render_dashboard.png) +Example: + +```bash +%env PSQL_COMMAND=PGPASSWORD=RgaMDfTS_password_FTPa7 psql -h dpg-a_server_in.oregon-postgres.render.com -U my_db_pxl0_user my_db_pxl0 +``` + +## Retrieve your Supabase connection string + +1. If you're new to Supabase, [create a project](/dashboard). + Make a note of your password, you will need this later. If you forget it, you can [reset it here](/dashboard/project/_/database/settings). + +1. On your project dashboard, click [Connect](/dashboard/project/_?showConnect=true&method=session) +1. Under Session pooler, Copy the connection string and replace the password placeholder with your database password. + + > [!NOTE] + > If you're in an [IPv6 environment](https://github.com/orgs/supabase/discussions/27034) or have the IPv4 Add-On, you can use the direct connection string instead of Supavisor in Session mode. + +## Migrate the database + +The fastest way to migrate your database is with the Supabase migration tool on [Google Colab](https://colab.research.google.com/github/mansueli/Supa-Migrate/blob/main/Migrate_Postgres_Supabase.ipynb). Alternatively, you can use the [pg_dump](https://www.postgresql.org/docs/current/app-pgdump.html) and [psql](https://www.postgresql.org/docs/current/app-psql.html) command line tools, which are included in a full Postgres installation. + +### Migrate using Colab + +1. Set the environment variables (`PSQL_COMMAND`, `SUPABASE_HOST`, `SUPABASE_PASSWORD`) in the Colab notebook. +1. Run the first two steps in [the notebook](https://colab.research.google.com/github/mansueli/Supa-Migrate/blob/main/Migrate_Postgres_Supabase.ipynb) in order. The first sets the variables and the second installs PSQL and the migration script. +1. Run the third step to start the migration. This will take a few minutes. + +### Migrate using CLI tools + +1. Export your Render database to a file in console + + Use `pg_dump` with your Render credentials to export your Render database to a file (e.g., `render_dump.sql`). + + ```bash + pg_dump --clean --if-exists --quote-all-identifiers \ + -h $RENDER_HOST -U $RENDER_USER -d $RENDER_DATABASE \ + --no-owner --no-privileges > render_dump.sql + ``` + +2. Import the database to your Supabase project + + Use `psql` to import the Render database file to your Supabase project. + + ```bash + psql -d "$YOUR_CONNECTION_STRING" -f render_dump.sql + ``` + +**Additional options** + +- To only migrate a single database schema, add the `--schema=PATTERN` parameter to your `pg_dump` command. +- To exclude a schema: `--exclude-schema=PATTERN`. +- To only migrate a single table: `--table=PATTERN`. +- To exclude a table: `--exclude-table=PATTERN`. + +Run `pg_dump --help` for a full list of options. + +> [!CAUTION] +> +> - If you're planning to migrate a database larger than 6 GB, we recommend [upgrading to at least a Large compute add-on](/docs/guides/platform/compute-and-disk). This will ensure you have the necessary resources to handle the migration efficiently. +> - We strongly advise you to pre-provision the disk space you will need for your migration. On paid projects, you can do this by navigating to the [Infrastructure settings](/dashboard/project/_/settings/infrastructure) page. For more information on disk scaling and disk limits, check out our [disk settings](/docs/guides/platform/compute-and-disk#disk) documentation. + +## Enterprise + +[Contact us](https://forms.supabase.com/enterprise) if you need more help migrating your project. diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 3001fc3cabb..b97bd3f4c52 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -756,6 +756,9 @@ importers: remark-stringify: specifier: ^11.0.0 version: 11.0.0 + sharp: + specifier: ^0.35.4 + version: 0.35.4(@types/node@22.13.14) ui: specifier: workspace:* version: link:../../packages/ui @@ -35809,7 +35812,6 @@ snapshots: '@img/sharp-win32-ia32': 0.35.4 '@img/sharp-win32-x64': 0.35.4 '@types/node': 22.13.14 - optional: true shebang-command@1.2.0: dependencies: