diff --git a/apps/www/_blog/2022-06-28-partial-postgresql-data-dumps-with-rls.mdx b/apps/www/_blog/2022-06-28-partial-postgresql-data-dumps-with-rls.mdx new file mode 100644 index 00000000000..f62e9b2cb0b --- /dev/null +++ b/apps/www/_blog/2022-06-28-partial-postgresql-data-dumps-with-rls.mdx @@ -0,0 +1,180 @@ +--- +title: Partial data dumps using Postgres Row Level Security +description: Using RLS to create seed files for local PostgreSQL testing. +author: paul_copplestone +image: partial-dumps/og-partial-dumps-with-rls.png +thumb: partial-dumps/og-partial-dumps-with-rls.png +tags: + - postgresql + - data +date: '2022-06-28' +toc_depth: 3 +--- + +When working with databases, it's common to create a `seed.sql` file which contains a subset of production data for testing. + +During early development, it's fine to dump the entire database and restore it on your development machine. +However, once you have production users this becomes a security issue - do you really want to dump your users' data onto your local machines? + +There are many ways to solve this, but recently I stumbled upon a neat way to do it using PostgreSQL's Row Level Security (RLS). + +The concept is simple: + +1. Create a database user with restricted access. +2. Define some RLS rules for that user, limiting what data they can access. +3. Run `pg_dump` as that user. + +For this scenario, let's imagine that you have a table called `profiles` in your database: + +```sql hideCopy +create table profiles ( + id serial primary key, + name text, + email text +); +``` + +| `id` | `name` | `email` | +| ---- | ------------ | ------------------------ | +| `1` | `Employee 1` | `employee1@supabase.com` | +| `2` | `Employee 2` | `employee2@supabase.com` | +| `3` | `Employee 3` | `employee3@supabase.com` | +| `4` | `Jenny` | `jenny@example.com` | +| `5` | `Joe` | `joe@example.com` | + +In this case, if we ran a `pg_dump` we will save Jenny and Joe's personal data. We don't want that, so let's create a Postgres user called `exporter`, who can only dump the data we want. + +### Step 1: Prepare a user + +Create a user to connect to the database. We'll call them `exporter` and grant them access to the public schema: + +```sql hideCopy +-- Create a new user with login privileges +create user exporter + with password 'exporter_secure_password'; + +-- Allow this user to select the rows we need +grant usage on schema public to exporter; +grant select on profiles to exporter; +``` + +### Step 2: Create data access rules + +Let's turn on RLS for this table and limit the data which `exporter` can access: + +```sql hideCopy hideLineNumbers +-- Turn on Row Level Security +alter table profiles + enable row level security; + +-- Only dump data for internal team members 1, 2, 3 +create policy "Data dump rule" on profiles + for select + to exporter + using ( + id in (1, 2, 3) + ); +``` + +### Step 3: Export the data + +Now we can use `pg_dump` to get only the data that we need. + +Run the dump with the `exporter` user that we created above and use the `--enable-row-security` flag to ensure that the dump succeeds. + +```bash hideCopy hideLineNumbers +# Dump all the data into a "seed.sql" file +# which we can use to restore our local databases. +pg_dump \ +-h db.host.supabase.co \ +-U exporter \ +-d postgres \ +-n public \ +--data-only \ +--enable-row-security \ +--table=profiles \ +> seed.sql +``` + +```bash hideCopy hideLineNumbers +-h db.host.supabase.co \ +``` + +And that's it. You can follow this same pattern for any tables that you want to dump. + +## Data access patterns + +RLS is a bit like appending a “where” clause to a `select`, so you can create all sorts of data access patterns. Let's see a few more which are useful for extracting seed data. + +### Using email rules + +Instead of using hardcoded numbers in our RLS policies, we could use email extensions to determine the users who we want to export: + +```sql hideCopy +-- Only dump data for supabase employees +create policy "Data dump rule" on profiles + for select + to exporter + using ( + substring(email from '@(.*)$') = 'supabase.com' + ); + +``` + +### Only recent data + +If we have a table with a lot of data, like an `analytics` table, we might only care about the last 2 months of data. + +```sql hideCopy +-- A fake analytics table where we store actions a user takes +create table analytics ( + id serial primary key, + ts timestamptz default now(), + profile_id references profiles, + event text +); +alter table profiles + enable row level security; + +-- Here is an "age" rule so that we only dump the most recent analytics +create policy "Data dump rule" on logs + for select + to exporter + using ( + profile_id in (1, 2, 3) and + ts > now() - interval '2 MONTHS' -- here's the magic + ); +``` + +### Using flags + +If you don't mind having some additional columns in you database, you can add flags to each row to determine whether it's safe to export. + +```sql hideCopy +create table profiles ( + id serial primary key, + name text, + email text, + is_exportable boolean -- make this "TRUE" if you want to allow access +); +alter table profiles + enable row level security; + +-- Only dump data for internal team members 1, 2, 3 +create policy "Data dump rule" on profiles + for select + to exporter + using ( is_exportable = true ); + +``` + +## Conclusion + +Using `seed` data isn't the only way to run development environments. It's also possible to run fully-masked copies of your database using tools like +[Snaplet](https://docs.snaplet.dev/tutorials/supabase-clone-environments). + +We're also bullish on copy-on-write strategies which allow users to "fork" a database at a point in time, +a strategy used by [Database Lab Engine](https://postgres.ai/docs/database-lab). +DLE uses the ZFS file system to achieve this, but it's within reach of the Postgres core once alternative storage strategies become easier to implement. + +If you want to try out the steps we described in this article, fire up a full PostgreSQL database: [database.new](https://database.new) diff --git a/apps/www/components/CodeBlock/CodeBlock.tsx b/apps/www/components/CodeBlock/CodeBlock.tsx index 25de64199bd..87960fa1191 100644 --- a/apps/www/components/CodeBlock/CodeBlock.tsx +++ b/apps/www/components/CodeBlock/CodeBlock.tsx @@ -1,39 +1,44 @@ -import { useCallback, useMemo } from 'react' -import reactSyntaxHighlighter, { Light as SyntaxHighlighter } from 'react-syntax-highlighter' -import monokaiCustomTheme from 'data/CodeEditorTheme' -import CodeBlockStyles from './CodeBlock.module.css' -import { Button, IconCopy } from '@supabase/ui' +import { Button, IconCheck, IconCopy, IconFile, IconTerminal } from '@supabase/ui' +import { useState } from 'react' import CopyToClipboard from 'react-copy-to-clipboard' -import rangeParser from 'parse-numeric-range' - -import classNames from 'classnames' - +import { Light as SyntaxHighlighter } from 'react-syntax-highlighter' +import bash from 'react-syntax-highlighter/dist/cjs/languages/hljs/bash' import js from 'react-syntax-highlighter/dist/cjs/languages/hljs/javascript' -import ts from 'react-syntax-highlighter/dist/cjs/languages/hljs/typescript' import py from 'react-syntax-highlighter/dist/cjs/languages/hljs/python' import sql from 'react-syntax-highlighter/dist/cjs/languages/hljs/sql' +import monokaiCustomTheme from './CodeBlock.utils' -export interface CodeBlockProps { - lang: 'js' | 'ts' | 'sql' | 'py' +interface Props { + lang: 'js' | 'sql' | 'py' | 'bash' startingLineNumber?: number hideCopy?: boolean + showLineNumbers?: boolean className?: string children?: string size?: 'small' | 'medium' | 'large' - /** - * Inline styling - * Supports CSS Properties in camelcase - */ - style?: React.CSSProperties | undefined - /** - * Lines to be highlighted. - * Supports individual lines: '14', multiple lines: '14,15', or a range of lines '14..19' - */ - highlightLines?: string - hideBorder?: boolean } -function CodeBlock(props: CodeBlockProps) { +function CodeBlock(props: Props) { + const [copied, setCopied] = useState(false) + + const firstLine = props.children ? props.children.split('\n')[0] : '' + + let filename = '' + + if (firstLine.includes('filename =')) { + filename = firstLine.split('=')[1] + } + + const content = + props.children && filename ? props.children.replace(`${firstLine}\n\n`, '') : props.children + + const handleCopy = () => { + setCopied(true) + setTimeout(() => { + setCopied(false) + }, 1000) + } + let lang = props.lang ? props.lang : props.className @@ -41,87 +46,90 @@ function CodeBlock(props: CodeBlockProps) { : 'js' // force jsx to be js highlighted if (lang === 'jsx') lang = 'js' - if (lang === 'tsx') lang = 'ts' SyntaxHighlighter.registerLanguage('js', js) - SyntaxHighlighter.registerLanguage('ts', ts) SyntaxHighlighter.registerLanguage('py', py) SyntaxHighlighter.registerLanguage('sql', sql) + SyntaxHighlighter.registerLanguage('bash', bash) // const large = props.size === 'large' ? true : false const large = false - const shouldHighlightLines = props.highlightLines !== undefined - const highlightLines = useMemo( - () => new Set(rangeParser(props.highlightLines ?? '')), - [props.highlightLines] - ) - - const lineProps = useCallback( - (lineNumber: number) => { - const shouldHighlightLine = !shouldHighlightLines || highlightLines.has(lineNumber) - - const style = shouldHighlightLine ? {} : { filter: 'grayscale(75%)', opacity: 0.5 } - - return { - className: classNames( - CodeBlockStyles['code-line'], - shouldHighlightLines && shouldHighlightLine && CodeBlockStyles['code-line--flash'] - ), - style, - } - }, - [highlightLines] - ) - return ( -
- + {filename && ( +
- - - + border-r + border-l + px-4 + font-sans + " + > + {lang === 'bash' ? ( + + ) : ( + + )} + {filename ?? 'index.js'}
- ) : null} + )} +
+ + {content} + + {!props.hideCopy && props.children ? ( +
+ + + +
+ ) : null} +
) } diff --git a/apps/www/components/CodeBlock/CodeBlock.utils.js b/apps/www/components/CodeBlock/CodeBlock.utils.js new file mode 100644 index 00000000000..4233bf1548a --- /dev/null +++ b/apps/www/components/CodeBlock/CodeBlock.utils.js @@ -0,0 +1,116 @@ +const monokaiCustomTheme = { + hljs: { + display: 'block', + overflowX: 'auto', + padding: '0.5em', + background: '#272822', + color: '#ddd', + }, + 'hljs-tag': { + color: '#569cd6', + }, + 'hljs-keyword': { + color: '#569cd6', + fontWeight: 'normal', + }, + 'hljs-selector-tag': { + color: '#569cd6', + fontWeight: 'normal', + }, + 'hljs-literal': { + color: '#569cd6', + fontWeight: 'normal', + }, + 'hljs-strong': { + color: '#569cd6', + }, + 'hljs-name': { + color: '#569cd6', + }, + 'hljs-code': { + color: '#66d9ef', + }, + 'hljs-class .hljs-title': { + color: 'gray', + }, + 'hljs-attribute': { + color: '#bf79db', + }, + 'hljs-symbol': { + color: '#bf79db', + }, + 'hljs-regexp': { + color: '#bf79db', + }, + 'hljs-link': { + color: '#bf79db', + }, + 'hljs-string': { + color: '#3ECF8E', + }, + 'hljs-bullet': { + color: '#3ECF8E', + }, + 'hljs-subst': { + color: '#3ECF8E', + }, + 'hljs-title': { + color: '#3ECF8E', + fontWeight: 'normal', + }, + 'hljs-section': { + color: '#3ECF8E', + fontWeight: 'normal', + }, + 'hljs-emphasis': { + color: '#3ECF8E', + }, + 'hljs-type': { + color: '#3ECF8E', + fontWeight: 'normal', + }, + 'hljs-built_in': { + color: '#3ECF8E', + }, + 'hljs-builtin-name': { + color: '#3ECF8E', + }, + 'hljs-selector-attr': { + color: '#3ECF8E', + }, + 'hljs-selector-pseudo': { + color: '#3ECF8E', + }, + 'hljs-addition': { + color: '#3ECF8E', + }, + 'hljs-variable': { + color: '#3ECF8E', + }, + 'hljs-template-tag': { + color: '#3ECF8E', + }, + 'hljs-template-variable': { + color: '#3ECF8E', + }, + 'hljs-comment': { + color: '#75715e', + }, + 'hljs-quote': { + color: '#75715e', + }, + 'hljs-deletion': { + color: '#75715e', + }, + 'hljs-meta': { + color: '#75715e', + }, + 'hljs-doctag': { + fontWeight: 'normal', + }, + 'hljs-selector-id': { + fontWeight: 'normal', + }, +} + +export default monokaiCustomTheme diff --git a/apps/www/pages/blog/[year]/[month]/[day]/[slug].tsx b/apps/www/pages/blog/[year]/[month]/[day]/[slug].tsx index 6ba7f9a0099..a2f5bfeaa06 100644 --- a/apps/www/pages/blog/[year]/[month]/[day]/[slug].tsx +++ b/apps/www/pages/blog/[year]/[month]/[day]/[slug].tsx @@ -30,26 +30,16 @@ const components = { }, ImageGrid, img: (props: any) => { - const classes = [ - 'next-image--dynamic-fill', - 'from-brand-500 to-brand-500', - 'rounded border bg-gradient-to-r via-blue-500', - ] - return (
- +
) }, diff --git a/apps/www/public/images/blog/partial-dumps/og-partial-dumps-with-rls.png b/apps/www/public/images/blog/partial-dumps/og-partial-dumps-with-rls.png new file mode 100644 index 00000000000..baabae93abe Binary files /dev/null and b/apps/www/public/images/blog/partial-dumps/og-partial-dumps-with-rls.png differ