mirror of
https://github.com/supabase/supabase.git
synced 2026-10-10 03:45:06 +03:00
Merge pull request #7478 from supabase/blog/partial_dumps
Adds a new blog post for partial dumps
This commit is contained in:
5 files changed
+403
-109
No files matched your search
@@ -0,0 +1,180 @@
|
||||
---
|
||||
title: Partial data dumps using Postgres Row Level Security
|
||||
description: Using RLS to create seed files for local PostgreSQL testing.
|
||||
author: paul_copplestone
|
||||
image: partial-dumps/og-partial-dumps-with-rls.png
|
||||
thumb: partial-dumps/og-partial-dumps-with-rls.png
|
||||
tags:
|
||||
- postgresql
|
||||
- data
|
||||
date: '2022-06-28'
|
||||
toc_depth: 3
|
||||
---
|
||||
|
||||
When working with databases, it's common to create a `seed.sql` file which contains a subset of production data for testing.
|
||||
|
||||
During early development, it's fine to dump the entire database and restore it on your development machine.
|
||||
However, once you have production users this becomes a security issue - do you really want to dump your users' data onto your local machines?
|
||||
|
||||
There are many ways to solve this, but recently I stumbled upon a neat way to do it using PostgreSQL's Row Level Security (RLS).
|
||||
|
||||
The concept is simple:
|
||||
|
||||
1. Create a database user with restricted access.
|
||||
2. Define some RLS rules for that user, limiting what data they can access.
|
||||
3. Run `pg_dump` as that user.
|
||||
|
||||
For this scenario, let's imagine that you have a table called `profiles` in your database:
|
||||
|
||||
```sql hideCopy
|
||||
create table profiles (
|
||||
id serial primary key,
|
||||
name text,
|
||||
email text
|
||||
);
|
||||
```
|
||||
|
||||
| `id` | `name` | `email` |
|
||||
| ---- | ------------ | ------------------------ |
|
||||
| `1` | `Employee 1` | `employee1@supabase.com` |
|
||||
| `2` | `Employee 2` | `employee2@supabase.com` |
|
||||
| `3` | `Employee 3` | `employee3@supabase.com` |
|
||||
| `4` | `Jenny` | `jenny@example.com` |
|
||||
| `5` | `Joe` | `joe@example.com` |
|
||||
|
||||
In this case, if we ran a `pg_dump` we will save Jenny and Joe's personal data. We don't want that, so let's create a Postgres user called `exporter`, who can only dump the data we want.
|
||||
|
||||
### Step 1: Prepare a user
|
||||
|
||||
Create a user to connect to the database. We'll call them `exporter` and grant them access to the public schema:
|
||||
|
||||
```sql hideCopy
|
||||
-- Create a new user with login privileges
|
||||
create user exporter
|
||||
with password 'exporter_secure_password';
|
||||
|
||||
-- Allow this user to select the rows we need
|
||||
grant usage on schema public to exporter;
|
||||
grant select on profiles to exporter;
|
||||
```
|
||||
|
||||
### Step 2: Create data access rules
|
||||
|
||||
Let's turn on RLS for this table and limit the data which `exporter` can access:
|
||||
|
||||
```sql hideCopy hideLineNumbers
|
||||
-- Turn on Row Level Security
|
||||
alter table profiles
|
||||
enable row level security;
|
||||
|
||||
-- Only dump data for internal team members 1, 2, 3
|
||||
create policy "Data dump rule" on profiles
|
||||
for select
|
||||
to exporter
|
||||
using (
|
||||
id in (1, 2, 3)
|
||||
);
|
||||
```
|
||||
|
||||
### Step 3: Export the data
|
||||
|
||||
Now we can use `pg_dump` to get only the data that we need.
|
||||
|
||||
Run the dump with the `exporter` user that we created above and use the `--enable-row-security` flag to ensure that the dump succeeds.
|
||||
|
||||
```bash hideCopy hideLineNumbers
|
||||
# Dump all the data into a "seed.sql" file
|
||||
# which we can use to restore our local databases.
|
||||
pg_dump \
|
||||
-h db.host.supabase.co \
|
||||
-U exporter \
|
||||
-d postgres \
|
||||
-n public \
|
||||
--data-only \
|
||||
--enable-row-security \
|
||||
--table=profiles \
|
||||
> seed.sql
|
||||
```
|
||||
|
||||
```bash hideCopy hideLineNumbers
|
||||
-h db.host.supabase.co \
|
||||
```
|
||||
|
||||
And that's it. You can follow this same pattern for any tables that you want to dump.
|
||||
|
||||
## Data access patterns
|
||||
|
||||
RLS is a bit like appending a “where” clause to a `select`, so you can create all sorts of data access patterns. Let's see a few more which are useful for extracting seed data.
|
||||
|
||||
### Using email rules
|
||||
|
||||
Instead of using hardcoded numbers in our RLS policies, we could use email extensions to determine the users who we want to export:
|
||||
|
||||
```sql hideCopy
|
||||
-- Only dump data for supabase employees
|
||||
create policy "Data dump rule" on profiles
|
||||
for select
|
||||
to exporter
|
||||
using (
|
||||
substring(email from '@(.*)$') = 'supabase.com'
|
||||
);
|
||||
|
||||
```
|
||||
|
||||
### Only recent data
|
||||
|
||||
If we have a table with a lot of data, like an `analytics` table, we might only care about the last 2 months of data.
|
||||
|
||||
```sql hideCopy
|
||||
-- A fake analytics table where we store actions a user takes
|
||||
create table analytics (
|
||||
id serial primary key,
|
||||
ts timestamptz default now(),
|
||||
profile_id references profiles,
|
||||
event text
|
||||
);
|
||||
alter table profiles
|
||||
enable row level security;
|
||||
|
||||
-- Here is an "age" rule so that we only dump the most recent analytics
|
||||
create policy "Data dump rule" on logs
|
||||
for select
|
||||
to exporter
|
||||
using (
|
||||
profile_id in (1, 2, 3) and
|
||||
ts > now() - interval '2 MONTHS' -- here's the magic
|
||||
);
|
||||
```
|
||||
|
||||
### Using flags
|
||||
|
||||
If you don't mind having some additional columns in you database, you can add flags to each row to determine whether it's safe to export.
|
||||
|
||||
```sql hideCopy
|
||||
create table profiles (
|
||||
id serial primary key,
|
||||
name text,
|
||||
email text,
|
||||
is_exportable boolean -- make this "TRUE" if you want to allow access
|
||||
);
|
||||
alter table profiles
|
||||
enable row level security;
|
||||
|
||||
-- Only dump data for internal team members 1, 2, 3
|
||||
create policy "Data dump rule" on profiles
|
||||
for select
|
||||
to exporter
|
||||
using ( is_exportable = true );
|
||||
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
Using `seed` data isn't the only way to run development environments. It's also possible to run fully-masked copies of your database using tools like
|
||||
[Snaplet](https://docs.snaplet.dev/tutorials/supabase-clone-environments).
|
||||
|
||||
We're also bullish on copy-on-write strategies which allow users to "fork" a database at a point in time,
|
||||
a strategy used by [Database Lab Engine](https://postgres.ai/docs/database-lab).
|
||||
DLE uses the ZFS file system to achieve this, but it's within reach of the Postgres core once alternative storage strategies become easier to implement.
|
||||
|
||||
If you want to try out the steps we described in this article, fire up a full PostgreSQL database: [database.new](https://database.new)
|
||||
@@ -1,39 +1,44 @@
|
||||
import { useCallback, useMemo } from 'react'
|
||||
import reactSyntaxHighlighter, { Light as SyntaxHighlighter } from 'react-syntax-highlighter'
|
||||
import monokaiCustomTheme from 'data/CodeEditorTheme'
|
||||
import CodeBlockStyles from './CodeBlock.module.css'
|
||||
import { Button, IconCopy } from '@supabase/ui'
|
||||
import { Button, IconCheck, IconCopy, IconFile, IconTerminal } from '@supabase/ui'
|
||||
import { useState } from 'react'
|
||||
import CopyToClipboard from 'react-copy-to-clipboard'
|
||||
import rangeParser from 'parse-numeric-range'
|
||||
|
||||
import classNames from 'classnames'
|
||||
|
||||
import { Light as SyntaxHighlighter } from 'react-syntax-highlighter'
|
||||
import bash from 'react-syntax-highlighter/dist/cjs/languages/hljs/bash'
|
||||
import js from 'react-syntax-highlighter/dist/cjs/languages/hljs/javascript'
|
||||
import ts from 'react-syntax-highlighter/dist/cjs/languages/hljs/typescript'
|
||||
import py from 'react-syntax-highlighter/dist/cjs/languages/hljs/python'
|
||||
import sql from 'react-syntax-highlighter/dist/cjs/languages/hljs/sql'
|
||||
import monokaiCustomTheme from './CodeBlock.utils'
|
||||
|
||||
export interface CodeBlockProps {
|
||||
lang: 'js' | 'ts' | 'sql' | 'py'
|
||||
interface Props {
|
||||
lang: 'js' | 'sql' | 'py' | 'bash'
|
||||
startingLineNumber?: number
|
||||
hideCopy?: boolean
|
||||
showLineNumbers?: boolean
|
||||
className?: string
|
||||
children?: string
|
||||
size?: 'small' | 'medium' | 'large'
|
||||
/**
|
||||
* Inline styling
|
||||
* Supports CSS Properties in camelcase
|
||||
*/
|
||||
style?: React.CSSProperties | undefined
|
||||
/**
|
||||
* Lines to be highlighted.
|
||||
* Supports individual lines: '14', multiple lines: '14,15', or a range of lines '14..19'
|
||||
*/
|
||||
highlightLines?: string
|
||||
hideBorder?: boolean
|
||||
}
|
||||
|
||||
function CodeBlock(props: CodeBlockProps) {
|
||||
function CodeBlock(props: Props) {
|
||||
const [copied, setCopied] = useState(false)
|
||||
|
||||
const firstLine = props.children ? props.children.split('\n')[0] : ''
|
||||
|
||||
let filename = ''
|
||||
|
||||
if (firstLine.includes('filename =')) {
|
||||
filename = firstLine.split('=')[1]
|
||||
}
|
||||
|
||||
const content =
|
||||
props.children && filename ? props.children.replace(`${firstLine}\n\n`, '') : props.children
|
||||
|
||||
const handleCopy = () => {
|
||||
setCopied(true)
|
||||
setTimeout(() => {
|
||||
setCopied(false)
|
||||
}, 1000)
|
||||
}
|
||||
|
||||
let lang = props.lang
|
||||
? props.lang
|
||||
: props.className
|
||||
@@ -41,87 +46,90 @@ function CodeBlock(props: CodeBlockProps) {
|
||||
: 'js'
|
||||
// force jsx to be js highlighted
|
||||
if (lang === 'jsx') lang = 'js'
|
||||
if (lang === 'tsx') lang = 'ts'
|
||||
|
||||
SyntaxHighlighter.registerLanguage('js', js)
|
||||
SyntaxHighlighter.registerLanguage('ts', ts)
|
||||
SyntaxHighlighter.registerLanguage('py', py)
|
||||
SyntaxHighlighter.registerLanguage('sql', sql)
|
||||
SyntaxHighlighter.registerLanguage('bash', bash)
|
||||
|
||||
// const large = props.size === 'large' ? true : false
|
||||
const large = false
|
||||
|
||||
const shouldHighlightLines = props.highlightLines !== undefined
|
||||
const highlightLines = useMemo(
|
||||
() => new Set(rangeParser(props.highlightLines ?? '')),
|
||||
[props.highlightLines]
|
||||
)
|
||||
|
||||
const lineProps = useCallback(
|
||||
(lineNumber: number) => {
|
||||
const shouldHighlightLine = !shouldHighlightLines || highlightLines.has(lineNumber)
|
||||
|
||||
const style = shouldHighlightLine ? {} : { filter: 'grayscale(75%)', opacity: 0.5 }
|
||||
|
||||
return {
|
||||
className: classNames(
|
||||
CodeBlockStyles['code-line'],
|
||||
shouldHighlightLines && shouldHighlightLine && CodeBlockStyles['code-line--flash']
|
||||
),
|
||||
style,
|
||||
}
|
||||
},
|
||||
[highlightLines]
|
||||
)
|
||||
|
||||
return (
|
||||
<div className="relative">
|
||||
<SyntaxHighlighter
|
||||
language={lang}
|
||||
style={monokaiCustomTheme}
|
||||
className={[
|
||||
CodeBlockStyles['code-block'],
|
||||
'!bg-scale-1200 dark:!bg-scale-100',
|
||||
props.hideBorder ? '' : 'border-scale-1100 dark:border-scale-400 rounded-lg border',
|
||||
].join(' ')}
|
||||
customStyle={{
|
||||
padding: 0,
|
||||
fontSize: large ? 18 : 12,
|
||||
lineHeight: large ? 1.2 : 1.2,
|
||||
<div className="not-prose dark overflow-hidden">
|
||||
{filename && (
|
||||
<div
|
||||
className="
|
||||
bg-scale-200
|
||||
text-scale-900
|
||||
flex
|
||||
h-8 w-full
|
||||
items-center
|
||||
|
||||
gap-1
|
||||
rounded-tr
|
||||
rounded-tl
|
||||
|
||||
border-t
|
||||
|
||||
...props.style,
|
||||
}}
|
||||
showLineNumbers={lang === 'cli' ? false : true}
|
||||
lineNumberContainerStyle={{
|
||||
paddingTop: '128px',
|
||||
}}
|
||||
lineNumberStyle={{
|
||||
display: 'inline-flex',
|
||||
justifyContent: 'flex-end',
|
||||
minWidth: '48px',
|
||||
// background: 'var(--colors-fixed-scale12)',
|
||||
paddingLeft: '21px',
|
||||
marginRight: '12px',
|
||||
color: 'var(--colors-fixed-scale7)',
|
||||
fontSize: large ? 14 : 12,
|
||||
paddingTop: '4px',
|
||||
paddingBottom: '4px',
|
||||
}}
|
||||
wrapLines={true}
|
||||
lineProps={lineProps}
|
||||
>
|
||||
{props.children}
|
||||
</SyntaxHighlighter>
|
||||
|
||||
{!props.hideCopy && props.children ? (
|
||||
<div className="dark absolute right-2 top-2">
|
||||
<CopyToClipboard text={props.children}>
|
||||
<Button type="default" icon={<IconCopy />}>
|
||||
Copy
|
||||
</Button>
|
||||
</CopyToClipboard>
|
||||
border-r
|
||||
border-l
|
||||
px-4
|
||||
font-sans
|
||||
"
|
||||
>
|
||||
{lang === 'bash' ? (
|
||||
<IconTerminal size={12} strokeWidth={2} />
|
||||
) : (
|
||||
<IconFile size={12} strokeWidth={2} />
|
||||
)}
|
||||
<span className="text-xs">{filename ?? 'index.js'}</span>
|
||||
</div>
|
||||
) : null}
|
||||
)}
|
||||
<div className="relative">
|
||||
<SyntaxHighlighter
|
||||
language={lang}
|
||||
style={monokaiCustomTheme}
|
||||
className={[!filename && 'rounded-t-lg', 'rounded-b-lg'].join(' ')}
|
||||
customStyle={{
|
||||
padding: '21px 24px',
|
||||
fontSize: large ? 18 : '0.875rem',
|
||||
lineHeight: large ? 1.6 : 1.4,
|
||||
background: '#181818',
|
||||
}}
|
||||
showLineNumbers={props.showLineNumbers}
|
||||
lineNumberStyle={{
|
||||
padding: '0px',
|
||||
marginRight: '21px',
|
||||
minWidth: '1.5em',
|
||||
opacity: '0.3',
|
||||
fontSize: large ? 14 : '0.75rem',
|
||||
}}
|
||||
>
|
||||
{content}
|
||||
</SyntaxHighlighter>
|
||||
{!props.hideCopy && props.children ? (
|
||||
<div className="dark absolute right-2 top-2">
|
||||
<CopyToClipboard text={props.children}>
|
||||
<Button
|
||||
type="text"
|
||||
icon={
|
||||
copied ? (
|
||||
<span className="text-brand-900">
|
||||
<IconCheck strokeWidth={3} />
|
||||
</span>
|
||||
) : (
|
||||
<IconCopy />
|
||||
)
|
||||
}
|
||||
onClick={() => handleCopy()}
|
||||
>
|
||||
{/* {copied ? 'Copied' : 'Copy'} */}
|
||||
</Button>
|
||||
</CopyToClipboard>
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
const monokaiCustomTheme = {
|
||||
hljs: {
|
||||
display: 'block',
|
||||
overflowX: 'auto',
|
||||
padding: '0.5em',
|
||||
background: '#272822',
|
||||
color: '#ddd',
|
||||
},
|
||||
'hljs-tag': {
|
||||
color: '#569cd6',
|
||||
},
|
||||
'hljs-keyword': {
|
||||
color: '#569cd6',
|
||||
fontWeight: 'normal',
|
||||
},
|
||||
'hljs-selector-tag': {
|
||||
color: '#569cd6',
|
||||
fontWeight: 'normal',
|
||||
},
|
||||
'hljs-literal': {
|
||||
color: '#569cd6',
|
||||
fontWeight: 'normal',
|
||||
},
|
||||
'hljs-strong': {
|
||||
color: '#569cd6',
|
||||
},
|
||||
'hljs-name': {
|
||||
color: '#569cd6',
|
||||
},
|
||||
'hljs-code': {
|
||||
color: '#66d9ef',
|
||||
},
|
||||
'hljs-class .hljs-title': {
|
||||
color: 'gray',
|
||||
},
|
||||
'hljs-attribute': {
|
||||
color: '#bf79db',
|
||||
},
|
||||
'hljs-symbol': {
|
||||
color: '#bf79db',
|
||||
},
|
||||
'hljs-regexp': {
|
||||
color: '#bf79db',
|
||||
},
|
||||
'hljs-link': {
|
||||
color: '#bf79db',
|
||||
},
|
||||
'hljs-string': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-bullet': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-subst': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-title': {
|
||||
color: '#3ECF8E',
|
||||
fontWeight: 'normal',
|
||||
},
|
||||
'hljs-section': {
|
||||
color: '#3ECF8E',
|
||||
fontWeight: 'normal',
|
||||
},
|
||||
'hljs-emphasis': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-type': {
|
||||
color: '#3ECF8E',
|
||||
fontWeight: 'normal',
|
||||
},
|
||||
'hljs-built_in': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-builtin-name': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-selector-attr': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-selector-pseudo': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-addition': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-variable': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-template-tag': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-template-variable': {
|
||||
color: '#3ECF8E',
|
||||
},
|
||||
'hljs-comment': {
|
||||
color: '#75715e',
|
||||
},
|
||||
'hljs-quote': {
|
||||
color: '#75715e',
|
||||
},
|
||||
'hljs-deletion': {
|
||||
color: '#75715e',
|
||||
},
|
||||
'hljs-meta': {
|
||||
color: '#75715e',
|
||||
},
|
||||
'hljs-doctag': {
|
||||
fontWeight: 'normal',
|
||||
},
|
||||
'hljs-selector-id': {
|
||||
fontWeight: 'normal',
|
||||
},
|
||||
}
|
||||
|
||||
export default monokaiCustomTheme
|
||||
@@ -30,26 +30,16 @@ const components = {
|
||||
},
|
||||
ImageGrid,
|
||||
img: (props: any) => {
|
||||
const classes = [
|
||||
'next-image--dynamic-fill',
|
||||
'from-brand-500 to-brand-500',
|
||||
'rounded border bg-gradient-to-r via-blue-500',
|
||||
]
|
||||
|
||||
return (
|
||||
<div
|
||||
className="
|
||||
next-image--dynamic-fill
|
||||
to-scale-400
|
||||
from-scale-500 rounded-md
|
||||
border bg-gradient-to-r
|
||||
"
|
||||
to-scale-400
|
||||
from-scale-500 rounded-md
|
||||
border bg-gradient-to-r
|
||||
"
|
||||
>
|
||||
<Image
|
||||
{...props}
|
||||
className="next-image--dynamic-fill to-brand-1000 from-brand-900 rounded-md border bg-gradient-to-r"
|
||||
layout="fill"
|
||||
/>
|
||||
<Image {...props} className="next-image--dynamic-fill rounded-md border" layout="fill" />
|
||||
</div>
|
||||
)
|
||||
},
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 233 KiB |
Reference in new issue
Block a user