Adds initial storage blog post

This commit is contained in:
Inian Parameshwaran committed 2021-03-30 12:16:44 +05:30
1 parent ac81a56f15
commit 59014ceb31
11 files changed
+214 -35

No files matched your search

+164
View File
@@ -0,0 +1,164 @@
---
title: Launching Supabase Storage (Alpha)
description: Learn why Kevin is building intheloop.dev with Supabase
author: inian
image: storage/ph-1.png
thumb: storage/ph-1.png
tags:
- Supabase
date: '03-30-2021'
---
# Storage Blog Post
Today, we are launching one of most requested features - Storage! When you sign up for Supabase, you get a Postgres database with realtime subscriptions, user management, auto-generated APIs and now a scalable object store. These services integrate very well with each other and you don't need to sign up for yet another service just for your storage requirements.
As with anything that we build in Supabase, Storage is fast, secure and scalable. You can use it to store terabytes of Machine Learning training data, your e-commerce product gallery or just your growing collection of JPEGs featuring cats playing synths in space.
This post outlines our design decisions while building Storage, and we'll show you how to use it in your own applications.
## Architecture
There are three key components to Supabase Storage:
- Backend (where the objects are actually stored).
- Middleware (access and permissions). This consists of an API Server and Postgres.
- Frontend (a nice UI).
![Storage Infrastructure](/new/images/blog/storage/infra.png)
The Storage API server sits behind Kong, an API Gateway. It talks to different storage backends like S3, Backblaze, etc to retrieve and store objects.
Object metadata and security rules are stored in your Postgres database.
We have built a powerful file explorer right into the dashboard, and using our Client libraries you can integrate Storage into your applications.
## Frontend
We set out to build the most usable front-end for storing content. Using Lists to display file hierarchies is a poor experience for exploring files - the most common use-case of a shared File system.
As avid Mac users, we've defaulted to using the column-based explorer in Finder as it allow us to quickly drill into nested folders and provides a clear birds eye view of file hierarchies at the same time.
We put in a lot of effort to make sure that the dashboard is fast and easy to navigate with our miller-based column view, allowing you to get to deeply nested folders quickly. If you're a fan of List Views though, don't worry! We have that option available too. The choice is yours.
![Frontend views](/new/images/blog/storage/ph-5.png)
Our File Browser also has a rich previews for a wide set of file types including images, audio, and videos.
![Frontend preview](/new/images/blog/storage/ph-2.png)
And if you already know the location of a file but want to save a few clicks, you can paste the path into the location bar and navigate to it directly.
![Frontend navigation](/new/images/blog/storage/ph-4.png)
## Designing the storage middleware
We focused on performance, security and interoperability with the rest of the Supabase ecosystem.
### **Integration with the Supabase ecosystem**
Supabase is a [collection of open source tools](https://supabase.io/docs#how-it-works) which integrate very well with each other. We evaluated open source object storage servers like [Ceph](https://ceph.io/), [Swift](https://www.openstack.org/software/releases/ocata/components/swift), [Minio](https://min.io/) and [Zenko](https://github.com/scality/Zenko) but none of these tools were a good fit for our existing ecosystem.
**Postgres Compatibility**
Each of these open source tools are amazing, but they all had a major drawback - we couldn't use Postgres as the server's datastore. If you haven't noticed yet, our team likes Postgres a lot 😉.
For example, Minio [uses etcd](https://docs.min.io/docs/minio-multi-user-quickstart-guide.html) (when used in multi-user gateway mode) and Zenko [requires mongodb and Kafka.](https://zenko.readthedocs.io/en/latest/operation/Architecture/index.html) Every project on Supabase is a dedicated Postgres database, so leveraging it for object and user metadata is more efficient, reducing the number of dependencies for anyone using the Supabase ecosystem of tools.
**Smaller footprint**
We are using managed services like [S3](https://aws.amazon.com/s3/), [Wasabi](https://wasabi.com/) and [Backblaze](https://www.backblaze.com/) for our storage backend. We won't be managing our own hard disks and storage capacity, and so we don't require a lot of features offered by existing tools. For example, a large part of Minio's codebase is to offer erasure encoding and bitrot protection, automatically making use of new disks attached to the cluster.
**Integration with Supabase Auth**
Existing tools do not play well with our authentication system. For example, Minio is bundled with its own [auth system](https://docs.min.io/docs/minio-admin-complete-guide.html#user) and there is no easy way to map Minio users to [Supabase users](https://supabase.io/docs/guides/auth).
In the end, we opted to build our own [Storage API server](https://github.com/supabase/storage-api).
## **Security**
### Authentication
Our storage service sits behind [Kong](https://github.com/kong/kong) along with other services like [PostgREST](https://github.com/PostgREST/postgrest) and [Realtime](https://github.com/supabase/realtime). This allows us to reuse our existing Authentication system - any requests with a valid API key are authenticated and passed to the storage API.
### Authorization
For Authorization, we wanted to avoid creating a new DSL like Firebase. Instead, we created one where you can write policies in the One True Language - SQL!
![One True Language](/new/images/blog/storage/true-language.png)
To do this, we leverage Postgres' Row Level Security. We create a table for `buckets` and `objects` inside each Supabase project. These tables are namespaced in a separate schema called `storage`.
The idea is simple - if a user is able to `select` from the `objects` table, they can retrieve the object too. Using Postgres' Row Level Security, you can define fine-grained Policies to determine access levels for different users.
When a user makes a request for a file, the API detects the user in the `Authorization` header and tries to `select` from the `objects` table. If a valid row is returned, the Storage API pipes the object back from S3. Similarly if the user is able to delete the row from the objects table, they can delete the object from the storage backend too.
For example, if you want to give "read" access to a bucket called `avatars` you would use this policy:
```sql
create policy "Read access for avatars."
on storage.objects for select using (
bucket_id = 'avatars'
);
```
Extending this example, if you want to give access to a subfolder only (in this case called `public`):
```sql
create policy "Read access for public avatars."
on storage.objects for select using (
bucket_id = 'avatars'
and (storage.foldername(name))[1] = 'public'
);
```
We have created helper functions like `foldername()`, `filename()` and `extension()` in the storage schema to make Policies even simpler.
This system integrates with our [User Management system](https://supabase.io/docs/guides/auth). Here is an example policy which gives access to a particular file to a Supabase user:
```sql
create policy crud_uid_file
on storage.objects for all using (
bucket_id = 'avatars'
and name = 'folder/only_uid.jpg'
and auth.uid() = 'd8c7bce9-cfeb-497b-bd61-e66ce2cbdaa2'
);
```
Using the power of Postgres' Row Level Security, you can create pretty much any policy imaginable.
### Performance
The storage server is built with [Fastify](https://www.fastify.io/) and Typescript. Fastify is one of the [fastest](https://www.fastify.io/benchmarks/) Node frameworks and our initial benchmark results look very promising.
We use Node streams everywhere. Objects are uploaded directly to S3 with minimal in-memory buffering. This minimizes RAM consumption even while uploading huge objects. The code does become a bit more complicated when using streams. For example, you can't figure out the size of the object being uploaded before streaming it to S3. But we believe this tradeoff is worth it.
Postgres is another hidden gem for our storage performance. For example, what if you had a bucket with thousands of objects, and you needed to find all objects which a user has access to? Without Postgres, you would retrieve all objects from that folder, evaluate each policy in the storage middleware and only return the objects which the user has access to. But we can avoid this completely with Postgres! We use Postgres to evaluate all the polices, and the storage middleware just returns the objects which the user has access to.
Supabase Storage has some opinionated defaults with respect to caching. Objects retrieved from S3 typically do not have a `Cache-Control` header. This isn't optimal for a Storage system since browsers don't cache objects completely without this header. Objects retrieved from Supabase Storage by default have a Cache-Control header of 1 hour. Of course, you can override this behaviour by specifying a different cache time when creating the object.
## Storage Backend
File objects are stored in an S3 bucket within the same region as your Supabase project. S3 has a high durability of 99.999999999% and is scalable as an object store.
While we started with S3, other storage backends like Wasabi, Backblaze and Openstack Swift expose an S3 compatible API. It will be simple to add more S3 compatible storage options in the future.
We have intentionally kept the API surface for the storage backend very small, in case the community also wants to add storage options which don't support the S3 API.
### Client libraries
You may be itching to get started tinkering around with our new Supabase storage - we have an example app prepared for you right [here](https://github.com/supabase/supabase-js/tree/master/example/next-storage) which simulates a simple context of setting an avatar image for a user. This will help you to get a high level overview of how to use your project's storage with our client library. For greater detail, refer to our storage API documentation here.
![Example app 1](/new/images/blog/storage/ph-7.png)
![Example app 2](/new/images/blog/storage/ph-6.png)
## What's next
- We currently support S3 and will be adding more storage backends. Vote for the storage backends you would like to see us implement here.
- We will integrate our storage service with a Content Delivery Network. With a CDN, objects are cached on a global network. This leads to faster access times for your users around the world.
- We are working on transformations like resizing of images and automatic optimization of different media types. This makes it easy to embed Supabase objects directly in your websites and mobile applications without additional processing.
- A better editor to make authoring policies easier and less error prone.
- Reach out to us if you would like to help out with adding storage support in one of the [community maintained client libraries](https://supabase.io/docs/reference/javascript/supabase-client).
Take it for a spin on our [dashboard](http://app.supabase.io/) and let us know what you think!
+7
View File
@@ -33,5 +33,12 @@
"author_image_url": "https://github.com/supabase.png",
"authorURL": "https://github.com/supabase",
"position": ""
},
"inian": {
"author": "Inian Parameshwaran",
"author_url": "https://twitter.com/everConfusedGuy",
"position": "Engineering",
"username": "inian",
"author_image_url": "https://avatars.githubusercontent.com/u/2155545?v=4"
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 59 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 81 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 281 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 123 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 133 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 207 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 116 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 130 KiB

+43 -35
View File
@@ -5,15 +5,23 @@
<link>https://supabase.io</link>
<description>Latest news from Supabase</description>
<language>en</language>
<lastBuildDate>Sun, 21 Mar 2021 16:00:00 GMT</lastBuildDate>
<lastBuildDate>Mon, 29 Mar 2021 18:30:00 GMT</lastBuildDate>
<atom:link href="https://supabase.io/blog/rss.xml" rel="self" type="application/rss+xml"/>
<item>
<guid>https://supabase.io/blog/2021/03/30/supabase-storage</guid>
<title>Launching Supabase Storage (Alpha)</title>
<link>https://supabase.io/blog/2021/03/30/supabase-storage</link>
<description>Learn why Kevin is building intheloop.dev with Supabase</description>
<pubDate>Mon, 29 Mar 2021 18:30:00 GMT</pubDate>
</item>
<item>
<guid>https://supabase.io/blog/2021/03/22/In-The-Loop</guid>
<title>Developers stay up to date with intheloop.dev</title>
<link>https://supabase.io/blog/2021/03/22/In-The-Loop</link>
<description>Learn why Kevin is building intheloop.dev with Supabase</description>
<pubDate>Sun, 21 Mar 2021 16:00:00 GMT</pubDate>
<pubDate>Sun, 21 Mar 2021 18:30:00 GMT</pubDate>
</item>
<item>
@@ -21,15 +29,15 @@
<title>Using Supabase in Replit</title>
<link>https://supabase.io/blog/2021/03/11/using-supabase-replit</link>
<description>Free hosted relational database from within your node.js repl</description>
<pubDate>Wed, 10 Mar 2021 16:00:00 GMT</pubDate>
<pubDate>Wed, 10 Mar 2021 18:30:00 GMT</pubDate>
</item>
<item>
<guid>https://supabase.io/blog/2021/03/08/toad-a-link-shorterner-with-simple-apis-for-low-coders</guid>
<title>Toad, a link shorterner with simple APIs for low-coders</title>
<link>https://supabase.io/blog/2021/03/08/toad-a-link-shorterner-with-simple-apis-for-low-coders</link>
<guid>https://supabase.io/blog/2021/03/08/toad-a-link-shortener-with-simple-apis-for-low-coders</guid>
<title>Toad, a link shortener with simple APIs for low-coders</title>
<link>https://supabase.io/blog/2021/03/08/toad-a-link-shortener-with-simple-apis-for-low-coders</link>
<description>An easy-to-use link shortening tool with simple APIs</description>
<pubDate>Sun, 07 Mar 2021 16:00:00 GMT</pubDate>
<pubDate>Sun, 07 Mar 2021 18:30:00 GMT</pubDate>
</item>
<item>
@@ -37,7 +45,7 @@
<title>Postgres as a CRON Server</title>
<link>https://supabase.io/blog/2021/03/05/postgres-as-a-cron-server</link>
<description>Running repetitive tasks with your Postgres database.</description>
<pubDate>Thu, 04 Mar 2021 16:00:00 GMT</pubDate>
<pubDate>Thu, 04 Mar 2021 18:30:00 GMT</pubDate>
</item>
<item>
@@ -45,7 +53,7 @@
<title>Supabase Beta February 2021</title>
<link>https://supabase.io/blog/2021/03/02/supabase-beta-february-2021</link>
<description>One year of building.</description>
<pubDate>Mon, 01 Mar 2021 16:00:00 GMT</pubDate>
<pubDate>Mon, 01 Mar 2021 18:30:00 GMT</pubDate>
</item>
<item>
@@ -53,7 +61,7 @@
<title>Cracking PostgreSQL Interview Questions</title>
<link>https://supabase.io/blog/2021/02/27/cracking-postgres-interview</link>
<description>Understand the top PostgreSQL Interview Questions</description>
<pubDate>Fri, 26 Feb 2021 16:00:00 GMT</pubDate>
<pubDate>Fri, 26 Feb 2021 18:30:00 GMT</pubDate>
</item>
<item>
@@ -61,7 +69,7 @@
<title>Roboflow.com choose Supabase to power Paint.wtf leaderboard</title>
<link>https://supabase.io/blog/2021/02/09/case-study-roboflow</link>
<description>Learn how Roboflow.com used Supabase to build their Paint.wtf leaderboard</description>
<pubDate>Mon, 08 Feb 2021 16:00:00 GMT</pubDate>
<pubDate>Mon, 08 Feb 2021 18:30:00 GMT</pubDate>
</item>
<item>
@@ -69,7 +77,7 @@
<title>Supabase Beta January 2021</title>
<link>https://supabase.io/blog/2021/02/02/supabase-beta-january-2021</link>
<description>Eleven months of building.</description>
<pubDate>Mon, 01 Feb 2021 16:00:00 GMT</pubDate>
<pubDate>Mon, 01 Feb 2021 18:30:00 GMT</pubDate>
</item>
<item>
@@ -77,7 +85,7 @@
<title>Supabase Beta December 2020</title>
<link>https://supabase.io/blog/2021/01/02/supabase-beta-december-2020</link>
<description>Ten months of building.</description>
<pubDate>Fri, 01 Jan 2021 16:00:00 GMT</pubDate>
<pubDate>Fri, 01 Jan 2021 18:30:00 GMT</pubDate>
</item>
<item>
@@ -85,7 +93,7 @@
<title>Making the Supabase Dashboard Supa-fast</title>
<link>https://supabase.io/blog/2020/12/13/supabase-dashboard-performance</link>
<description>Improving the performance of the Supabase dashboard</description>
<pubDate>Sat, 12 Dec 2020 16:00:00 GMT</pubDate>
<pubDate>Sat, 12 Dec 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -93,7 +101,7 @@
<title>Supabase Partners With Strive School To Help Teach Open Source</title>
<link>https://supabase.io/blog/2020/12/02/supabase-striveschool</link>
<description>Supabase Partners With Strive School To Help Teach Open Source To The Next Generation Of Developers</description>
<pubDate>Tue, 01 Dec 2020 16:00:00 GMT</pubDate>
<pubDate>Tue, 01 Dec 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -101,7 +109,7 @@
<title>Xendit Built a Counter-Fraud Watchlist for the Fintech Industry</title>
<link>https://supabase.io/blog/2020/12/02/case-study-xendit</link>
<description>See how Xendit use Supabase to build a full-text search engine.</description>
<pubDate>Tue, 01 Dec 2020 16:00:00 GMT</pubDate>
<pubDate>Tue, 01 Dec 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -109,7 +117,7 @@
<title>TAYFA Built a No-Code Website Builder in Seven Days</title>
<link>https://supabase.io/blog/2020/12/02/case-study-tayfa</link>
<description>See how Tayfa went from idea to paying customer in less than 30 days.</description>
<pubDate>Tue, 01 Dec 2020 16:00:00 GMT</pubDate>
<pubDate>Tue, 01 Dec 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -117,7 +125,7 @@
<title>Monitoro Built a Web Crawler Handling Millions of API Requests</title>
<link>https://supabase.io/blog/2020/12/02/case-study-monitoro</link>
<description>See how Monitoro built an automated scraping platform using Supabase.</description>
<pubDate>Tue, 01 Dec 2020 16:00:00 GMT</pubDate>
<pubDate>Tue, 01 Dec 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -125,7 +133,7 @@
<title>Supabase Alpha November 2020</title>
<link>https://supabase.io/blog/2020/12/01/supabase-alpha-november-2020</link>
<description>Nine months of building.</description>
<pubDate>Mon, 30 Nov 2020 16:00:00 GMT</pubDate>
<pubDate>Mon, 30 Nov 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -133,7 +141,7 @@
<title>Postgres Views</title>
<link>https://supabase.io/blog/2020/11/18/postgresql-views</link>
<description>Creating and using a view in PostgreSQL.</description>
<pubDate>Tue, 17 Nov 2020 16:00:00 GMT</pubDate>
<pubDate>Tue, 17 Nov 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -141,7 +149,7 @@
<title>Supabase Alpha October 2020</title>
<link>https://supabase.io/blog/2020/11/02/supabase-alpha-october-2020</link>
<description>Eight months of building.</description>
<pubDate>Sun, 01 Nov 2020 16:00:00 GMT</pubDate>
<pubDate>Sun, 01 Nov 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -149,7 +157,7 @@
<title>Supabase.js 1.0</title>
<link>https://supabase.io/blog/2020/10/30/improved-dx</link>
<description>We're releasing a new version of our Supabase client with some awesome new improvements.</description>
<pubDate>Thu, 29 Oct 2020 16:00:00 GMT</pubDate>
<pubDate>Thu, 29 Oct 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -157,7 +165,7 @@
<title>Supabase Alpha September 2020</title>
<link>https://supabase.io/blog/2020/10/03/supabase-alpha-september-2020</link>
<description>Seven months of building.</description>
<pubDate>Fri, 02 Oct 2020 16:00:00 GMT</pubDate>
<pubDate>Fri, 02 Oct 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -165,7 +173,7 @@
<title>Supabase Hacktoberfest 2020</title>
<link>https://supabase.io/blog/2020/09/11/supabase-hacktoberfest-2020</link>
<description>Join us for a celebration of open source software and learn how to contribute to Supabase.</description>
<pubDate>Thu, 10 Sep 2020 16:00:00 GMT</pubDate>
<pubDate>Thu, 10 Sep 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -173,7 +181,7 @@
<title>Supabase Alpha August 2020</title>
<link>https://supabase.io/blog/2020/09/03/supabase-alpha-august-2020</link>
<description>Six months of building</description>
<pubDate>Wed, 02 Sep 2020 16:00:00 GMT</pubDate>
<pubDate>Wed, 02 Sep 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -181,7 +189,7 @@
<title>Supabase Auth</title>
<link>https://supabase.io/blog/2020/08/05/supabase-auth</link>
<description>Authenticate and authorize your users with Supabase Auth</description>
<pubDate>Tue, 04 Aug 2020 16:00:00 GMT</pubDate>
<pubDate>Tue, 04 Aug 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -189,7 +197,7 @@
<title>Supabase Alpha July 2020</title>
<link>https://supabase.io/blog/2020/08/02/supabase-alpha-july-2020</link>
<description>Five months of building</description>
<pubDate>Sat, 01 Aug 2020 16:00:00 GMT</pubDate>
<pubDate>Sat, 01 Aug 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -197,7 +205,7 @@
<title>Continuous PostgreSQL Backups using WAL-G</title>
<link>https://supabase.io/blog/2020/08/02/continuous-postgresql-backup-walg</link>
<description>Have you ever wanted to restore your database's state to a particular moment in time? This post explains how, using WAL-G.</description>
<pubDate>Sat, 01 Aug 2020 16:00:00 GMT</pubDate>
<pubDate>Sat, 01 Aug 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -205,7 +213,7 @@
<title>Alpha Launch Postmortem</title>
<link>https://supabase.io/blog/2020/07/10/alpha-launch-postmortem</link>
<description>Everything that went wrong with Supabase's launch</description>
<pubDate>Thu, 09 Jul 2020 16:00:00 GMT</pubDate>
<pubDate>Thu, 09 Jul 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -213,7 +221,7 @@
<title>What are PostgreSQL Templates?</title>
<link>https://supabase.io/blog/2020/07/09/postgresql-templates</link>
<description>What are PostgreSQL templates and what are they used for?</description>
<pubDate>Wed, 08 Jul 2020 16:00:00 GMT</pubDate>
<pubDate>Wed, 08 Jul 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -221,7 +229,7 @@
<title>Physical vs Logical Backups in PostgreSQL</title>
<link>https://supabase.io/blog/2020/07/17/postgresql-physical-logical-backups</link>
<description>What are physical and logical backups in Postgres?</description>
<pubDate>Mon, 06 Jul 2020 16:00:00 GMT</pubDate>
<pubDate>Mon, 06 Jul 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -229,7 +237,7 @@
<title>Supabase Alpha June 2020</title>
<link>https://supabase.io/blog/2020/07/01/supabase-alpha-june-2020</link>
<description>Four months of building</description>
<pubDate>Tue, 30 Jun 2020 16:00:00 GMT</pubDate>
<pubDate>Tue, 30 Jun 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -237,7 +245,7 @@
<title>Steve Chavez has joined Supabase</title>
<link>https://supabase.io/blog/2020/06/15/supabase-steve-chavez</link>
<description>Steve joins Supabase to help build Auth.</description>
<pubDate>Sun, 14 Jun 2020 16:00:00 GMT</pubDate>
<pubDate>Sun, 14 Jun 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -245,7 +253,7 @@
<title>Supabase Alpha May 2020</title>
<link>https://supabase.io/blog/2020/06/01/supabase-alpha-may-2020</link>
<description>Three months of building</description>
<pubDate>Sun, 31 May 2020 16:00:00 GMT</pubDate>
<pubDate>Sun, 31 May 2020 18:30:00 GMT</pubDate>
</item>
<item>
@@ -253,7 +261,7 @@
<title>Supabase Alpha April 2020</title>
<link>https://supabase.io/blog/2020/05/01/supabase-alpha-april-2020</link>
<description>Two months of building</description>
<pubDate>Sun, 31 May 2020 16:00:00 GMT</pubDate>
<pubDate>Sun, 31 May 2020 18:30:00 GMT</pubDate>
</item>
</channel>