feat(studio): move Warehouse setup into Integrations (#50247)

## What kind of change does this PR introduce?

Feature and information architecture change. Builds on #50246.

## What is the current behaviour?

Warehouse setup, progress, errors, table selection, and connection
details all live in the transient Connect sheet. Closing the sheet hides
the current replication state, and the integration is absent from the
Integrations page.

## What is the new behaviour?

Warehouse now has a persistent Overview page at
`/project/{ref}/integrations/warehouse/overview`:

- Before setup, the existing schema and table picker enables Warehouse.
- During setup, the page shows the current phase and per-table backfill
state where available.
- Setup and status failures remain visible on the page with retry
actions where possible.
- Once complete, the page shows Status, Tables, then Connect.
- The Connect sheet becomes read-only. Before Warehouse is ready, it
links directly to the Overview page for setup, progress, or recovery.

| Before | After |
| --- | --- |
| <img width="1200" height="907" alt="Chives Pantry Supabase"
src="https://github.com/user-attachments/assets/82dc7fb0-3859-499e-96ab-56f70a5c7325"
/> | <img width="1200" height="907" alt="Chives Pantry Supabase"
src="https://github.com/user-attachments/assets/7428e301-27f8-48fe-91e0-6880b132920d"
/> |
| <img width="1200" height="907" alt="Chives Pantry Supabase"
src="https://github.com/user-attachments/assets/82dc7fb0-3859-499e-96ab-56f70a5c7325"
/> | <img width="1200" height="907" alt="54709"
src="https://github.com/user-attachments/assets/0c589e5b-e13c-4554-a6ee-6730d0e95c07"
/> |
| <img width="1200" height="907" alt="ETL BigTable ETL Team Supabase"
src="https://github.com/user-attachments/assets/18ec9f6c-3724-453f-bbbb-c7149758246e"
/> | <img width="1200" height="907" alt="Regular AWS Teamer Supabase"
src="https://github.com/user-attachments/assets/3819c9fe-e56d-4cec-988d-5e724f8d990a"
/> |

## To test

Use a project whose organisation is included in the Warehouse
allow-list.

1. Open `/project/{ref}/integrations`, filter by **Data platform**, and
open Warehouse.
2. Before setup, confirm the existing schema and table picker appears
and starts with no tables selected.
3. Start setup and confirm the Status section polls through setup and
table backfill phases.
4. Confirm setup failures remain visible and expose Retry when the API
returns affected tables.
5. After setup, confirm the section order is Status, Tables, Connect.
6. Confirm existing replicated tables are selected and locked, while
additional tables can be added.
7. Open `/project/{ref}?showConnect=true&connectTab=warehouse` and
confirm it links to the Overview page before setup, during setup, and
after a setup failure.
8. Once setup is complete, confirm the Connect sheet shows the FlightSQL
and DuckDB connection controls from #50246.
9. Repeat the Overview checks with **One-Click Integrations** turned
off.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Supabase Warehouse to the integrations catalog, with overview
documentation and availability-aware display.
* Added Warehouse setup and management flows, including schema and table
selection, replication progress, status details, connection options, and
retry actions.
* Added DuckDB and FlightSQL engine selection with Connect sheet URL and
preference synchronization.
* Added table replication status, lag, timestamps, and size information.

* **Bug Fixes**
* Warehouse setup status requests no longer retry automatically after
failures.
* Improved recovery messaging and retry behavior for setup and
connection errors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
This commit is contained in:
Danny WhiteandJoshen Lim authored and GitHub committed 2026-09-15 12:38:13 +10:00
1 parent ad198b15ab
commit 1755580dcf
24 files changed
+1584 -684

No files matched your search

@@ -6,13 +6,16 @@ import { cn, Sheet, SheetContent, SheetDescription, SheetHeader, SheetTitle } fr
import type { ConnectMode, ProjectKeys } from './Connect.types'
import { ConnectConfigSection, ModeSelector } from './ConnectConfigSection'
import { resolveConnectSheetHydration } from './ConnectSheet.utils'
import {
CLEARED_CONNECT_SHEET_QUERY_PARAMS,
resolveConnectSheetHydration,
} from './ConnectSheet.utils'
import { ConnectStepsSection } from './ConnectStepsSection'
import { useAvailableConnectModes } from './useAvailableConnectModes'
import { useConnectSheetParams } from './useConnectSheetParams'
import { useConnectSheetShortcut } from './useConnectSheetShortcut'
import { useConnectState } from './useConnectState'
import { WarehouseModePanel } from './WarehouseModePanel/WarehouseModePanel'
import { WarehouseTab } from './WarehouseTab'
import { useAPIKeys } from '@/data/api-keys/api-keys-query'
import { useProjectApiUrl } from '@/data/config/project-endpoint-query'
import { useAsyncCheckPermissions } from '@/hooks/misc/useCheckPermissions'
@@ -45,6 +48,7 @@ export const ConnectSheet = () => {
method: queryMethod,
type: queryType,
mcpClient: queryMcpClient,
warehouseQueryEngine: queryWarehouseQueryEngine,
} = params
useEffect(() => {
@@ -64,6 +68,7 @@ export const ConnectSheet = () => {
method: queryMethod,
type: queryType,
mcpClient: queryMcpClient,
warehouseQueryEngine: queryWarehouseQueryEngine,
},
storedPrefs,
availableModeIds
@@ -81,6 +86,7 @@ export const ConnectSheet = () => {
queryMethod,
queryType,
queryMcpClient,
queryWarehouseQueryEngine,
storedPrefs,
availableModeIds,
track,
@@ -91,14 +97,7 @@ export const ConnectSheet = () => {
])
const clearAllQueryParams = () => {
setQueryParams({
connectTab: null,
framework: null,
using: null,
method: null,
type: null,
mcpClient: null,
})
setQueryParams(CLEARED_CONNECT_SHEET_QUERY_PARAMS)
}
const handleOpenChange = (sheetOpen: boolean) => {
@@ -137,6 +136,7 @@ export const ConnectSheet = () => {
method: null,
type: null,
mcpClient: null,
warehouseQueryEngine: null,
})
}
@@ -179,7 +179,7 @@ export const ConnectSheet = () => {
</div>
{state.mode === 'warehouse' ? (
<WarehouseModePanel />
<WarehouseTab />
) : (
<>
{activeFields.length > 0 && (
@@ -9,6 +9,7 @@ export type ConnectSheetQueryParams = {
method: string | null
type: string | null
mcpClient: string | null
warehouseQueryEngine: string | null
}
export type ConnectSheetUrlUpdates = Partial<Record<keyof ConnectSheetQueryParams, string | null>>
@@ -21,6 +22,16 @@ export type ConnectSheetHydration = {
urlUpdates: ConnectSheetUrlUpdates
}
export const CLEARED_CONNECT_SHEET_QUERY_PARAMS = {
connectTab: null,
framework: null,
using: null,
method: null,
type: null,
mcpClient: null,
warehouseQueryEngine: null,
}
function isConnectMode(value: string): value is ConnectMode {
return CONNECT_MODES.some((mode) => mode === value)
}
@@ -58,6 +69,8 @@ export function resolveConnectSheetHydration(
const effectiveMethod = query.method ?? storedPrefs.method ?? null
const effectiveType = query.type ?? storedPrefs.type ?? null
const effectiveMcpClient = query.mcpClient ?? storedPrefs.mcpClient ?? null
const effectiveWarehouseQueryEngine =
query.warehouseQueryEngine ?? storedPrefs.warehouseQueryEngine ?? null
const mappedMode = mapConnectTabToMode(effectiveTab)
const mode = mappedMode && availableModeIds.includes(mappedMode) ? mappedMode : null
@@ -95,6 +108,13 @@ export function resolveConnectSheetHydration(
fieldUpdates.push({ fieldId: 'mcpClient', value: effectiveMcpClient })
if (query.mcpClient === null) urlUpdates.mcpClient = effectiveMcpClient
}
} else if (
mappedMode === 'warehouse' &&
(effectiveWarehouseQueryEngine === 'flightsql' || effectiveWarehouseQueryEngine === 'duckdb')
) {
if (query.warehouseQueryEngine === null) {
urlUpdates.warehouseQueryEngine = effectiveWarehouseQueryEngine
}
}
return { mode, fieldUpdates, urlUpdates }
@@ -1,152 +0,0 @@
import { screen, waitFor } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import { platformComponents as components } from 'api-types'
import { mockAnimationsApi } from 'jsdom-testing-mocks'
import { HttpResponse } from 'msw'
import { describe, expect, test, vi } from 'vitest'
import { WarehouseConnectionDetails } from './WarehouseConnectionDetails'
import { customRender } from '@/tests/lib/custom-render'
import { addAPIMock } from '@/tests/lib/msw'
type WarehouseCatalogResponse = components['schemas']['WarehouseCatalogResponse']
type UpdateWarehouseCatalogBody = components['schemas']['UpdateWarehouseCatalogBody']
const CATALOG_PASSWORD = 'pwd'
const CATALOG_URL = 'postgres://postgres:pwd@db.example.supabase.co:5432/postgres'
const mockCatalog = (catalog: WarehouseCatalogResponse) =>
addAPIMock({
method: 'get',
path: '/platform/warehouse/:ref/catalog',
response: () => HttpResponse.json<WarehouseCatalogResponse>(catalog),
})
mockAnimationsApi()
describe('WarehouseConnectionDetails', () => {
test('shows FlightSQL by default without loading DuckDB catalog access', async () => {
customRender(<WarehouseConnectionDetails onEditTables={vi.fn()} />)
expect(await screen.findByRole('combobox', { name: 'Query engine' })).toBeInTheDocument()
expect(screen.getByDisplayValue('default.warehouse.supabase.io')).toBeInTheDocument()
expect(
screen.getByDisplayValue(
'flightsql://postgres:[YOUR-PASSWORD]@default.warehouse.supabase.io:443?tls=enabled'
)
).toBeInTheDocument()
expect(screen.queryByRole('switch')).not.toBeInTheDocument()
})
test('shows the persistent catalog switch when DuckDB is selected', async () => {
mockCatalog({ enabled: false })
customRender(<WarehouseConnectionDetails onEditTables={vi.fn()} />)
await userEvent.click(await screen.findByRole('combobox', { name: 'Query engine' }))
await userEvent.click(screen.getByRole('option', { name: 'DuckDB' }))
expect(
await screen.findByRole('switch', { name: 'Enable DuckDB catalog access' })
).not.toBeChecked()
expect(screen.queryByRole('heading', { name: 'Follow these steps' })).not.toBeInTheDocument()
})
test('shows DuckDB credentials as numbered steps and removes them when changing engines', async () => {
mockCatalog({
enabled: true,
credentials: {
catalog_url: CATALOG_URL,
data_path: 's3://warehouse/',
metadata_schema: 'ducklake',
s3_access_key_id: 'access-key-id',
s3_endpoint: 'default.storage.supabase.co/storage/v1/s3',
s3_region: 'ap-southeast-1',
s3_secret_access_key: 's3-secret',
},
})
customRender(<WarehouseConnectionDetails onEditTables={vi.fn()} />)
await userEvent.click(await screen.findByRole('combobox', { name: 'Query engine' }))
await userEvent.click(screen.getByRole('option', { name: 'DuckDB' }))
expect(
await screen.findByRole('switch', { name: 'Enable DuckDB catalog access' })
).toBeChecked()
expect(screen.getByRole('heading', { name: 'Follow these steps' })).toBeInTheDocument()
expect(screen.getByText('Set environment variables')).toBeInTheDocument()
expect(screen.getByText('Attach Warehouse')).toBeInTheDocument()
expect(
screen.getByRole('button', { name: 'Copy all DuckLake environment variables' })
).toBeInTheDocument()
expect(screen.queryByText('s3-secret')).not.toBeInTheDocument()
await userEvent.click(screen.getByRole('button', { name: 'Reveal DUCKLAKE_S3_SECRET' }))
expect(screen.getByText('s3-secret')).toBeInTheDocument()
expect(screen.queryByText(CATALOG_PASSWORD)).not.toBeInTheDocument()
await userEvent.click(screen.getByRole('button', { name: 'Reveal DUCKLAKE_METADATA_PASSWORD' }))
expect(screen.getByText(CATALOG_PASSWORD)).toBeInTheDocument()
await userEvent.click(screen.getByRole('combobox', { name: 'Query engine' }))
await userEvent.click(screen.getByRole('option', { name: 'FlightSQL' }))
expect(screen.queryByRole('heading', { name: 'Follow these steps' })).not.toBeInTheDocument()
expect(screen.queryByRole('button', { name: 'Copy prompt' })).not.toBeInTheDocument()
})
test('enables and disables DuckDB catalog access', async () => {
let catalog: WarehouseCatalogResponse = { enabled: false }
const requestBodies: UpdateWarehouseCatalogBody[] = []
let finishUpdatingCatalog: (() => void) | undefined
const updatingCatalog = new Promise<void>((resolve) => {
finishUpdatingCatalog = resolve
})
addAPIMock({
method: 'get',
path: '/platform/warehouse/:ref/catalog',
response: () => HttpResponse.json<WarehouseCatalogResponse>(catalog),
})
addAPIMock({
method: 'post',
path: '/platform/warehouse/:ref/catalog',
response: async ({ request }) => {
const body = (await request.json()) as UpdateWarehouseCatalogBody
requestBodies.push(body)
await updatingCatalog
catalog = { enabled: body.enabled }
return HttpResponse.json<WarehouseCatalogResponse>(catalog)
},
})
customRender(<WarehouseConnectionDetails onEditTables={vi.fn()} />)
await userEvent.click(await screen.findByRole('combobox', { name: 'Query engine' }))
await userEvent.click(screen.getByRole('option', { name: 'DuckDB' }))
const catalogSwitch = await screen.findByRole('switch', {
name: 'Enable DuckDB catalog access',
})
expect(catalogSwitch).not.toBeChecked()
await userEvent.click(catalogSwitch)
expect(
await screen.findByRole('status', { name: 'Updating DuckDB catalog access' })
).toBeInTheDocument()
expect(catalogSwitch).toBeDisabled()
expect(catalogSwitch).toHaveAttribute('aria-busy', 'true')
finishUpdatingCatalog?.()
await waitFor(() => expect(catalogSwitch).toBeChecked())
expect(
screen.queryByRole('status', { name: 'Updating DuckDB catalog access' })
).not.toBeInTheDocument()
await userEvent.click(catalogSwitch)
await waitFor(() => expect(catalogSwitch).not.toBeChecked())
expect(requestBodies).toEqual([{ enabled: true }, { enabled: false }])
})
})
@@ -1,55 +0,0 @@
import { Loader2 } from 'lucide-react'
import { Badge } from 'ui'
import { Admonition } from 'ui-patterns/Admonition'
import type { WarehouseSetupStatusResponse } from '@/data/warehouse/warehouse-setup-status-query'
const TABLE_STATE_BADGE: Record<
WarehouseSetupStatusResponse['tables'][number]['state'],
{ label: string; variant: 'warning' | 'success' | 'destructive' }
> = {
syncing: { label: 'Backfilling', variant: 'warning' },
live: { label: 'Synced', variant: 'success' },
error: { label: 'Error', variant: 'destructive' },
}
export interface WarehouseEnablingProgressProps {
status: WarehouseSetupStatusResponse
}
export const WarehouseEnablingProgress = ({ status }: WarehouseEnablingProgressProps) => {
return (
<div>
<Admonition
type="default"
icon={
<Loader2 size={16} strokeWidth={1.5} className="animate-spin text-foreground-light" />
}
description="Setting up your Warehouse — this can take a few minutes while we backfill selected tables."
className="mb-5"
/>
<div className="border rounded-md overflow-hidden divide-y">
{status.tables.map((table) => {
const badge = TABLE_STATE_BADGE[table.state]
return (
<div
key={`${table.schema}.${table.name}`}
className="flex items-center gap-2 px-3 py-2.5"
>
<span className="text-sm font-mono text-foreground flex-1">
{table.schema}.{table.name}
</span>
<Badge variant={badge.variant}>{badge.label}</Badge>
</div>
)
})}
{status.tables.length === 0 && (
<p className="px-3 py-2.5 text-sm text-foreground-lighter">
No tables are being copied yet.
</p>
)}
</div>
</div>
)
}
@@ -1,130 +0,0 @@
import { useParams } from 'common'
import { useState } from 'react'
import { Button } from 'ui'
import { GenericSkeletonLoader } from 'ui-patterns/ShimmeringLoader'
import { WarehouseConnectionDetails } from './WarehouseConnectionDetails'
import { WarehouseEnablingProgress } from './WarehouseEnablingProgress'
import type { WarehouseSetupTarget } from './WarehouseModePanel.utils'
import { WarehouseSchemaTablePicker } from './WarehouseSchemaTablePicker'
import { AlertError } from '@/components/ui/AlertError'
import { useWarehouseSetupMutation } from '@/data/warehouse/warehouse-setup-mutation'
import { useWarehouseSetupStatusQuery } from '@/data/warehouse/warehouse-setup-status-query'
const POLLING_SETUP_STATUSES = new Set(['setting_up', 'copying'])
interface WarehouseSetupCompleteProps {
onSubmit: (targets: WarehouseSetupTarget[]) => void
isSubmitting: boolean
}
const WarehouseSetupComplete = ({ onSubmit, isSubmitting }: WarehouseSetupCompleteProps) => {
const [isEditingTables, setIsEditingTables] = useState(false)
if (isEditingTables) {
return (
<div className="p-8">
<WarehouseSchemaTablePicker
onSubmit={onSubmit}
isSubmitting={isSubmitting}
onBack={() => setIsEditingTables(false)}
/>
</div>
)
}
return <WarehouseConnectionDetails onEditTables={() => setIsEditingTables(true)} />
}
export const WarehouseModePanel = () => {
const { ref: projectRef } = useParams()
const { data, isPending, isError, error } = useWarehouseSetupStatusQuery(
{ projectRef },
{
refetchInterval: (query) => {
const status = query.state.data?.setup_status
return status && POLLING_SETUP_STATUSES.has(status) ? 3000 : false
},
}
)
const setupMutation = useWarehouseSetupMutation()
const handleSetup = (targets: WarehouseSetupTarget[]) => {
if (!projectRef || targets.length === 0) return
setupMutation.mutate({ projectRef, body: { targets } })
}
if (isPending) {
return (
<div className="p-8">
<GenericSkeletonLoader />
</div>
)
}
if (isError) {
return (
<div className="p-8">
<AlertError subject="Failed to load Warehouse status" error={error} />
</div>
)
}
if (!data) {
return (
<div className="p-8">
<GenericSkeletonLoader />
</div>
)
}
const status = data.setup_status
if (status === 'not_started') {
return (
<div className="p-8">
<WarehouseSchemaTablePicker onSubmit={handleSetup} isSubmitting={setupMutation.isPending} />
</div>
)
}
if (status === 'setting_up' || status === 'copying') {
return (
<div className="p-8">
<WarehouseEnablingProgress status={data} />
</div>
)
}
if (status === 'error') {
const retryTargets: WarehouseSetupTarget[] = (data.tables ?? []).map((table) => ({
type: 'table' as const,
schema: table.schema,
name: table.name,
}))
const failingStep = data.steps.find((step) => step.status === 'error')
return (
<div className="p-8">
<AlertError
subject="Warehouse setup failed"
error={{
message:
failingStep?.message ?? 'An unknown error occurred while setting up Warehouse.',
}}
additionalActions={
retryTargets.length > 0 ? (
<Button loading={setupMutation.isPending} onClick={() => handleSetup(retryTargets)}>
Retry
</Button>
) : undefined
}
/>
</div>
)
}
// status === 'complete'
return <WarehouseSetupComplete onSubmit={handleSetup} isSubmitting={setupMutation.isPending} />
}
@@ -1,263 +0,0 @@
import { useParams } from 'common'
import { ArrowLeft, ChevronRight, Warehouse } from 'lucide-react'
import { useMemo, useState } from 'react'
import { Button, Checkbox, Collapsible, CollapsibleContent, CollapsibleTrigger } from 'ui'
import { GenericSkeletonLoader } from 'ui-patterns/ShimmeringLoader'
import {
buildSelectionFromPublicationTables,
buildWarehouseSetupTargets,
getSchemaCheckedState,
getSchemaTableKey,
getSelectedTableCount,
isSelectableWarehouseSchema,
type SchemaTableSelection,
type SchemaWithTables,
type WarehouseSetupTarget,
} from './WarehouseModePanel.utils'
import { AlertError } from '@/components/ui/AlertError'
import { useSchemasQuery } from '@/data/database/schemas-query'
import { useReplicationPublicationQuery } from '@/data/replication/publication-query'
import { useReplicationSourcesQuery } from '@/data/replication/sources-query'
import { useTablesQuery } from '@/data/tables/tables-query'
import { useSelectedProjectQuery } from '@/hooks/misc/useSelectedProject'
import { WAREHOUSE_PUBLICATION_NAME } from '@/lib/warehouse'
export interface WarehouseSchemaTablePickerProps {
onSubmit: (targets: WarehouseSetupTarget[]) => void
isSubmitting: boolean
/**
* Provided only when the picker was opened to edit an already-enabled Warehouse, which is what
* gives it something to navigate back to (the connection details).
*/
onBack?: () => void
}
export const WarehouseSchemaTablePicker = ({
onSubmit,
isSubmitting,
onBack,
}: WarehouseSchemaTablePickerProps) => {
const { ref: projectRef } = useParams()
const { data: project } = useSelectedProjectQuery()
const isEditing = onBack !== undefined
// `null` until the user touches a checkbox, so the selection seeded from the existing
// publication can arrive asynchronously without an effect syncing it into state.
const [selectionOverride, setSelectionOverride] = useState<SchemaTableSelection | null>(null)
const [expandedOverrides, setExpandedOverrides] = useState<Record<string, boolean>>({})
const {
data: schemas,
isPending: isSchemasPending,
isError: isSchemasError,
error: schemasError,
} = useSchemasQuery({ projectRef, connectionString: project?.connectionString })
const {
data: tables,
isPending: isTablesPending,
isError: isTablesError,
error: tablesError,
} = useTablesQuery({ projectRef, connectionString: project?.connectionString })
// The `supabase_warehouse` publication is the source of truth for what's currently replicated.
// Reading the sources query directly (rather than via useReplicationSourceId) to get its
// loading state: the publications query stays disabled until a source id exists, so without it
// the list would render un-checked and then flash back to a loader once publications kick in.
const { data: sourcesData, isLoading: isSourcesLoading } = useReplicationSourcesQuery({
projectRef,
})
const sourceId = sourcesData?.sources.find((source) => source.name === projectRef)?.id
const {
data: publication,
isError: isPublicationsError,
error: publicationsError,
} = useReplicationPublicationQuery({
projectRef,
sourceId,
publicationName: WAREHOUSE_PUBLICATION_NAME,
})
// Derived from data presence rather than fetch status, so there's no render gap between the
// publication query becoming enabled and it actually starting to fetch.
const isSelectionPending =
isSourcesLoading ||
(sourceId !== undefined && publication === undefined && !isPublicationsError)
const initialSelection = useMemo(
() => buildSelectionFromPublicationTables(publication?.tables ?? []),
[publication]
)
const selection = selectionOverride ?? initialSelection
const schemasWithTables: SchemaWithTables[] = useMemo(() => {
if (!schemas || !tables) return []
return schemas
.filter((schema) => isSelectableWarehouseSchema(schema.name))
.map((schema) => ({
schema: schema.name,
tables: tables.filter((table) => table.schema === schema.name).map((table) => table.name),
}))
.sort((a, b) => a.schema.localeCompare(b.schema))
}, [schemas, tables])
const selectedCount = getSelectedTableCount(selection)
const updateSelection = (updater: (current: SchemaTableSelection) => SchemaTableSelection) => {
setSelectionOverride((prev) => updater(prev ?? initialSelection))
}
const toggleTable = (schema: string, table: string) => {
const key = getSchemaTableKey(schema, table)
updateSelection((current) => ({ ...current, [key]: !current[key] }))
}
const toggleSchema = (schema: SchemaWithTables) => {
const allSelected =
schema.tables.length > 0 &&
schema.tables.every((table) => selection[getSchemaTableKey(schema.schema, table)])
updateSelection((current) => {
const next = { ...current }
schema.tables.forEach((table) => {
next[getSchemaTableKey(schema.schema, table)] = !allSelected
})
return next
})
}
const setExpanded = (schemaName: string, isOpen: boolean) => {
setExpandedOverrides((prev) => ({ ...prev, [schemaName]: isOpen }))
}
const handleSubmit = () => {
const targets = buildWarehouseSetupTargets(selection, schemasWithTables)
if (targets.length === 0) return
onSubmit(targets)
}
// Waiting on the publication too, so the pre-checked selection is in place before the user can
// start toggling (an early toggle would otherwise pin an override that omits existing tables).
if (isSchemasPending || isTablesPending || isSelectionPending) return <GenericSkeletonLoader />
if (isSchemasError) return <AlertError subject="Failed to load schemas" error={schemasError} />
if (isTablesError) return <AlertError subject="Failed to load tables" error={tablesError} />
// Only blocking when editing: a first-time setup starts from an empty selection anyway, so a
// failed publication lookup shouldn't stop the user from enabling Warehouse at all.
if (isEditing && isPublicationsError) {
return <AlertError subject="Failed to load replicated tables" error={publicationsError} />
}
return (
<div>
<div className="flex gap-3 mb-5">
<div className="w-8 h-8 rounded-md bg-brand-200 flex items-center justify-center shrink-0">
<Warehouse size={16} strokeWidth={1.5} className="text-brand" />
</div>
<div>
<p className="text-sm font-medium text-foreground">
{isEditing ? 'Edit replicated tables' : 'Enable Warehouse'}
</p>
<p className="text-sm text-foreground-light mt-1 max-w-xl">
{isEditing
? 'Choose which schemas or tables to replicate to your Warehouse. Tables already replicating are selected.'
: 'Replicate your database to a low-latency analytical endpoint over FlightSQL. Choose which schemas or tables to replicate — you can change this later.'}
</p>
</div>
</div>
<p className="text-sm font-medium text-foreground-light mb-2">
Schemas and tables to replicate
</p>
<div className="border rounded-md overflow-hidden divide-y">
{schemasWithTables.map((schema) => {
const keys = schema.tables.map((table) => getSchemaTableKey(schema.schema, table))
const checkedCount = keys.filter((key) => selection[key]).length
const checkedState = getSchemaCheckedState({
selectedCount: checkedCount,
totalCount: keys.length,
})
const isOpen = expandedOverrides[schema.schema] ?? checkedCount > 0
return (
<Collapsible
key={schema.schema}
open={isOpen}
onOpenChange={(open) => setExpanded(schema.schema, open)}
>
<div className="flex items-center gap-2 px-3 py-2 bg-surface-75">
<CollapsibleTrigger
aria-label={isOpen ? `Collapse ${schema.schema}` : `Expand ${schema.schema}`}
className="group text-foreground-lighter"
>
<ChevronRight
size={14}
className="transition-transform group-data-[state=open]:rotate-90"
/>
</CollapsibleTrigger>
<Checkbox
checked={checkedState}
onCheckedChange={() => toggleSchema(schema)}
disabled={schema.tables.length === 0}
aria-label={`Select all tables in ${schema.schema}`}
// The shared Checkbox only fills itself for `data-state=checked`, so a partial
// selection would otherwise render identically to an empty one. A muted fill
// keeps all three states visually distinct.
className="data-[state=indeterminate]:border-foreground-lighter data-[state=indeterminate]:bg-foreground-lighter"
/>
<span className="text-sm font-mono text-foreground">{schema.schema}</span>
<span className="text-xs text-foreground-lighter ml-auto">
{checkedCount}/{keys.length} tables
</span>
</div>
<CollapsibleContent>
{schema.tables.map((table) => {
const key = getSchemaTableKey(schema.schema, table)
return (
<div key={key} className="flex items-center gap-2 pl-10 pr-3 py-2">
<Checkbox
checked={!!selection[key]}
onCheckedChange={() => toggleTable(schema.schema, table)}
aria-label={`Select ${schema.schema}.${table}`}
/>
<span className="text-sm font-mono text-foreground-light">{table}</span>
</div>
)
})}
{schema.tables.length === 0 && (
<p className="pl-10 pr-3 py-2 text-sm text-foreground-lighter">
No tables in this schema.
</p>
)}
</CollapsibleContent>
</Collapsible>
)
})}
</div>
<div className="flex items-center justify-between mt-4">
<span className="text-sm text-foreground-lighter">
{selectedCount} table{selectedCount === 1 ? '' : 's'} selected
</span>
<div className="flex items-center gap-2">
{isEditing && (
<Button icon={<ArrowLeft size={14} />} onClick={onBack}>
Back
</Button>
)}
<Button
variant="primary"
disabled={selectedCount === 0}
loading={isSubmitting}
onClick={handleSubmit}
>
{isEditing ? 'Update replicated tables' : 'Enable Warehouse'}
</Button>
</div>
</div>
</div>
)
}
@@ -0,0 +1,298 @@
import { QueryClient } from '@tanstack/react-query'
import { screen, waitFor } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import { platformComponents as components } from 'api-types'
import { mockAnimationsApi } from 'jsdom-testing-mocks'
import { HttpResponse } from 'msw'
import { describe, expect, test, vi } from 'vitest'
import { WarehouseTab } from './WarehouseTab'
import { customRender } from '@/tests/lib/custom-render'
import { addAPIMock, type APIErrorBody } from '@/tests/lib/msw'
type WarehouseSetupStatusResponse = components['schemas']['WarehouseSetupStatusResponse']
type WarehouseCatalogResponse = components['schemas']['WarehouseCatalogResponse']
type UpdateWarehouseCatalogBody = components['schemas']['UpdateWarehouseCatalogBody']
const FDW_STATUS: WarehouseSetupStatusResponse['fdw_status'] = {
extension_available: true,
extension_installed: true,
foreign_schema_imported: true,
schema_created: true,
server_configured: true,
wrapper_installed: true,
}
const mockSetupStatus = (status: Partial<WarehouseSetupStatusResponse>) =>
addAPIMock({
method: 'get',
path: '/platform/warehouse/:ref/setup-status',
response: () =>
HttpResponse.json<WarehouseSetupStatusResponse>({
fdw_status: FDW_STATUS,
setup_status: 'not_started',
steps: [],
tables: [],
...status,
}),
})
// Placeholder host and password, matching the fixtures in `lib/warehouse.test.ts`. A realistic
// `db.<ref>.supabase.co` host with a plausible password reads as a real credential to secret
// scanning.
const CATALOG_PASSWORD = 'pwd'
const CATALOG_URL = 'postgres://postgres:pwd@db.example.supabase.co:5432/postgres'
const mockCatalog = (catalog: WarehouseCatalogResponse) =>
addAPIMock({
method: 'get',
path: '/platform/warehouse/:ref/catalog',
response: () => HttpResponse.json<WarehouseCatalogResponse>(catalog),
})
// The query engine selector is a Radix Select.
mockAnimationsApi()
describe('WarehouseTab', () => {
test('points at the integration and shows no setup UI when Warehouse is not set up', async () => {
mockSetupStatus({ setup_status: 'not_started' })
customRender(<WarehouseTab />)
expect(await screen.findByText('Warehouse is not set up')).toBeInTheDocument()
expect(screen.getByRole('link', { name: 'Choose tables' })).toHaveAttribute(
'href',
'/project/default/integrations/warehouse/overview'
)
// The whole point of the split: Connect never provisions anything.
expect(screen.queryByText('Endpoint')).not.toBeInTheDocument()
expect(screen.queryByRole('button', { name: /Enable Warehouse/i })).not.toBeInTheDocument()
})
test('explains that setup is still running rather than showing empty connection details', async () => {
mockSetupStatus({ setup_status: 'copying' })
customRender(<WarehouseTab />)
expect(await screen.findByText('Warehouse is being set up')).toBeInTheDocument()
expect(screen.getByRole('link', { name: 'View progress' })).toBeInTheDocument()
})
test('points to the integration when setup reports an error', async () => {
mockSetupStatus({ setup_status: 'error' })
customRender(<WarehouseTab />)
expect(await screen.findByText('Warehouse setup failed')).toBeInTheDocument()
expect(
screen.getByText('Review the error and retry setup to get connection details.')
).toBeInTheDocument()
expect(screen.getByRole('link', { name: 'View Warehouse' })).toHaveAttribute(
'href',
'/project/default/integrations/warehouse/overview'
)
})
test('renders connection details and offers catalog access only for DuckDB', async () => {
mockSetupStatus({ setup_status: 'complete' })
mockCatalog({ enabled: false })
const onUrlUpdate = vi.fn()
const { container } = customRender(<WarehouseTab />, {
nuqs: { hasMemory: true, onUrlUpdate },
})
expect(await screen.findByRole('combobox', { name: 'Query engine' })).toBeInTheDocument()
expect(container.querySelector('.border-0.shadow-none')).toBeInTheDocument()
expect(container.querySelector('[data-orientation="horizontal"]')).not.toBeInTheDocument()
expect(screen.queryByRole('heading', { name: 'Connect' })).not.toBeInTheDocument()
expect(screen.getByDisplayValue('default.warehouse.supabase.io')).toBeInTheDocument()
expect(
screen.getByDisplayValue(
'flightsql://postgres:[YOUR-PASSWORD]@default.warehouse.supabase.io:443?tls=enabled'
)
).toBeInTheDocument()
// FlightSQL is the default engine and needs no catalog access, so nothing here provisions.
expect(screen.queryByRole('switch')).not.toBeInTheDocument()
await userEvent.click(screen.getByRole('combobox', { name: 'Query engine' }))
await userEvent.click(screen.getByRole('option', { name: 'DuckDB' }))
expect(
await screen.findByRole('switch', { name: 'Enable DuckDB catalog access' })
).not.toBeChecked()
expect(onUrlUpdate).toHaveBeenLastCalledWith(
expect.objectContaining({ queryString: '?warehouseQueryEngine=duckdb' })
)
})
test('hydrates the selected query engine from the URL', async () => {
mockSetupStatus({ setup_status: 'complete' })
mockCatalog({ enabled: false })
customRender(<WarehouseTab />, {
nuqs: { searchParams: { warehouseQueryEngine: 'duckdb' } },
})
expect(await screen.findByRole('combobox', { name: 'Query engine' })).toHaveTextContent(
'DuckDB'
)
expect(
await screen.findByRole('switch', { name: 'Enable DuckDB catalog access' })
).not.toBeChecked()
})
test('renders the DuckLake attach script when catalog access is on', async () => {
mockSetupStatus({ setup_status: 'complete' })
mockCatalog({
enabled: true,
credentials: {
catalog_url: CATALOG_URL,
data_path: 's3://warehouse/',
metadata_schema: 'ducklake',
s3_access_key_id: 'access-key-id',
s3_endpoint: 'default.storage.supabase.co/storage/v1/s3',
s3_region: 'ap-southeast-1',
s3_secret_access_key: 's3-secret',
},
})
customRender(<WarehouseTab />, { nuqs: { hasMemory: true } })
await userEvent.click(await screen.findByRole('combobox', { name: 'Query engine' }))
await userEvent.click(await screen.findByRole('option', { name: 'DuckDB' }))
expect(
await screen.findByRole('switch', { name: 'Enable DuckDB catalog access' })
).toBeChecked()
expect(await screen.findByText('DUCKLAKE_S3_SECRET=')).toBeInTheDocument()
expect(screen.getByText('DUCKLAKE_METADATA_PASSWORD=')).toBeInTheDocument()
expect(
screen.getByRole('button', { name: 'Copy all DuckLake environment variables' })
).toBeInTheDocument()
expect(screen.getByRole('button', { name: 'Copy prompt' })).toBeInTheDocument()
expect(screen.getByText('Set environment variables')).toBeInTheDocument()
expect(screen.getByText('Attach Warehouse')).toBeInTheDocument()
const stepsSection = screen.getByRole('heading', { name: 'Follow these steps' }).parentElement
?.parentElement
expect(stepsSection).toHaveClass('border-t', 'bg-muted/50')
expect(screen.queryByText('s3-secret')).not.toBeInTheDocument()
await userEvent.click(screen.getByRole('button', { name: 'Reveal DUCKLAKE_S3_SECRET' }))
expect(screen.getByText('s3-secret')).toBeInTheDocument()
expect(screen.queryByText(CATALOG_PASSWORD)).not.toBeInTheDocument()
await userEvent.click(screen.getByRole('button', { name: 'Reveal DUCKLAKE_METADATA_PASSWORD' }))
expect(screen.getByText(CATALOG_PASSWORD)).toBeInTheDocument()
await userEvent.click(screen.getByRole('combobox', { name: 'Query engine' }))
await userEvent.click(screen.getByRole('option', { name: 'FlightSQL' }))
expect(screen.queryByRole('heading', { name: 'Follow these steps' })).not.toBeInTheDocument()
expect(screen.queryByRole('button', { name: 'Copy prompt' })).not.toBeInTheDocument()
})
test('shows progress while updating DuckDB catalog access', async () => {
let catalog: WarehouseCatalogResponse = { enabled: false }
const requestBodies: UpdateWarehouseCatalogBody[] = []
let finishUpdatingCatalog: (() => void) | undefined
const updatingCatalog = new Promise<void>((resolve) => {
finishUpdatingCatalog = resolve
})
mockSetupStatus({ setup_status: 'complete' })
addAPIMock({
method: 'get',
path: '/platform/warehouse/:ref/catalog',
response: () => HttpResponse.json<WarehouseCatalogResponse>(catalog),
})
addAPIMock({
method: 'post',
path: '/platform/warehouse/:ref/catalog',
response: async ({ request }) => {
const body = (await request.json()) as UpdateWarehouseCatalogBody
requestBodies.push(body)
await updatingCatalog
catalog = { enabled: body.enabled }
return HttpResponse.json<WarehouseCatalogResponse>(catalog)
},
})
customRender(<WarehouseTab />, { nuqs: { hasMemory: true } })
await userEvent.click(await screen.findByRole('combobox', { name: 'Query engine' }))
await userEvent.click(screen.getByRole('option', { name: 'DuckDB' }))
const catalogSwitch = await screen.findByRole('switch', {
name: 'Enable DuckDB catalog access',
})
await userEvent.click(catalogSwitch)
expect(
await screen.findByRole('status', { name: 'Updating DuckDB catalog access' })
).toBeInTheDocument()
expect(catalogSwitch).toBeDisabled()
expect(catalogSwitch).toHaveAttribute('aria-busy', 'true')
finishUpdatingCatalog?.()
await waitFor(() => expect(catalogSwitch).toBeChecked())
expect(
screen.queryByRole('status', { name: 'Updating DuckDB catalog access' })
).not.toBeInTheDocument()
expect(requestBodies).toEqual([{ enabled: true }])
})
test('surfaces a 544 status failure without retrying indefinitely and allows retrying', async () => {
let requestCount = 0
addAPIMock({
method: 'get',
path: '/platform/warehouse/:ref/setup-status',
response: () => {
requestCount += 1
if (requestCount === 1) {
return HttpResponse.json<APIErrorBody>(
{ message: 'Warehouse status request timed out' },
{ status: 544 }
)
}
return HttpResponse.json<WarehouseSetupStatusResponse>({
fdw_status: FDW_STATUS,
setup_status: 'not_started',
steps: [],
tables: [],
})
},
})
// AlertError renders a project-scoped link to the AI assistant.
addAPIMock({
method: 'get',
path: '/platform/projects/:ref',
// @ts-expect-error partial project response
response: {
cloud_provider: 'localhost',
id: 1,
inserted_at: '2021-08-02T06:40:40.646Z',
name: 'Default Project',
organization_id: 1,
ref: 'default',
region: 'local',
status: 'ACTIVE_HEALTHY',
},
})
const queryClient = new QueryClient({
defaultOptions: { queries: { retry: 3, retryDelay: 0 } },
})
customRender(<WarehouseTab />, { queryClient })
expect(await screen.findByText('Failed to load Warehouse status')).toBeInTheDocument()
expect(requestCount).toBe(1)
await userEvent.click(screen.getByRole('button', { name: 'Retry' }))
expect(await screen.findByText('Warehouse is not set up')).toBeInTheDocument()
expect(requestCount).toBe(2)
})
})
@@ -0,0 +1,100 @@
import { useParams } from 'common'
import Link from 'next/link'
import { Button } from 'ui'
import { Admonition } from 'ui-patterns/Admonition'
import { GenericSkeletonLoader } from 'ui-patterns/ShimmeringLoader'
import {
isWarehouseProvisioned,
isWarehouseSettingUp,
type WarehouseSetupStatus,
} from '../Integrations/Warehouse/Warehouse.utils'
import { WarehouseConnectionCard } from '../Integrations/Warehouse/WarehouseConnectSection'
import { AlertError } from '@/components/ui/AlertError'
import { checkLocalETLNotSetUp } from '@/data/replication/utils'
import { useWarehouseSetupStatusQuery } from '@/data/warehouse/warehouse-setup-status-query'
const getNotProvisionedContent = (setupStatus?: WarehouseSetupStatus) => {
if (isWarehouseSettingUp(setupStatus)) {
return {
type: 'default' as const,
title: 'Warehouse is being set up',
description: 'Connection details appear here once the first backfill finishes.',
action: 'View progress',
}
}
if (setupStatus === 'error') {
return {
type: 'warning' as const,
title: 'Warehouse setup failed',
description: 'Review the error and retry setup to get connection details.',
action: 'View Warehouse',
}
}
return {
type: 'default' as const,
title: 'Warehouse is not set up',
description: 'Choose which schemas or tables to replicate in order to connect to Warehouse.',
action: 'Choose tables',
}
}
/**
* Read-only view of the Warehouse connection details. Setting Warehouse up, changing which tables
* replicate, and disabling it all live on the Warehouse integration instead.
*/
export const WarehouseTab = () => {
const { ref: projectRef } = useParams()
const { data, isPending, isFetching, isError, error, refetch } = useWarehouseSetupStatusQuery({
projectRef,
})
let content: React.ReactNode
if (isPending) content = <GenericSkeletonLoader />
// Warehouse rides on the replication API, which isn't wired up in local development. Same
// treatment Pipelines gives it, so a local dev doesn't read this as a broken build.
else if (isError && checkLocalETLNotSetUp(error)) {
content = (
<Admonition
type="default"
title="Warehouse is unavailable locally"
description="Configure the replication API to set up Warehouse in local development."
/>
)
} else if (isError) {
content = (
<AlertError
subject="Failed to load Warehouse status"
error={error}
additionalActions={
<Button variant="default" loading={isFetching} onClick={() => refetch()}>
Retry
</Button>
}
/>
)
} else if (!isWarehouseProvisioned(data?.setup_status)) {
const { type, title, description, action } = getNotProvisionedContent(data?.setup_status)
content = (
<Admonition
type={type}
layout="responsive"
title={title}
description={description}
actions={[
<Button key="open-warehouse" asChild variant="default">
<Link href={`/project/${projectRef}/integrations/warehouse/overview`}>{action}</Link>
</Button>,
]}
/>
)
} else {
return <WarehouseConnectionCard variant="sheet" />
}
return <div className="p-8">{content}</div>
}
@@ -14,6 +14,7 @@ const EMPTY_QUERY: ConnectSheetQueryParams = {
method: null,
type: null,
mcpClient: null,
warehouseQueryEngine: null,
}
const EMPTY_PREFS: ConnectSheetPrefs = {}
@@ -173,4 +174,25 @@ describe('resolveConnectSheetHydration', () => {
expect(result.fieldUpdates).toEqual([])
expect(result.urlUpdates).toEqual({})
})
test('hydrates the Warehouse query engine from stored preferences', () => {
const result = resolveConnectSheetHydration(
{ ...EMPTY_QUERY, connectTab: 'warehouse' },
{ warehouseQueryEngine: 'duckdb' },
[...ALL_MODES, 'warehouse']
)
expect(result.mode).toBe('warehouse')
expect(result.urlUpdates.warehouseQueryEngine).toBe('duckdb')
})
test('keeps the Warehouse query engine already present in the URL', () => {
const result = resolveConnectSheetHydration(
{ ...EMPTY_QUERY, connectTab: 'warehouse', warehouseQueryEngine: 'duckdb' },
{ warehouseQueryEngine: 'flightsql' },
[...ALL_MODES, 'warehouse']
)
expect(result.urlUpdates.warehouseQueryEngine).toBeUndefined()
})
})
@@ -11,6 +11,7 @@ export type ConnectSheetPrefs = {
method?: string
type?: string
mcpClient?: string
warehouseQueryEngine?: string
}
const DEFAULT_CONNECT_SHEET_PREFS: ConnectSheetPrefs = {}
@@ -32,6 +33,7 @@ export function useConnectSheetParams() {
method: parseAsString,
type: parseAsString,
mcpClient: parseAsString,
warehouseQueryEngine: parseAsString,
})
const [storedPrefs, setStoredPrefs] = useLocalStorage<ConnectSheetPrefs>(
@@ -1,6 +1,6 @@
import { getEnableWebhooksSQL } from '@supabase/pg-meta'
import type { Tables } from 'common/marketplace.types'
import { Clock5, Code2, Layers, Timer, Vault, Webhook } from 'lucide-react'
import { Clock5, Code2, Layers, Timer, Vault, Warehouse, Webhook } from 'lucide-react'
import dynamic from 'next/dynamic'
import Image from 'next/image'
import { ComponentType, ReactNode } from 'react'
@@ -413,6 +413,41 @@ const SUPABASE_INTEGRATIONS: Array<IntegrationDefinition> = [
return null
},
},
{
id: 'warehouse',
type: 'custom' as const,
source: 'Official' as const,
status: 'alpha' as const,
requiredExtensions: [],
name: `Warehouse`,
icon: ({ className, ...props } = {}) => (
<Warehouse className={cn('inset-0 p-2 text-black w-full h-full', className)} {...props} />
),
description: 'Replicate your database to an analytical endpoint',
// No Warehouse guide exists yet. Point this at `${DOCS_URL}/guides/warehouse` once one does.
docsUrl: null,
author: authorSupabase,
navigation: [
{
route: 'overview',
label: 'Overview',
},
],
navigate: ({ pageId = 'overview' }) => {
if (pageId === 'overview') {
return dynamic(
() =>
import('@/components/interfaces/Integrations/Warehouse/OverviewTab').then(
(mod) => mod.WarehouseOverviewTab
),
{
loading: Loading,
}
)
}
return null
},
},
{
id: 'graphiql',
type: 'postgres_extension' as const,
@@ -651,6 +686,7 @@ const INTEGRATIONS_WITH_CATEGORIES = [
vault: ['security'],
webhooks: ['api'],
data_api: ['api', 'data-platform'],
warehouse: ['data-platform'],
graphiql: ['api', 'devtools'],
}
return {
@@ -20,6 +20,7 @@ import {
} from '@/data/marketplace/integrations-query'
import { useCLIReleaseVersionQuery } from '@/data/misc/cli-release-version-query'
import { useIsFeatureEnabled } from '@/hooks/misc/useIsFeatureEnabled'
import { useIsWarehouseEnabled } from '@/hooks/misc/useIsWarehouseEnabled'
const renderMarketplaceLogo = (listingLogo?: string | null) => {
const MarketplaceLogo = ({ className, ...props }: { className?: string } = {}) => (
@@ -92,6 +93,7 @@ const useMarketplaceListings = () => {
*/
export const useAvailableIntegrations = () => {
const { integrationsWrappers } = useIsFeatureEnabled(['integrations:wrappers'])
const isWarehouseAvailable = useIsWarehouseEnabled()
const { data: cliData } = useCLIReleaseVersionQuery()
const isCLI = !!cliData?.current
@@ -106,7 +108,7 @@ export const useAvailableIntegrations = () => {
// [Joshen] Format marketplace integrations into existing ones for now
// Likely that we might need to change, but can look into separately
// Wrappers from marketplace are excluded here — they are merged into the
// Wrappers from marketplace are excluded here. They are merged into the
// hardcoded studio wrappers below as content overrides.
const marketplaceIntegrations: IntegrationDefinition[] = useMemo(
() =>
@@ -228,6 +230,13 @@ export const useAvailableIntegrations = () => {
return false
}
// Warehouse is gated to an allow-list of orgs, and the API rejects every
// `/platform/warehouse/{ref}/*` call for the rest, so hide it entirely instead of showing an
// integration whose every request would fail.
if (integration.id === 'warehouse' && !isWarehouseAvailable) {
return false
}
return true
}).map((integration) => {
const isWrapper = integration.type === 'wrapper'
@@ -266,7 +275,7 @@ export const useAvailableIntegrations = () => {
...Object.fromEntries(Object.entries(overrides).filter(([, v]) => v != null)),
}
})
}, [integrationsWrappers, isCLI, marketplaceWrappers])
}, [integrationsWrappers, isCLI, isWarehouseAvailable, marketplaceWrappers])
const dataWithMarketplace = useMemo(() => {
return [...marketplaceIntegrations, ...allIntegrations].sort((a, b) =>
@@ -90,6 +90,10 @@ export const useInstalledIntegrations = () => {
.filter((integration) => {
if (integration.id === 'webhooks') return isHooksEnabled
if (integration.id === 'data_api') return true
// Availability is already gated by the org allow-list in `useAvailableIntegrations`, and
// whether Warehouse is set up on this project is handled inside its tabs, following the same split
// Data API uses.
if (integration.id === 'warehouse') return true
if (integration.id === 'stripe_sync_engine') {
return isStripeSyncEngineInstalled(schemas)
}
@@ -0,0 +1,224 @@
import { screen } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import { platformComponents as components } from 'api-types'
import dayjs from 'dayjs'
import duration from 'dayjs/plugin/duration'
import { mockAnimationsApi } from 'jsdom-testing-mocks'
import { HttpResponse } from 'msw'
import { beforeEach, describe, expect, test, vi } from 'vitest'
import { WarehouseOverviewTab } from './OverviewTab'
import { customRender } from '@/tests/lib/custom-render'
import { addAPIMock, type APIErrorBody } from '@/tests/lib/msw'
type WarehouseSetupStatusResponse = components['schemas']['WarehouseSetupStatusResponse']
// Both integration shells are live, and the flag reads a context plus ConfigCat that
// `customRender` doesn't provide.
const mockIsMarketplaceEnabled = vi.fn(() => false)
vi.mock('@/components/interfaces/App/FeaturePreview/FeaturePreviewContext', () => ({
useIsMarketplaceEnabled: () => mockIsMarketplaceEnabled(),
}))
vi.mock('../Integration/IntegrationOverviewTab', () => ({
IntegrationOverviewTab: ({ children }: { children: React.ReactNode }) => <div>{children}</div>,
}))
// Exercised by its own unit tests, and it fires four upstream queries of its own.
vi.mock('./WarehouseSchemaTablePicker', () => ({
WarehouseSchemaTablePicker: ({ error }: { error?: { message: string } | null }) => (
<section>
<h2>Tables</h2>
<span>Replicated tables picker</span>
{!!error && <span>Picker error: {error.message}</span>}
</section>
),
}))
mockAnimationsApi()
dayjs.extend(duration)
const FDW_STATUS: WarehouseSetupStatusResponse['fdw_status'] = {
extension_available: true,
extension_installed: true,
foreign_schema_imported: true,
schema_created: true,
server_configured: true,
wrapper_installed: true,
}
const mockSetupStatus = (status: Partial<WarehouseSetupStatusResponse>) =>
addAPIMock({
method: 'get',
path: '/platform/warehouse/:ref/setup-status',
response: () =>
HttpResponse.json<WarehouseSetupStatusResponse>({
fdw_status: FDW_STATUS,
setup_status: 'not_started',
steps: [],
tables: [],
...status,
}),
})
const mockProject = () =>
addAPIMock({
method: 'get',
path: '/platform/projects/:ref',
// @ts-expect-error partial project response used by AlertError
response: {
cloud_provider: 'localhost',
id: 1,
inserted_at: '2021-08-02T06:40:40.646Z',
name: 'Default Project',
organization_id: 1,
ref: 'default',
region: 'local',
status: 'ACTIVE_HEALTHY',
},
})
describe('WarehouseOverviewTab', () => {
beforeEach(() => {
mockIsMarketplaceEnabled.mockReturnValue(true)
})
test.each([false, true])(
'offers only the picker before setup (marketplace: %s)',
async (isMarketplaceEnabled) => {
mockIsMarketplaceEnabled.mockReturnValue(isMarketplaceEnabled)
mockSetupStatus({ setup_status: 'not_started' })
customRender(<WarehouseOverviewTab />)
expect(await screen.findByText('Replicated tables picker')).toBeInTheDocument()
expect(screen.queryByText('External access')).not.toBeInTheDocument()
expect(screen.queryByRole('button', { name: 'Disable Warehouse' })).not.toBeInTheDocument()
}
)
test.each(['setting_up', 'copying'] as const)(
'shows setup progress for %s',
async (setupStatus) => {
mockSetupStatus({
setup_status: setupStatus,
tables:
setupStatus === 'copying'
? [
{ schema: 'public', name: 'orders', copy_name: 'public.orders', state: 'live' },
{
schema: 'public',
name: 'customers',
copy_name: 'public.customers',
state: 'syncing',
},
]
: [],
})
customRender(<WarehouseOverviewTab />)
expect(await screen.findByText('Warehouse is being set up')).toBeInTheDocument()
expect(screen.getByRole('heading', { name: 'Status' })).toBeInTheDocument()
if (setupStatus === 'copying') {
expect(
screen.getByText('Backfilling selected tables. 1 of 2 tables synced.')
).toBeInTheDocument()
expect(screen.getByText('orders')).toBeInTheDocument()
expect(screen.getByText('customers')).toBeInTheDocument()
}
}
)
test('shows the setup failure and retry action', async () => {
mockProject()
mockSetupStatus({
setup_status: 'error',
steps: [{ name: 'warehouse_copy', status: 'error', message: 'Failed to copy public.orders' }],
tables: [{ schema: 'public', name: 'orders', copy_name: 'public.orders', state: 'error' }],
})
customRender(<WarehouseOverviewTab />)
expect(await screen.findByText('Warehouse setup failed')).toBeInTheDocument()
expect(screen.getByText('Failed to copy public.orders')).toBeInTheDocument()
expect(screen.getByRole('button', { name: 'Retry' })).toBeInTheDocument()
})
test('shows the latest error when retrying setup fails', async () => {
mockProject()
mockSetupStatus({
setup_status: 'error',
steps: [{ name: 'warehouse_copy', status: 'error', message: 'Initial setup failed' }],
tables: [{ schema: 'public', name: 'orders', copy_name: 'public.orders', state: 'error' }],
})
addAPIMock({
method: 'post',
path: '/platform/warehouse/:ref/setup',
response: () =>
HttpResponse.json<APIErrorBody>({ message: 'Retry could not be started' }, { status: 500 }),
})
customRender(<WarehouseOverviewTab />)
await userEvent.click(await screen.findByRole('button', { name: 'Retry' }))
expect(await screen.findByText('Retry could not be started')).toBeInTheDocument()
expect(screen.queryByText('Initial setup failed')).not.toBeInTheDocument()
})
test('shows Status, Tables, then Connect once setup is complete', async () => {
mockSetupStatus({
setup_status: 'complete',
tables: [
{
schema: 'public',
name: 'orders',
copy_name: 'public.orders',
state: 'live',
lag_ms: 4000,
},
{
schema: 'public',
name: 'customers',
copy_name: 'public.customers',
state: 'syncing',
lag_ms: 18000,
},
{ schema: 'analytics', name: 'events', copy_name: 'analytics.events', state: 'error' },
],
})
customRender(<WarehouseOverviewTab />)
// findByRole throws on duplicates, so this also guards the section titles staying distinct.
for (const name of ['Status', 'Tables', 'Connect']) {
expect(await screen.findByRole('heading', { name })).toBeInTheDocument()
}
expect(screen.getByText('Replicated tables picker')).toBeInTheDocument()
expect(screen.getByText('Synced')).toBeInTheDocument()
expect(screen.getByText('Backfilling')).toBeInTheDocument()
expect(screen.getByText('Error')).toBeInTheDocument()
const headings = screen
.getAllByRole('heading')
.map((heading) => heading.textContent)
.filter((heading) => ['Status', 'Tables', 'Connect'].includes(heading ?? ''))
expect(headings).toEqual(['Status', 'Tables', 'Connect'])
expect(screen.queryByRole('button', { name: 'Disable Warehouse' })).not.toBeInTheDocument()
})
test('shows a status query failure without blocking an unrelated route', async () => {
addAPIMock({
method: 'get',
path: '/platform/warehouse/:ref/setup-status',
response: () => HttpResponse.json<APIErrorBody>({ message: 'Boom' }, { status: 500 }),
})
mockProject()
customRender(<WarehouseOverviewTab />)
expect(await screen.findByText('Failed to load Warehouse status')).toBeInTheDocument()
})
})
@@ -0,0 +1,32 @@
import { useParams } from 'common'
import { Badge } from 'ui'
import { IntegrationOverviewTab } from '../Integration/IntegrationOverviewTab'
import { isWarehouseProvisioned } from './Warehouse.utils'
import { WarehouseSetupPanel } from './WarehouseSetupPanel'
import { useIsMarketplaceEnabled } from '@/components/interfaces/App/FeaturePreview/FeaturePreviewContext'
import { useWarehouseSetupStatusQuery } from '@/data/warehouse/warehouse-setup-status-query'
export const WarehouseOverviewTab = () => {
const { ref } = useParams()
const isMarketplaceEnabled = useIsMarketplaceEnabled()
const { data: isProvisioned } = useWarehouseSetupStatusQuery(
{ projectRef: ref },
{ select: (data) => isWarehouseProvisioned(data?.setup_status) }
)
// The marketplace shell supplies its own padding, so returning the content bare avoids
// double-padding it.
if (isMarketplaceEnabled) return <WarehouseSetupPanel />
return (
<IntegrationOverviewTab
status={isProvisioned ? <Badge variant="success">Enabled</Badge> : undefined}
>
<div className="px-4 md:px-10 max-w-4xl space-y-4">
<WarehouseSetupPanel />
</div>
</IntegrationOverviewTab>
)
}
@@ -6,10 +6,11 @@ import {
getSchemaCheckedState,
getSchemaTableKey,
getSelectedTableCount,
hasSelectionChanged,
isSelectableWarehouseSchema,
type SchemaTableSelection,
type SchemaWithTables,
} from '../WarehouseModePanel/WarehouseModePanel.utils'
} from './Warehouse.utils'
import { WAREHOUSE_METADATA_SCHEMA } from '@/lib/warehouse'
describe('WarehouseModePanel.utils:isSelectableWarehouseSchema', () => {
@@ -143,6 +144,36 @@ describe('WarehouseModePanel.utils:getSelectedTableCount', () => {
})
})
describe('WarehouseModePanel.utils:hasSelectionChanged', () => {
test('is false for two empty selections', () => {
expect(hasSelectionChanged({}, {})).toBe(false)
})
test('is false when the same tables are selected', () => {
const initial: SchemaTableSelection = { 'public.orders': true, 'public.customers': false }
const selection: SchemaTableSelection = { 'public.orders': true }
expect(hasSelectionChanged(selection, initial)).toBe(false)
})
test('is true when a table is added on top of the initial selection', () => {
const initial: SchemaTableSelection = { 'public.orders': true }
const selection: SchemaTableSelection = { 'public.orders': true, 'public.customers': true }
expect(hasSelectionChanged(selection, initial)).toBe(true)
})
test('is true when a table is removed from the initial selection', () => {
const initial: SchemaTableSelection = { 'public.orders': true, 'public.customers': true }
const selection: SchemaTableSelection = { 'public.orders': true, 'public.customers': false }
expect(hasSelectionChanged(selection, initial)).toBe(true)
})
test('ignores explicit false entries that mirror an absent key', () => {
const initial: SchemaTableSelection = {}
const selection: SchemaTableSelection = { 'public.orders': false }
expect(hasSelectionChanged(selection, initial)).toBe(false)
})
})
describe('WarehouseModePanel.utils:buildWarehouseSetupTargets', () => {
test('returns an empty array for an empty selection', () => {
const schemas: SchemaWithTables[] = [{ schema: 'public', tables: ['orders', 'customers'] }]
@@ -5,6 +5,18 @@ import { WAREHOUSE_METADATA_SCHEMA } from '@/lib/warehouse'
export type WarehouseSetupBody = components['schemas']['WarehouseSetupBody']
export type WarehouseSetupTarget = WarehouseSetupBody['targets'][number]
export type WarehouseSetupStatus =
components['schemas']['WarehouseSetupStatusResponse']['setup_status']
export type WarehouseSetupTable =
components['schemas']['WarehouseSetupStatusResponse']['tables'][number]
export function isWarehouseProvisioned(setupStatus?: WarehouseSetupStatus): boolean {
return setupStatus === 'complete'
}
export function isWarehouseSettingUp(setupStatus?: WarehouseSetupStatus): boolean {
return setupStatus === 'setting_up' || setupStatus === 'copying'
}
/** Selection map keyed by `${schema}.${table}`. */
export type SchemaTableSelection = Record<string, boolean>
@@ -17,8 +29,8 @@ export function getSchemaTableKey(schema: string, table: string): string {
/**
* Internal schemas that still hold product data users legitimately want in their warehouse.
* Everything else in `INTERNAL_SCHEMAS` is Supabase infrastructure — `vault` (secrets),
* `pgsodium`, `cron`/`pgmq` bookkeeping, migration history — which should never be offered as a
* Everything else in `INTERNAL_SCHEMAS` is Supabase infrastructure: `vault` (secrets),
* `pgsodium`, `cron`/`pgmq` bookkeeping, and migration history. These should never be offered as a
* replication target.
*/
const REPLICABLE_INTERNAL_SCHEMAS = ['auth', 'storage']
@@ -32,7 +44,7 @@ const NON_SELECTABLE_SCHEMAS = new Set(
*
* `WAREHOUSE_METADATA_SCHEMA` is excluded on top of the infrastructure schemas above: it holds the
* DuckLake catalog describing the Warehouse itself, so replicating it would feed every Warehouse
* write back in as more catalog rows to replicate. The platform rejects it server-side too — this
* write back in as more catalog rows to replicate. The platform rejects it server-side too. This
* just keeps it out of the picker so the user never picks a target that can only fail.
*/
export function isSelectableWarehouseSchema(schemaName: string): boolean {
@@ -47,6 +59,22 @@ export function getSelectedTableCount(selection: SchemaTableSelection): number {
return Object.values(selection).filter(Boolean).length
}
/**
* Compares two selections by their effective selected keys rather than by reference or raw
* object shape, since `toggleTable`/`toggleSchema` can write explicit `false` entries that
* shouldn't count as a difference from a key simply being absent.
*/
export function hasSelectionChanged(
selection: SchemaTableSelection,
initialSelection: SchemaTableSelection
): boolean {
const currentKeys = Object.keys(selection).filter((key) => selection[key])
const initialKeys = Object.keys(initialSelection).filter((key) => initialSelection[key])
if (currentKeys.length !== initialKeys.length) return true
const initialKeySet = new Set(initialKeys)
return currentKeys.some((key) => !initialKeySet.has(key))
}
/**
* Seeds the picker's selection from the tables already in the `supabase_warehouse` publication, so
* editing an existing setup starts from what's actually replicated instead of an empty selection.
@@ -108,6 +136,12 @@ export function buildWarehouseSetupTargets(
return targets
}
export function buildRetryTargets(
tables: Pick<WarehouseSetupTable, 'schema' | 'name'>[] = []
): WarehouseSetupTarget[] {
return tables.map((table) => ({ type: 'table', schema: table.schema, name: table.name }))
}
export type WarehouseCatalogCredentials = NonNullable<
components['schemas']['WarehouseCatalogResponse']['credentials']
>
@@ -4,13 +4,16 @@ import Link from 'next/link'
import { useRef, useState } from 'react'
import { toast } from 'sonner'
import {
Badge,
Button,
Card,
CardContent,
cn,
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
Separator,
Switch,
Tooltip,
TooltipContent,
@@ -20,12 +23,21 @@ import { Admonition } from 'ui-patterns/Admonition'
import { CodeBlock } from 'ui-patterns/CodeBlock'
import { Input } from 'ui-patterns/DataInputs/Input'
import { FormLayout } from 'ui-patterns/form/Layout/FormLayout'
import {
PageSection,
PageSectionContent,
PageSectionDescription,
PageSectionMeta,
PageSectionSummary,
PageSectionTitle,
} from 'ui-patterns/PageSection'
import { GenericSkeletonLoader } from 'ui-patterns/ShimmeringLoader'
import { ConnectSheetStep } from '../ConnectSheetStep'
import { EnvRow } from '../content/server/common/EnvRow'
import { CopyPromptButton } from '../CopyPromptAdmonition'
import type { WarehouseCatalogCredentials } from './WarehouseModePanel.utils'
import { ConnectSheetStep } from '../../ConnectSheet/ConnectSheetStep'
import { EnvRow } from '../../ConnectSheet/content/server/common/EnvRow'
import { CopyPromptButton } from '../../ConnectSheet/CopyPromptAdmonition'
import { useConnectSheetParams } from '../../ConnectSheet/useConnectSheetParams'
import type { WarehouseCatalogCredentials } from './Warehouse.utils'
import { AlertError } from '@/components/ui/AlertError'
import CopyButton from '@/components/ui/CopyButton'
import { useUpdateWarehouseCatalogMutation } from '@/data/warehouse/warehouse-catalog-mutation'
@@ -40,6 +52,10 @@ import {
parseWarehouseCatalogUrl,
} from '@/lib/warehouse'
/**
* Only engines we can produce real configuration for. DuckLake exposes a Postgres catalog over S3
* rather than an Iceberg REST catalog, so Trino, Spark and PyIceberg need API support first.
*/
const QUERY_ENGINES = [
{ value: 'flightsql', label: 'FlightSQL' },
{ value: 'duckdb', label: 'DuckDB' },
@@ -47,10 +63,6 @@ const QUERY_ENGINES = [
type QueryEngine = (typeof QUERY_ENGINES)[number]['value']
export interface WarehouseConnectionDetailsProps {
onEditTables: () => void
}
function FieldRow({
id,
label,
@@ -68,7 +80,7 @@ function FieldRow({
}
const FlightSqlContent = ({ projectRef }: { projectRef: string }) => (
<div className="space-y-4">
<CardContent className="space-y-4">
<FieldRow id="warehouse-flightsql-endpoint" label="Endpoint">
<Input
id="warehouse-flightsql-endpoint"
@@ -106,7 +118,7 @@ const FlightSqlContent = ({ projectRef }: { projectRef: string }) => (
value={getWarehouseUsqlCommand(projectRef)}
/>
</FieldRow>
</div>
</CardContent>
)
const DuckLakeSecretRow = ({ name, value }: { name: string; value: string }) => {
@@ -134,6 +146,10 @@ const DuckLakeSecretRow = ({ name, value }: { name: string; value: string }) =>
)
}
/**
* The DuckDB setup script inlines everything except the two passwords, which it reads via
* `getenv()` -- so those are the only credential values surfaced as their own rows here.
*/
const DuckLakeEnvironmentVariables = ({
credentials,
password,
@@ -169,13 +185,19 @@ const DuckLakeEnvironmentVariables = ({
)
}
const DuckLakeSetup = ({ credentials }: { credentials: WarehouseCatalogCredentials }) => {
const DuckLakeSetup = ({
credentials,
variant,
}: {
credentials: WarehouseCatalogCredentials
variant: 'default' | 'sheet'
}) => {
const connection = parseWarehouseCatalogUrl(credentials.catalog_url)
const stepsContainerRef = useRef<HTMLDivElement | null>(null)
if (connection === null) {
return (
<div className="space-y-4 border-t bg-muted/50 p-8">
const errorContent = (
<>
<Admonition
type="warning"
title="Could not read the catalog connection details"
@@ -191,7 +213,31 @@ const DuckLakeSetup = ({ credentials }: { credentials: WarehouseCatalogCredentia
value={credentials.catalog_url}
/>
</FieldRow>
</div>
</>
)
return variant === 'sheet' ? (
<div className="space-y-4 border-t bg-muted/50 p-8">{errorContent}</div>
) : (
<CardContent className="space-y-4">{errorContent}</CardContent>
)
}
const script = (
<CodeBlock
className="[&_code]:text-foreground"
language="sql"
hideLineNumbers
value={getDuckLakeSetupScript({ credentials, connection })}
/>
)
if (variant === 'default') {
return (
<CardContent className="space-y-4">
<DuckLakeEnvironmentVariables credentials={credentials} password={connection.password} />
{script}
</CardContent>
)
}
@@ -214,12 +260,7 @@ const DuckLakeSetup = ({ credentials }: { credentials: WarehouseCatalogCredentia
title="Attach Warehouse"
description="Run this script in DuckDB to configure the secrets and attach Warehouse."
>
<CodeBlock
className="[&_code]:text-foreground"
language="sql"
hideLineNumbers
value={getDuckLakeSetupScript({ credentials, connection })}
/>
{script}
</ConnectSheetStep>
</div>
</div>
@@ -241,32 +282,48 @@ const CatalogAccessToggle = ({
})
return (
<FormLayout
layout="flex-row-reverse"
label="Enable DuckDB catalog access"
description="Creates the credentials DuckDB needs to attach Warehouse. Not required for FlightSQL."
>
<div className="flex items-center justify-end gap-2">
{catalogMutation.isPending && (
<span role="status" aria-label="Updating DuckDB catalog access">
<Loader2 size={16} className="animate-spin text-foreground-muted" aria-hidden />
<CardContent>
<FormLayout
layout="flex-row-reverse"
label="Enable DuckDB catalog access"
description="Creates the credentials DuckDB needs to attach Warehouse. Not required for FlightSQL."
>
<div className="flex items-center justify-end gap-2">
<span
role="status"
aria-label={catalogMutation.isPending ? 'Updating DuckDB catalog access' : undefined}
className="sr-only"
>
{catalogMutation.isPending ? 'Updating DuckDB catalog access' : ''}
</span>
)}
<Switch
aria-label="Enable DuckDB catalog access"
aria-busy={catalogMutation.isPending}
checked={isEnabled}
disabled={catalogMutation.isPending}
onCheckedChange={(enabled) => catalogMutation.mutate({ projectRef, body: { enabled } })}
/>
</div>
</FormLayout>
{catalogMutation.isPending && (
<Loader2
size={16}
className="animate-spin text-foreground-muted motion-reduce:animate-none"
aria-hidden
/>
)}
<Switch
aria-label="Enable DuckDB catalog access"
aria-busy={catalogMutation.isPending}
checked={isEnabled}
disabled={catalogMutation.isPending}
onCheckedChange={(enabled) => catalogMutation.mutate({ projectRef, body: { enabled } })}
/>
</div>
</FormLayout>
</CardContent>
)
}
export const WarehouseConnectionDetails = ({ onEditTables }: WarehouseConnectionDetailsProps) => {
interface WarehouseConnectionCardProps {
variant?: 'default' | 'sheet'
}
export const WarehouseConnectionCard = ({ variant = 'default' }: WarehouseConnectionCardProps) => {
const { ref: projectRef } = useParams()
const [engine, setEngine] = useState<QueryEngine>('flightsql')
const { params, setConnectParams } = useConnectSheetParams()
const engine: QueryEngine = params.warehouseQueryEngine === 'duckdb' ? 'duckdb' : 'flightsql'
const {
data: catalog,
@@ -283,18 +340,23 @@ export const WarehouseConnectionDetails = ({ onEditTables }: WarehouseConnection
catalogStatus = 'DuckDB catalog access loaded'
}
return (
<div>
<div className="space-y-4 p-8">
<div className="flex items-center gap-3">
<Badge variant="success">Warehouse enabled</Badge>
<Button className="ml-auto" variant="text" size="tiny" onClick={onEditTables}>
Edit replicated tables
</Button>
</div>
const isSheet = variant === 'sheet'
const connectionCard = (
<Card
className={cn(
isSheet &&
'relative space-y-4 rounded-none border-0 bg-transparent shadow-none [&>div]:border-0 [&>div]:p-0'
)}
>
<CardContent className="border-none">
<FieldRow id="warehouse-query-engine" label="Query engine">
<Select value={engine} onValueChange={(value) => setEngine(value as QueryEngine)}>
<Select
value={engine}
onValueChange={(value) =>
setConnectParams({ warehouseQueryEngine: value as QueryEngine })
}
>
<SelectTrigger id="warehouse-query-engine" className="ml-auto w-48">
<SelectValue />
</SelectTrigger>
@@ -307,26 +369,62 @@ export const WarehouseConnectionDetails = ({ onEditTables }: WarehouseConnection
</SelectContent>
</Select>
</FieldRow>
</CardContent>
{engine === 'flightsql' && <FlightSqlContent projectRef={projectRef} />}
{!isSheet && <Separator />}
{engine === 'duckdb' && (
<>
{isCatalogPending && <GenericSkeletonLoader />}
{isCatalogError && (
{engine === 'flightsql' && <FlightSqlContent projectRef={projectRef} />}
{engine === 'duckdb' && (
<>
{isCatalogPending && (
<CardContent>
<GenericSkeletonLoader />
</CardContent>
)}
{isCatalogError && (
<CardContent>
<AlertError subject="Failed to load DuckLake catalog access" error={catalogError} />
)}
{catalog && <CatalogAccessToggle projectRef={projectRef} isEnabled={catalog.enabled} />}
</>
)}
<span className="sr-only" role="status" aria-live="polite">
{catalogStatus}
</span>
</div>
</CardContent>
)}
{catalog && <CatalogAccessToggle projectRef={projectRef} isEnabled={catalog.enabled} />}
{!isSheet && catalog?.enabled && catalog.credentials && (
<DuckLakeSetup credentials={catalog.credentials} variant="default" />
)}
</>
)}
<span className="sr-only" role="status" aria-live="polite">
{catalogStatus}
</span>
</Card>
)
if (!isSheet) return connectionCard
return (
<div>
<div className="p-8">{connectionCard}</div>
{engine === 'duckdb' && catalog?.enabled && catalog.credentials && (
<DuckLakeSetup credentials={catalog.credentials} />
<DuckLakeSetup credentials={catalog.credentials} variant="sheet" />
)}
</div>
)
}
export const WarehouseConnectSection = () => {
return (
<PageSection className="pt-0!">
<PageSectionMeta>
<PageSectionSummary>
<PageSectionTitle>Connect</PageSectionTitle>
<PageSectionDescription>
Point an analytical tool at Warehouse without querying your primary database.
</PageSectionDescription>
</PageSectionSummary>
</PageSectionMeta>
<PageSectionContent>
<WarehouseConnectionCard />
</PageSectionContent>
</PageSection>
)
}
@@ -0,0 +1,318 @@
import { useParams } from 'common'
import { ChevronRight } from 'lucide-react'
import { useMemo, useState } from 'react'
import {
Button,
Card,
CardContent,
CardFooter,
Checkbox,
Collapsible,
CollapsibleContent,
CollapsibleTrigger,
} from 'ui'
import {
PageSection,
PageSectionContent,
PageSectionDescription,
PageSectionMeta,
PageSectionSummary,
PageSectionTitle,
} from 'ui-patterns/PageSection'
import { GenericSkeletonLoader } from 'ui-patterns/ShimmeringLoader'
import {
buildSelectionFromPublicationTables,
buildWarehouseSetupTargets,
getSchemaCheckedState,
getSchemaTableKey,
getSelectedTableCount,
hasSelectionChanged,
isSelectableWarehouseSchema,
type SchemaTableSelection,
type SchemaWithTables,
type WarehouseSetupTarget,
} from './Warehouse.utils'
import { AlertError } from '@/components/ui/AlertError'
import { useSchemasQuery } from '@/data/database/schemas-query'
import { useReplicationPublicationQuery } from '@/data/replication/publication-query'
import { useReplicationSourcesQuery } from '@/data/replication/sources-query'
import { useTablesQuery } from '@/data/tables/tables-query'
import { useSelectedProjectQuery } from '@/hooks/misc/useSelectedProject'
import { WAREHOUSE_PUBLICATION_NAME } from '@/lib/warehouse'
export interface WarehouseSchemaTablePickerProps {
onSubmit: (targets: WarehouseSetupTarget[]) => void
isSubmitting: boolean
isEditing?: boolean
error?: { message: string } | null
}
export const WarehouseSchemaTablePicker = ({
onSubmit,
isSubmitting,
isEditing = false,
error,
}: WarehouseSchemaTablePickerProps) => {
const { ref: projectRef } = useParams()
const { data: project } = useSelectedProjectQuery()
// `null` until the user touches a checkbox, so the selection seeded from the existing
// publication can arrive asynchronously without an effect syncing it into state.
const [selectionOverride, setSelectionOverride] = useState<SchemaTableSelection | null>(null)
const [expandedOverrides, setExpandedOverrides] = useState<Record<string, boolean>>({})
const {
data: schemas,
isPending: isSchemasPending,
isError: isSchemasError,
error: schemasError,
} = useSchemasQuery({ projectRef, connectionString: project?.connectionString })
const {
data: tables,
isPending: isTablesPending,
isError: isTablesError,
error: tablesError,
} = useTablesQuery({ projectRef, connectionString: project?.connectionString })
// The `supabase_warehouse` publication is the source of truth for what's currently replicated.
// Reading the sources query directly (rather than via useReplicationSourceId) to get its
// loading state: the publications query stays disabled until a source id exists, so without it
// the list would render un-checked and then flash back to a loader once publications kick in.
const {
data: sourcesData,
isLoading: isSourcesLoading,
isError: isSourcesError,
error: sourcesError,
} = useReplicationSourcesQuery({ projectRef })
const sourceId = sourcesData?.sources.find((source) => source.name === projectRef)?.id
const {
data: publication,
isError: isPublicationsError,
error: publicationsError,
} = useReplicationPublicationQuery({
projectRef,
sourceId,
publicationName: WAREHOUSE_PUBLICATION_NAME,
})
// Derived from data presence rather than fetch status, so there's no render gap between the
// publication query becoming enabled and it actually starting to fetch.
const isSelectionPending =
isSourcesLoading ||
(sourceId !== undefined && publication === undefined && !isPublicationsError)
const initialSelection = useMemo(
() => buildSelectionFromPublicationTables(publication?.tables ?? []),
[publication]
)
const selection = selectionOverride ?? initialSelection
const schemasWithTables: SchemaWithTables[] = useMemo(() => {
if (!schemas || !tables) return []
return schemas
.filter((schema) => isSelectableWarehouseSchema(schema.name))
.map((schema) => ({
schema: schema.name,
tables: tables.filter((table) => table.schema === schema.name).map((table) => table.name),
}))
.sort((a, b) => a.schema.localeCompare(b.schema))
}, [schemas, tables])
const selectedCount = getSelectedTableCount(selection)
const hasChanges = hasSelectionChanged(selection, initialSelection)
const updateSelection = (updater: (current: SchemaTableSelection) => SchemaTableSelection) => {
setSelectionOverride((prev) => updater(prev ?? initialSelection))
}
const toggleTable = (schema: string, table: string) => {
const key = getSchemaTableKey(schema, table)
if (isEditing && initialSelection[key]) return
updateSelection((current) => ({ ...current, [key]: !current[key] }))
}
const toggleSchema = (schema: SchemaWithTables) => {
const selectableTables = schema.tables.filter(
(table) => !isEditing || !initialSelection[getSchemaTableKey(schema.schema, table)]
)
const areAllSelectableTablesSelected =
selectableTables.length > 0 &&
selectableTables.every((table) => selection[getSchemaTableKey(schema.schema, table)])
updateSelection((current) => {
const next = { ...current }
selectableTables.forEach((table) => {
next[getSchemaTableKey(schema.schema, table)] = !areAllSelectableTablesSelected
})
return next
})
}
const setExpanded = (schemaName: string, isOpen: boolean) => {
setExpandedOverrides((prev) => ({ ...prev, [schemaName]: isOpen }))
}
const handleSubmit = () => {
const targets = buildWarehouseSetupTargets(selection, schemasWithTables)
if (targets.length === 0) return
onSubmit(targets)
}
// Waiting on the publication too, so the pre-checked selection is in place before the user can
// start toggling (an early toggle would otherwise pin an override that omits existing tables).
if (isSchemasPending || isTablesPending || isSelectionPending) {
return (
<PageSection className="pt-0!">
<PageSectionMeta>
<PageSectionSummary>
<PageSectionTitle>Tables</PageSectionTitle>
<PageSectionDescription>
{isEditing
? 'Add more schemas or tables to Warehouse.'
: 'Choose which schemas or tables to replicate.'}
</PageSectionDescription>
</PageSectionSummary>
</PageSectionMeta>
<PageSectionContent>
<Card>
<CardContent>
<GenericSkeletonLoader />
</CardContent>
</Card>
</PageSectionContent>
</PageSection>
)
}
if (isSchemasError) return <AlertError subject="Failed to load schemas" error={schemasError} />
if (isTablesError) return <AlertError subject="Failed to load tables" error={tablesError} />
if (isEditing && isSourcesError) {
return <AlertError subject="Failed to load replicated tables" error={sourcesError} />
}
// Only blocking when editing: a first-time setup starts from an empty selection anyway, so a
// failed publication lookup shouldn't stop the user from enabling Warehouse at all.
if (isEditing && isPublicationsError) {
return <AlertError subject="Failed to load replicated tables" error={publicationsError} />
}
return (
<PageSection className="pt-0!">
<PageSectionMeta>
<PageSectionSummary>
<PageSectionTitle>Tables</PageSectionTitle>
<PageSectionDescription>
{isEditing
? 'Add more schemas or tables to Warehouse. Tables already replicating cannot be removed yet.'
: 'Choose which schemas or tables to replicate.'}
</PageSectionDescription>
</PageSectionSummary>
</PageSectionMeta>
<PageSectionContent className="space-y-4">
{!!error && (
<AlertError
subject={isEditing ? 'Failed to add replicated tables' : 'Failed to enable Warehouse'}
error={error}
/>
)}
<Card>
<CardContent className="space-y-3">
<p className="text-sm font-medium text-foreground-light">
Schemas and tables to replicate
</p>
<div className="overflow-hidden rounded-md border divide-y">
{schemasWithTables.map((schema) => {
const keys = schema.tables.map((table) => getSchemaTableKey(schema.schema, table))
const checkedCount = keys.filter((key) => selection[key]).length
const checkedState = getSchemaCheckedState({
selectedCount: checkedCount,
totalCount: keys.length,
})
const isOpen = expandedOverrides[schema.schema] ?? checkedCount > 0
const selectableTableCount = schema.tables.filter(
(table) =>
!isEditing || !initialSelection[getSchemaTableKey(schema.schema, table)]
).length
return (
<Collapsible
key={schema.schema}
open={isOpen}
onOpenChange={(open) => setExpanded(schema.schema, open)}
>
<div className="flex items-center gap-2 px-3 py-2 bg-surface-75">
<CollapsibleTrigger
aria-label={
isOpen ? `Collapse ${schema.schema}` : `Expand ${schema.schema}`
}
className="group text-foreground-lighter"
>
<ChevronRight
size={14}
className="transition-transform group-data-[state=open]:rotate-90"
/>
</CollapsibleTrigger>
<Checkbox
checked={checkedState}
onCheckedChange={() => toggleSchema(schema)}
disabled={selectableTableCount === 0}
aria-label={`Select all tables in ${schema.schema}`}
// The shared Checkbox only fills itself for `data-state=checked`, so a partial
// selection would otherwise render identically to an empty one. A muted fill
// keeps all three states visually distinct.
className="data-[state=indeterminate]:border-foreground-lighter data-[state=indeterminate]:bg-foreground-lighter"
/>
<span className="text-sm font-mono text-foreground">{schema.schema}</span>
<span className="text-xs text-foreground-lighter ml-auto">
{checkedCount}/{keys.length} tables
</span>
</div>
<CollapsibleContent>
{schema.tables.map((table) => {
const key = getSchemaTableKey(schema.schema, table)
return (
<div key={key} className="flex items-center gap-2 pl-10 pr-3 py-2">
<Checkbox
checked={!!selection[key]}
disabled={isEditing && !!initialSelection[key]}
onCheckedChange={() => toggleTable(schema.schema, table)}
aria-label={`Select ${schema.schema}.${table}`}
/>
<span className="text-sm font-mono text-foreground-light">{table}</span>
</div>
)
})}
{schema.tables.length === 0 && (
<p className="pl-10 pr-3 py-2 text-sm text-foreground-lighter">
No tables in this schema.
</p>
)}
</CollapsibleContent>
</Collapsible>
)
})}
</div>
</CardContent>
<CardFooter className="justify-between">
<span className="text-sm text-foreground-lighter">
{selectedCount} table{selectedCount === 1 ? '' : 's'} selected
</span>
<div className="flex items-center gap-x-2">
{hasChanges && <Button onClick={() => setSelectionOverride(null)}>Cancel</Button>}
<Button
variant="primary"
disabled={selectedCount === 0 || !hasChanges}
loading={isSubmitting}
onClick={handleSubmit}
>
{isEditing ? 'Add replicated tables' : 'Enable Warehouse'}
</Button>
</div>
</CardFooter>
</Card>
</PageSectionContent>
</PageSection>
)
}
@@ -0,0 +1,128 @@
import { useParams } from 'common'
import { Button } from 'ui'
import { Admonition } from 'ui-patterns/Admonition'
import { GenericSkeletonLoader } from 'ui-patterns/ShimmeringLoader'
import {
buildRetryTargets,
isWarehouseSettingUp,
type WarehouseSetupTarget,
} from './Warehouse.utils'
import { WarehouseConnectSection } from './WarehouseConnectSection'
import { WarehouseSchemaTablePicker } from './WarehouseSchemaTablePicker'
import {
WarehouseEnablingProgress,
WarehouseReplicatedTablesSection,
} from './WarehouseTableStatusList'
import { AlertError } from '@/components/ui/AlertError'
import { checkLocalETLNotSetUp } from '@/data/replication/utils'
import { useWarehouseSetupMutation } from '@/data/warehouse/warehouse-setup-mutation'
import { useWarehouseSetupStatusQuery } from '@/data/warehouse/warehouse-setup-status-query'
export const WarehouseSetupPanel = () => {
const { ref: projectRef } = useParams()
const { data, isPending, isFetching, isError, error, refetch } = useWarehouseSetupStatusQuery(
{ projectRef },
{
refetchInterval: (query) =>
isWarehouseSettingUp(query.state.data?.setup_status) ? 3000 : false,
}
)
// Rendered inline by the picker rather than as a toast: a setup failure is something the user
// has to act on, so it must not disappear.
const setupMutation = useWarehouseSetupMutation({ onError: () => {} })
const handleSetup = (targets: WarehouseSetupTarget[]) => {
if (!projectRef || targets.length === 0) return
setupMutation.mutate({ projectRef, body: { targets } })
}
if (isPending) return <GenericSkeletonLoader />
// Warehouse rides on the replication API, which isn't wired up in local development. Same
// treatment Pipelines gives it, so a local dev doesn't read this as a broken build.
if (isError && checkLocalETLNotSetUp(error)) {
return (
<Admonition
type="default"
title="Warehouse is unavailable locally"
description="Configure the replication API to set up Warehouse in local development."
/>
)
}
if (isError) {
return (
<AlertError
subject="Failed to load Warehouse status"
error={error}
additionalActions={
<Button variant="default" loading={isFetching} onClick={() => refetch()}>
Retry
</Button>
}
/>
)
}
if (!data) return <GenericSkeletonLoader />
const status = data.setup_status
if (status === 'not_started') {
return (
<WarehouseSchemaTablePicker
onSubmit={handleSetup}
isSubmitting={setupMutation.isPending}
error={setupMutation.error}
/>
)
}
if (isWarehouseSettingUp(status)) {
return <WarehouseEnablingProgress status={data} />
}
if (status === 'error') {
const retryTargets = buildRetryTargets(data.tables)
const failingStep = data.steps.find((step) => step.status === 'error')
return (
<AlertError
subject="Warehouse setup failed"
showErrorPrefix={false}
error={{
message:
setupMutation.error?.message ??
failingStep?.message ??
'An unknown error occurred while setting up Warehouse.',
}}
additionalActions={
retryTargets.length > 0 ? (
<Button
variant="default"
loading={setupMutation.isPending}
onClick={() => handleSetup(retryTargets)}
>
Retry
</Button>
) : undefined
}
/>
)
}
return (
<>
<WarehouseReplicatedTablesSection tables={data.tables} />
<WarehouseSchemaTablePicker
isEditing
onSubmit={handleSetup}
isSubmitting={setupMutation.isPending}
error={setupMutation.error}
/>
<WarehouseConnectSection />
</>
)
}
@@ -0,0 +1,139 @@
import dayjs from 'dayjs'
import { Badge, Card, CardContent, Tooltip, TooltipContent, TooltipTrigger } from 'ui'
import { Admonition } from 'ui-patterns/Admonition'
import {
PageSection,
PageSectionContent,
PageSectionDescription,
PageSectionMeta,
PageSectionSummary,
PageSectionTitle,
} from 'ui-patterns/PageSection'
import type { WarehouseSetupTable } from './Warehouse.utils'
import type { WarehouseSetupStatusResponse } from '@/data/warehouse/warehouse-setup-status-query'
import { formatBytes } from '@/lib/helpers'
const TABLE_STATE_BADGE: Record<
WarehouseSetupTable['state'],
{ label: string; variant: 'warning' | 'success' | 'destructive' }
> = {
syncing: { label: 'Backfilling', variant: 'warning' },
live: { label: 'Synced', variant: 'success' },
error: { label: 'Error', variant: 'destructive' },
}
/**
* Replication lag is always non-zero, so a raw figure reads as a problem when it isn't. Anything
* inside this window is normal streaming latency.
*/
const CAUGHT_UP_THRESHOLD_MS = 10_000
const getLagLabel = (lagMs: number) => {
if (lagMs < CAUGHT_UP_THRESHOLD_MS) return 'Caught up'
return `${dayjs.duration(lagMs, 'milliseconds').humanize()} behind`
}
const TableLag = ({ table }: { table: WarehouseSetupTable }) => {
if (table.lag_ms === undefined) return null
const label = getLagLabel(table.lag_ms)
if (!table.last_synced_at) {
return <span className="text-sm text-foreground-light">{label}</span>
}
return (
<Tooltip>
<TooltipTrigger className="text-sm text-foreground-light underline decoration-dotted decoration-foreground-muted/30 underline-offset-4 transition-[text-decoration-color] duration-200 hover:decoration-foreground-lighter">
{label}
</TooltipTrigger>
<TooltipContent side="bottom">
Last synced {dayjs(table.last_synced_at).fromNow()}
</TooltipContent>
</Tooltip>
)
}
interface WarehouseTableStatusListProps {
tables: WarehouseSetupTable[]
}
const WarehouseTableStatusList = ({ tables }: WarehouseTableStatusListProps) => {
return (
<Card>
<CardContent className="p-0 divide-y">
{tables.map((table) => {
const badge = TABLE_STATE_BADGE[table.state]
return (
<div
key={`${table.schema}.${table.name}`}
className="flex items-center gap-4 px-3 py-2.5"
>
<span className="flex-1 truncate text-sm">
<span className="text-foreground-lighter">{table.schema}.</span>
<span className="text-foreground">{table.name}</span>
</span>
<TableLag table={table} />
{table.warehouse_size_bytes !== undefined && (
<span className="text-sm text-foreground-light tabular-nums">
{formatBytes(table.warehouse_size_bytes)}
</span>
)}
<Badge variant={badge.variant}>{badge.label}</Badge>
</div>
)
})}
{tables.length === 0 && (
<p className="px-3 py-2.5 text-sm text-foreground-lighter">
No tables are being copied yet.
</p>
)}
</CardContent>
</Card>
)
}
export const WarehouseReplicatedTablesSection = ({ tables }: WarehouseTableStatusListProps) => (
<PageSection className="pt-0!">
<PageSectionMeta>
<PageSectionSummary>
<PageSectionTitle>Status</PageSectionTitle>
<PageSectionDescription>Replication status of the selected tables.</PageSectionDescription>
</PageSectionSummary>
</PageSectionMeta>
<PageSectionContent>
<WarehouseTableStatusList tables={tables} />
</PageSectionContent>
</PageSection>
)
interface WarehouseEnablingProgressProps {
status: WarehouseSetupStatusResponse
}
export const WarehouseEnablingProgress = ({ status }: WarehouseEnablingProgressProps) => {
const syncedTableCount = status.tables.filter((table) => table.state === 'live').length
const progressDescription =
status.setup_status === 'setting_up'
? 'Creating the replication pipeline. Connection details appear once the first backfill finishes.'
: `Backfilling selected tables. ${syncedTableCount} of ${status.tables.length} tables synced.`
return (
<PageSection className="pt-0!">
<PageSectionMeta>
<PageSectionSummary>
<PageSectionTitle>Status</PageSectionTitle>
<PageSectionDescription>Warehouse setup progress.</PageSectionDescription>
</PageSectionSummary>
</PageSectionMeta>
<PageSectionContent className="space-y-4">
<Admonition
type="default"
title="Warehouse is being set up"
description={progressDescription}
/>
{status.tables.length > 0 && <WarehouseTableStatusList tables={status.tables} />}
</PageSectionContent>
</PageSection>
)
}
@@ -39,5 +39,6 @@ export const useWarehouseSetupStatusQuery = <TData = WarehouseSetupStatusData>(
queryKey: warehouseKeys.setupStatus(projectRef),
queryFn: ({ signal }) => getWarehouseSetupStatus({ projectRef }, signal),
enabled: enabled && typeof projectRef !== 'undefined',
retry: false,
...options,
})
@@ -44,6 +44,7 @@ const INTEGRATION_OVERVIEWS = {
stripe_sync_engine: () => import('@/static-data/integrations/stripe_sync_engine/overview.md'),
stripe_wrapper: () => import('@/static-data/integrations/stripe_wrapper/overview.md'),
vault: () => import('@/static-data/integrations/vault/overview.md'),
warehouse: () => import('@/static-data/integrations/warehouse/overview.md'),
webhooks: () => import('@/static-data/integrations/webhooks/overview.md'),
} satisfies Record<string, () => Promise<{ default: string }>>
@@ -0,0 +1,3 @@
Supabase Warehouse keeps a copy of your database optimized for analytical queries. Applications keep using
Postgres for transactional workloads, while analytical tools connect to a separate FlightSQL endpoint.
This keeps the two paths independent, so analytical queries never run against your primary database.