Commit Graph
4 Commits
Author SHA1 Message Date
Danny WhiteandJoshen Lim a351a36e9b refactor(studio): centralise replication payload builders (#49842)
## What kind of change does this PR introduce?

Studio data-layer refactor.

## What is the current behavior?

Pipeline creation, editing, and validation build similar destination and
pipeline payloads separately. The duplicated mappings rely on type
assertions and can drift between actions.

## What is the new behavior?

Uses shared typed builders for create, update, and validation payloads
across the existing destinations. Update payloads continue to omit blank
secrets, while create payloads preserve their current values. This PR
does not add table partitioning configuration.

## To test

This is a data-layer refactor. No visible behaviour should change.

1. Open **Database > Replication** and click **Start a new pipeline**.
2. Select **BigQuery**, or any other enabled destination.
3. Edit a few non-secret fields and expand **Advanced settings**.
4. Confirm the form remains usable and no runtime errors appear.

Create, update, validation, and secret-handling behaviour is covered by
the focused tests and CI. Deploy previews and fresh local projects do
not have the existing destinations or credentials needed to exercise
those paths manually.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved replication destination configuration handling during
creation, updates, and validation.
* Applied consistent configuration mapping across supported destination
types.
* Ensured blank secret values are omitted during updates while retained
when creating destinations.
* Standardized table synchronization defaults when no specific setting
is provided.
* **Tests**
* Added coverage for BigQuery configuration mapping and secret handling.
  * Updated DuckLake tests for destination updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
2026-09-07 15:02:09 +08:00
Joshen Lim ea909e998b Local only: skip loading destinations if ETL API is not set up (#47031)
## Context

> [!IMPORTANT]  
> Changes in this PR only apply to the local environment - there should
not be any changes to staging (nor production)

Given that read replicas currently sit under database replication, the
UI currently waits for replication destinations to load before rendering
the page. However for local development, setting up of the ETL API isn't
necessary nor applicable for everyone so this indirectly adds friction
if we just want to work with read replicas.

## Changes involved
- Opting to skip retrying fetching ETL related requests if the error
returned is "replication API URL is not configured"
  - This is indicative that the local platform isn't set up for ETL yet
- ^ Database replication page will hence not wait for ETL requests to
succeed before finally rendering the UI
  - Node diagram will also then render properly (just read replicas)
- Add a small admonition to visualize this
<img width="1079" height="301" alt="image"
src="https://github.com/user-attachments/assets/32bd5d2f-a76e-417e-bedf-9a04de3bb305"
/>

## To test
- Will only be able to test locally - basically just head over to the
database replication page (unless you somehow already have ETL API set
up locally)
- But can also verify that there's no changes on staging preview


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved error messaging when ETL is not configured in local
development environments
* Enhanced error handling for replication API failures with better
non-retryable error detection

* **Improvements**
* Refined replication diagram rendering based on destination setup state
  * Updated dropdown menu interactions for read replica management

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-19 17:27:44 +08:00
Charis 3b7052b5a9 cleanup: fix import order and prefixes for studio/data (#44501) 2026-04-03 09:15:57 +02:00
Riccardo Busetti 1a3ba9f15a ref(etl): Rename ETL Replication to Replication and update docs (#40769)
* ref(etl): Rename ETL Replication to Replication and update docs

* Fix

* Fix

* Fix

* Fix

* Fix
2025-11-25 20:45:50 +08:00