Commit Graph
8 Commits
Author SHA1 Message Date
Alaister YoungandAlaister Young 5950b6ca04 test(e2e/studio): stabilize flaky and TZ/OS-sensitive specs (#46039)
Backports a batch of e2e test stabilization fixes — each commit is
scoped to a single failure class and only touches `e2e/studio/` files.

**Changed:**
- **`_global.setup` — playwright-locks cleanup was dead code**: the
lock-cleanup block was at the bottom of `Global Setup`, but every branch
above it returns early — so it never ran. Tests that use
`withFileOnceSetup` (cron-jobs) would see a stale `setup.done.json`
marker from the previous run and silently skip their setup, leaving e.g.
`pg_cron` uninstalled and all 11 cron-jobs specs failing. Moved the
cleanup to before any early return.
- **filter-bar — Home key**: macOS Chromium doesn't honor a standalone
`Home` keypress inside text inputs (macOS routes "go to line start" via
`Cmd+ArrowLeft` / `Fn+ArrowLeft`). Tests that expected the cursor to
jump to position 0 silently kept the previous selection. Replaced with
`el.setSelectionRange(0, 0)` so the assertion runs against a known
cursor position on every OS.
- **filter-bar — date filters**: tests inserted rows with `CURRENT_DATE`
/ `NOW()` (postgres session TZ = UTC) and asserted with JS-local dates
from `getDateValue()`. Near midnight the two diverged and the filter
returned 0 rows. Switched the inserts to explicit `getDateValue()`
strings so insert and assert use the same calendar day.
- **queue-table-operations — `networkidle`**: Studio holds long-poll /
SSE connections (PostHog, realtime), so `page.reload({ waitUntil:
'networkidle' })` never resolves and timed out. Replaced with a targeted
`waitForTableToLoad` API waiter.
- **sql-editor — RLS smoke test**: a hard-coded table name
(`pw_rls_smoke_test`) collided across 3 parallel workers running against
the same db. Suffixed with `test.info().parallelIndex`.
- **table-editor — FK spec timeout**: `waitForApiResponseWithTimeout`
for `query?key=table-update` returns `null` on timeout (silent), then
the panel-close assertion fails. Bumped 15s → 30s to absorb
parallel-load latency.
- **table-editor / storage-helpers — URL encoding & redirect race**:
post-action URL assertions were over-specific (`%20` vs `+` encoding)
and the bucket-delete redirect could race other history updates. Relaxed
the regex to accept both encodings; asserting the row removal directly
is a more stable signal than the redirect URL.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Improved end-to-end determinism with explicit dates/timestamps and
stable cursor positioning
* Prevented parallel-test collisions by using unique identifiers for
resources
* Made page reloads and API waits more robust for long-lived connections
and increased timeouts
* Strengthened assertions to rely on stable UI signals instead of
transient navigation/network state
* Ensured test setup reliably cleans up temporary locks before any setup
steps run

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46039?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com>
2026-05-18 17:31:40 +08:00
Gildas Garcia 13bfc4502a chore: e2e general improvements (#43547)
## Problem

e2e tests are still flaky and not as fast as they could be

## Solution

- [x] Reset supabase instance: this makes the database visualiser tests
faster as there are less tables to screenshot
- [x] Improve the single file setup utilities so that they never block
local tests by cleaning them up before starting
- [x] Disable animations while running the tests (less time waiting for
animations to complete
- [x] Add utility functions that help reproducing flaky tests locally
2026-03-09 15:38:06 +01:00
Ali Waseem e07a3e2838 feature: login w/ Github for production E2E tests (#41495)
* updated github script for login

* refactored and cleans up github login

* wip

* updated github script

* removed context dir for page access

* updated auth script

* updated the login methods

* updated README

* updated minor bugs

* added error handling
2025-12-19 10:48:27 -07:00
Ali WaseemandCharis 22cfd2be14 feat: Run E2E tests aganist Platform Pt.1 (#41032)
* added packages for creating projects

* updated scripts

* remove ami version

* cleaned up common

* updated tests

* refactored helpers

* updated env

* updated config

* updated to reference env

* updated global setup

* updated type logic and scripts

* added mocking of hcaptcha

* added log statements

* updated local env

* update env file

* updated env vars

* updated logging

* updated to remove check

* updated print and project names

* updated helpers

* updated url

* updated setup

* updated storage helpers to account for listing files

* updated setup and tests

* updated timeout only for setup

* updated helper to account for different api response

* added ignores for tests

* updated lock file

* updated database spec to add exact

* updated timeouts

* removed check for table grid footer

* updated test runner

* updated is_platform

* updated playwright config

* updated worker settings

* removed dotenvx

* updated README

* updated to remove comment

* Update e2e/studio/scripts/common/retriedFetch.ts

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

---------

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>
2025-12-05 10:50:16 -07:00
CharisandAli Waseem 044a575552 test: parallelize table editor tests (#40404)
Table editor tests were being run serially because they would otherwise
stomp all over each other in setup. But this made them really slow. This
PR changes them to run in parallel, and makes sure the setup step runs
only once by using a file lock.

Also cleans up miscellaneous other sources of flakiness in tests.

Co-authored-by: Ali Waseem <waseema393@gmail.com>
2025-11-19 07:14:13 -07:00
Ali WaseemandCharis 7898abe06c Fix: Make self hosted tests more reliable locally and in CI (#40479)
* added ui studio command

* added SKIP ASSET UPLOAD option

* updated failing tests

* updated e2e scripts to include CI instructions

* removed unused envs and slack messages

* updated timeouts and max retries on CI

* updated generated local env

* updated env

* updated env generation for e2e tests

* remove vercel preview

* updated tests and trying junit report on PR

* update test comment

* updated commit hash for reporter

* updated file path

* updated runner to blacksmith

* update test file for reporting

* Update .github/workflows/studio-e2e-test.yml

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

* Update package.json

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

* Update e2e/studio/playwright.config.ts

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>

* updated to skip comment on forks

---------

Co-authored-by: Charis <26616127+charislam@users.noreply.github.com>
2025-11-17 15:00:31 -07:00
Jordi Enric 25abebc32e a new hope (#38893)
* a new hope

* run tests in ci against cli mode

* summary

* try vercel action to run e2e against studio self hosted preview

* believe

* debug

* gh pages artifact

* test

* rm pages step

* fix automation bypass missing

* continue on error

* only install necessary deps for CI

* fix bypass

* remove

* fail job if test fails

* disable customer query if is_platform false

* vercel check

* fix var name, make comment update instead

* check bypass on runtime

* add env var

* fix tests going to project ref instead of default

* fix

* better dates in comment

* Update E2E test workflow to include flaky test detection and improve summary output

* fix

* fix dumb mistake
2025-09-23 12:02:23 +02:00
55c3fdb49a E2E Testing Setup Consolidation (#35080)
* add new e2e folder

* add local supabase and confitional storage

* fix e2e selfhosted

* update actions

* add correct e2e folder

* fix e2e actions

* fix action project ids

* fix permissions

* fix script

* fix playwright install

* playwright root

* pnpm i

* fix api rul

* add env docs

* update run script

* only install deps for e2e

* use same dep

* only install deps for tests

* upd lockfile

* use official vercel integration

* use vercel cli

* remove old folder

* fix script

* rm filter

* rename e2e studio package

* fix install browsers

* add polling for vercel build

* use vercel-preview-url package

* undo actions

* rename ci env to ci

* chore:add rls check and make playwright test less flakey (#35348)

* update ci action

* fix paths

* fix browser install

* run ci against staging

* try caching builds

* fix envs

* fix env check

* fix sign in

* fix sign in url

* fix envs and url

* fix caching

* fix race condition in sign in page

* fix race condition in sign in page

* add check to see if being redirected

* fix caching, check IS_PLATFORM var

* log is_platform

* try vercel build

* fix vercel project id

* fix path

* add temp vercel.json file

* fix paths

* undo project id stuff

* rm cwd

* fix path

* fix paths again

* fix path

* fix base url

* try different fix

* fix config base url

* fix base studio url issues

* retain video on fails

* Update e2e/studio/README.md

Co-authored-by: Copple <10214025+kiwicopple@users.noreply.github.com>

* Update e2e/studio/README.md

Co-authored-by: Copple <10214025+kiwicopple@users.noreply.github.com>

* fix env file naming

* undo caching

* rm old tests folder

* fix readme scripts

* rm vercel deploy for now, just run build locally

* fix url

* fix build script

* fix is_platform

* fix stuck studio start

* fix env vars

* retain network and logs on fail for better debugging

* add apiurl env

* back to vercel

* disable catpcha

* fix test

* update environment configuration to remove default URLs for CI and streamline API base URL handling

* fix typeerr

* fix urls in home.spec

* fix urls in logs.spec

* fix urls in sqleditor spec

* fix table editor spec

* add tourl util

* use staging api in ci

* re add base url env var

* fix url in projects page

* fix url in sql editor spec

* fix sign in not waiting for cookies omfg

* fix env var name

* fix sql-editor test

* simplify table removal

* add opt out telemetry step

* fix logs tests

* fix table editor spec

* remove flaky steps from table editor tests

* use vercel deployment events instead of build

* add studio check

* fix condition

* debug event

* rm if

* trigger deploy

* undo ac

* make opt out button step optional, some envs dont hav eit

* use testid for sql run button

* use id instaed of timestamp in logs tests

* empty

* rm retries

* up glbal timeout

* chore: fix failing sql-editor playwright test (#35767)

* chore: fix failing sql-editor playwright test

* chore: minor fixes

* Chore/update playwright config (#35826)

chore: update playwright config

* rm supabase project from e2e tests

* refactor and simplify environments

* fix sql editor test

* fix ci env vars

* fix

* fix on windows

* update readme

* add playwright install script to readme

* rm turbopack trace flag

* npm to pnpm for scripts

* delete ivan lines

---------

Co-authored-by: Michael Ong <minghao_3728@hotmail.com>
Co-authored-by: Copple <10214025+kiwicopple@users.noreply.github.com>
2025-06-03 17:37:39 +02:00