Bug fix
## What is the current behavior?
Read-only users cannot download files because the download feature
requires minting a temporary API key, which is properly blocked for
read-only users.
## What is the new behavior?
Instead of using temporary API keys, we now create signed URLs for the
files to be downloaded. We batch-create signed URLs for an entire
folder's worth of files, requiring only a single management API call,
then use those signed URLs to download the files. This allows read-only
users to download files without needing elevated permissions.
## Additional context
Resolves FE-2737
## Context
Related to Dashboard Scalability, specifically having Postgres team as
CODEOWNERS for dashboard queries
This is just a clean up as we're currently piling manual queries into
one folder under `data/sql/queries`, whereas I reckon it'll be better
for each query to sit within their RQ folder for better context.
Am opting the naming format for files housing queries to be `*.sql.ts`,
and also updating CODEOWNERS to reflect as such
Next step will also be to shift all the dashboard queries within pg-meta
into studio itself as requested by the pg-meta team
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Consolidated and reorganized internal module structure for the data
layer to improve maintainability and reduce redundancy.
* Streamlined import paths across components to align with new
consolidated module organization.
* **Chores**
* Updated code ownership patterns to reflect reorganized file structure.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Ali Waseem <waseema393@gmail.com>
* fix(studio): use paginated buckets query for useSelectedBucket
Part of the effort to migrate all buckets list queries to the paginated
version. This migrates the useSelectedBucket hook, which is used on the
bucket details page to fetch the bucket info.
* fix(studio): flash of stale content when switching buckets
There's a flash of stale content when switching between buckets in the
File Explorer. This happens since the change to useSelectedBucket,
because when the bucket changes, the new data is no longer available
immediately. It must be awaited, and meanwhile `columns` in global
storage state is stale. This commit adds a check for whether the state
is stale by checking the current ID against the stored ID.
---------
Co-authored-by: Ali Waseem <waseema393@gmail.com>
The command menu search option uses the unpaginated version of the
buckets endpoint, which times out / causes jankiness when users have
tends of thousands of buckets.
Switch to using the paginated version. The old behaviour is preserved,
where only the first 10 results are shown, and the user has to specify a
more specific search query to get other buckets. This is for UX
simplicity, since this is a convenience feature, and fully
infinite-loading search is available in the full Storage interface.
* feat(studio): show icon on tables exposed by data api
Show an icon in the Table Editor for tables that are exposed via the
Data API.
* refactor(studio): move table api access query up to parent
* fix(studio): invalidate table privileges query on table create
* fix(studio): make RLS disabled warning on table editor more obvious
Change from warning -> danger
* Only show add foreign key in side panel if isSuccess
* chore(studio): put data api exposed badge behind feature flag
Only show badge for now if flag `dataApiExposedBadge` is enabled.
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
The global storage size validation depends on an unpaginated buckets
query to determine whether it is lower than any individual bucket's
cutoff. This causes a problem for users with tens of thousands of
buckets.
There's a bit of a UX/performance problem here, because in order to
determine whether any bucket's `file_size_limit` exceeds the global
setting, we need to get the max `file_size_limit` of `storage.buckets`
-- however, that column is not indexed.
My workaround is:
- Below a certain threshold (10,000) buckets, the query for max
`file_size_limit` is automatically run on form submit.
- Above that threshold, the user must confirm whether they want to run
the query. They're still allowed to change the storage config without
running it -- this does open a loophole where they can have a global
storage setting lower than an individual bucket's file size limit, but
though this is a potentially confusing situation, it's not strictly an
error.
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Bump the deps, refactor deprecated code.
* Migrate keepPreviousData usage.
* Migrate all uses of InfiniteQuery.
* Fix refetchInterval in queries.
* Migrate all use of isLoading to isPending in mutations.
* Fix accessing location in claim-project.
* Fix a bug in duplicate query keys.
* Migrate all queries to use isPending.
* Revert "Fix accessing location in claim-project."
This reverts commit 2a07df64b5.
* Revert the rss.xml file to master.
* feat(studio): use bucket pagination for api docs
API docs V2 updated to use pagination for fetchign storage buckets.
Also some refactoring to split up large components for docs V2
interfaces.
* fix(studio): use bucket endpoint for getting single bucket
The bucket list endpoint now accepts pagination options. Adapting the
storage buckets table UI to paginate in pages of 100 buckets at a time,
with infinite loading/scrolling.
* Midway
* Hook up create/delete table, and update all iceberg RQs with mgmt API routes
* Add insert data CTA, update create table logic for decimal and fixed types
* Clean up
* Nit
* Fix test
* Refactor the iceberg queries and mutations to use getOrRefreshTemporaryApiKey.
* Fix the uses of the mutations.
* Hardcode the temp api keys queries at 60 seconds.
* When creating a s3 vectors fdw, add the target schema as a server option.
* Fix a bug in the row link, it didn't work when you middle click.
* Refactor the protected schema logic to include schemas from all fdws.
* Refactor the protected schema logic to include fdw type and name. Change the ProtectedSchemaWarning to show different messages for the 2 fdw types.
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Retrieve temporary API key within iceberg-namespace-tables-query
* Retrieve temporary API key within iceberg-namespace-table-delete-mutation
* Retrieve temporary API key within iceberg-namespaces-query
* Fix
* Retrieve temporary API key within iceberg-namespace-create-mutation
* Update based on comments
* Fix childProps in Admonition so that they're added as a prop to the main div.
* Replace the admonition with Alert in Wrapper tab page to add a gap between childs.
* Add s3 vectors fdw.
* Minor fix to FormSection.
* Update the fdw mutations to support passing in options.
* Refactor the vector flow to create fdws.
* Revert cron description change.
* If the bucket can't be created, don't create a fdw.
* Update/delete the fdw when deleting a table or a bucket.
* Minor fixes.
* Clean up the delete modal.
* Handle edge cases when missing a wrapper.
* Remove the admonition in the create bucket modal.
* Fix the loading state when creating a bucket.
* Fix the createWrapper sheet to work with s3 vectors.
* Fix undefined wrapperMeta issue.
* Create the schema when installing a wrapper.
* Tiny cleanup.
* Clean up unneeded useState. Create a wrapper only if the all conditions are met.
* Fix all comments.
* Add s3 vectors for docs.
* Add a link and fix the file name for S3 Vectors in docs.
* Hide the table editor button if the wrapper instance is missing.
* Small fixes.
* midway
* Hook up connect tables e2e
* Update isEnabled flag
* Change connectingStep to enum
* Nit
* Nit
* Nit
* Only show replication status if theres a publication
* Improve loading state in etl page
* Refactor + Clean up + Add long poll logic when adding more tables
* Initial work on vector buckets.
* Add more queries and UI.
* chore(studio): vector buckets contents (#39705)
* language
* match analytics buckets
* shared empty state
* proper loading state
* create dialog cleanup
* simplify copywriting
* proper details
* filtering
* use real data
* chore(studio): vector buckets polish A (#39759)
* empty state
* better checks
* bucket name in sheet
* fix typo
* sheet improvements
* rename vector index to vector table
* cleanup
* More work on the vector indexes.
* Add delete for bucket and indexes. Create index should now work.
* chore(studio): vector buckets polish B (#39794)
* loading
* delete modal
* fix modal behaviour
* Add modal for deleting vector buckets.
* Bring back the shimmering loader.
* chore(studio): vector buckets polish C (#39887)
* fix bucketname on deletion dialog
* empty states
* remove extraneous padding
* proper loading and not-found cases
* Minor fixes.
* Minor updates to the vector components.
* Add more instructions to the cursor rules.
* Some refactors and shifting files around
* Refactor analytics buckets detial to improve time to fcp
* Fix
* Refactors + simplify
* Last few nits, just leaving comments
* If the bucket is not empty, delete all indexes first.
---------
Co-authored-by: Danny White <3104761+dnywh@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* First pass of deprecating old storage UI components
* Clean up local-storage flag for new storage UI
* Add coming soon UI for analytics and vector buckets pages
* Set up warning states for analytics buckets
* fix spelling and margin
* Smol fix
* nit
* Surface wrappers upgrade check in UI instead of toast
* Nit clean up
* update tests
---------
Co-authored-by: Danny White <3104761+dnywh@users.noreply.github.com>
* Add custom types for queries, mutations and infinite queries.
* Migrate all queries to use the new type.
* Migrate all infinite queries to useCustomInfiniteQueryOptions.
* Migrate all mutations to use useCustomMutationOptions.
* Add type to all imports in `types` folder.
* Migrate all uses of invalidateQueries to use object syntax.
* Migrate the remainder of useInfiniteQuery.
* Migrate all setQueriesData.
* Migrate all fetchQuery uses.
* Migrate some leftover functions from RQ.
* Fix issues found by Charis.
* Update perms checking in audit logs
* Deprecate useCheckPermissions, useIsPermissionsLoaded and useCheckProjectPermissions as they're no longer used
* Rename useAsyncCheckProjectPermissions to useAsyncCheckPermissions
* Fix TS
* Wrap up remaining swapping of useCheckPermissions
* remove OrganizationSettingsLayout on billing, team, integrations and usage org pages
* Fix missing loading state in usage page
* Add comments for next step
* feat: basic clarity improvements
* feat: all switches
* feat: basic errors
* fix: simplify
* fix: properly separate sections
* fix: remove smart toggle closure
* fix: remember file size unit
* fix: validation
* fix: UpgradeToPro styles
* fix: UpgradeToPro styles for full-width
* feat: create modal match
* fix: tests
* Small fix for error message and desc for file size limit in create and edit bucket modals
* Prettify error message handling for storage settings
* Nit
* set default unit as MB
* dotted underline and default cursor for tooltip
* fix: default MB tests
* minor form style and copy improvements
* remove extraneous divs
* fix mime check on bucket creation
* Clean up + minor refactors including improving perms loading state in storage settings
* EditBucketModal ensure that file size limit unit defaults to MB if bucket doesnt have a file size limit
* Refactor EditBucketModal to use react query mutation
* Refactor
* Fix unit tests for edit bucket modal
* last round of clean up
* last fix
* Tiny unrelated fix
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Replace all usage of useProjectContext with useSelectedProjectQuery
* Replace all usage of useSelectedProject with useSelectedProjectQuery
* Replace all usage of useProjectByRef with useProjectByRefQuery
* Replace all usage of useSelectedOrganization with useSelectedOrganizationQuery
* Deprecate useSelectedProject, useSelectedOrganization, and useProjectByRef hooks
* Deprecate ProjecContext
* fix: use storage endpoint in S3 settings
* Update retrieval of endpoint for analytics bucket related components
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Change the import foreign schema to always create a schema.
* Fix the Iceberg wrapper.
* Refactor the create wrapper to always create a new schema.
* Remove unneeded props.
* Smol fixes
* Prevent double error toasts
* Smol fix
* Fix the wrapper creation to include the correct api key.
* Fix a bug with the new api keys.
* Handle both types of keys when fetching the iceberg namespaces.
* Add a field to all wrappers to hold the schema name which can import foreign schema.
* sq
* When importing a foreign schema, save the schema in a special field.
* Fix a type error.
* Fix importing foreign schema overriding wrapper server options with unencrypted values
* Add comment
* Handle duplicate and empty schemas when importing a foreign schema.
* Update the copy in the foreign schema wrappers.
* Remove unnecessary code.
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Update studio testing setup files
Improves API mocking type safety and polyfills browser APIs necessary to run tests with framer-motion components
* fix missing listen call for msw, resolve test type error
* fix imports
* Update studio testing setup files
Improves API mocking type safety and polyfills browser APIs necessary to run tests with framer-motion components
* fix missing listen call for msw, resolve test type error
* fix imports
* Move non-layout Storage related components to `components/interfaces`
* Fix paths
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Deprecate getAPIKeys from project settings v2, use new api keys endpoint instead
* use getPreferredKeys where appropriate
* Prevent usage of secret key for storage and realtime inspector
* Add dashboard API api-keys endpoint
* Simplify
* Disable edge functions test if legacy api keys are disabled
* Revert
* Fix graphiql
* Remove all usage of api keys from project settings, except DisplayApiSettings
* Update
* Fix
* Small fix for an undefined upload url.
* Fix the storage state initialization when the resumable url changes.
---------
Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
* Migrate the CreateBucketModal to use RHF.
* Minor fixes for the create bucket modal.
* Use the bucket type from the API everywhere.
* Improve the types on some functions. Add "skip" mode to FDWCreateMutation.
* Expand the CreateBucketModal to explain what will happen when creating an iceberg bucket.
* Add a page for iceberg buckets.
* Add error states to the iceberg explorer.
* Fix the names for the FDWs.
* Add a dialog for setting up namespace.
* Restructure the new files.
* Add a download env button.
* Add warning to install the wrappers extension when creating an iceberg bucket.
* Add failover details when the FDW is not setup.
* Fix a lint error.
* Fix a type error.
* Minor fixes.
* Fix the server name.
* Add an icon for iceberg buckets.
* Fix the import foreign schema dialog.
* Make the setup wrapper button functional.
* Fix a bad useMemo dependency.
* Small changes to the iceberg bucket page.
* Minor fix for the edit wrapper sheet.
* Rename the files from kebab-case to PascalCase.
* Rename the files again to include Analytic instead of Iceberg.
* Rename Iceberg type to Analytic.
* Add a switch for creating namespace in the import foreign schema dialog.
* Fix the CreateBucketModal.
* Fix the delete modal feature.
* Fix the S3 keys in the FDW.
* Only create a namespace if the switch is true.
* Regenerate and fix the types.
* Fix the FDW create mutation to handle numbers in the FDW names.
* Make the icon smaller.
* Check whether the namespace exists, if it doesn't create it.
* Hide action from the analytic bucket which don't work.
* Invalidate namespaces when creating them.
* Add small explanation for the creation of namespaces.
* Minor fixes.
* Tons of changes to make the namespace feature work.
* Fix type errors.
* Fix bad import.
* Minor copy fixes.
* Replace the multiple cards with a table of namespaces.
* update copy icon
* tiny copy update
* Fix the empty state for foreign tables.
* Hide the analytics bucket option for self-hosted.
* Minor copy fixes.
* Expand the CTA on no namespaces state.
* More minor fixes.
* More small fixes.
---------
Co-authored-by: Alaister Young <a@alaisteryoung.com>
Charis