The bucket list endpoint now accepts pagination options. Adapting the
storage buckets table UI to paginate in pages of 100 buckets at a time,
with infinite loading/scrolling.
* Midway
* Hook up create/delete table, and update all iceberg RQs with mgmt API routes
* Add insert data CTA, update create table logic for decimal and fixed types
* Clean up
* Nit
* Fix test
* Refactor the iceberg queries and mutations to use getOrRefreshTemporaryApiKey.
* Fix the uses of the mutations.
* Hardcode the temp api keys queries at 60 seconds.
* When creating a s3 vectors fdw, add the target schema as a server option.
* Fix a bug in the row link, it didn't work when you middle click.
* Refactor the protected schema logic to include schemas from all fdws.
* Refactor the protected schema logic to include fdw type and name. Change the ProtectedSchemaWarning to show different messages for the 2 fdw types.
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Retrieve temporary API key within iceberg-namespace-tables-query
* Retrieve temporary API key within iceberg-namespace-table-delete-mutation
* Retrieve temporary API key within iceberg-namespaces-query
* Fix
* Retrieve temporary API key within iceberg-namespace-create-mutation
* Update based on comments
* Fix childProps in Admonition so that they're added as a prop to the main div.
* Replace the admonition with Alert in Wrapper tab page to add a gap between childs.
* Add s3 vectors fdw.
* Minor fix to FormSection.
* Update the fdw mutations to support passing in options.
* Refactor the vector flow to create fdws.
* Revert cron description change.
* If the bucket can't be created, don't create a fdw.
* Update/delete the fdw when deleting a table or a bucket.
* Minor fixes.
* Clean up the delete modal.
* Handle edge cases when missing a wrapper.
* Remove the admonition in the create bucket modal.
* Fix the loading state when creating a bucket.
* Fix the createWrapper sheet to work with s3 vectors.
* Fix undefined wrapperMeta issue.
* Create the schema when installing a wrapper.
* Tiny cleanup.
* Clean up unneeded useState. Create a wrapper only if the all conditions are met.
* Fix all comments.
* Add s3 vectors for docs.
* Add a link and fix the file name for S3 Vectors in docs.
* Hide the table editor button if the wrapper instance is missing.
* Small fixes.
* midway
* Hook up connect tables e2e
* Update isEnabled flag
* Change connectingStep to enum
* Nit
* Nit
* Nit
* Only show replication status if theres a publication
* Improve loading state in etl page
* Refactor + Clean up + Add long poll logic when adding more tables
* Initial work on vector buckets.
* Add more queries and UI.
* chore(studio): vector buckets contents (#39705)
* language
* match analytics buckets
* shared empty state
* proper loading state
* create dialog cleanup
* simplify copywriting
* proper details
* filtering
* use real data
* chore(studio): vector buckets polish A (#39759)
* empty state
* better checks
* bucket name in sheet
* fix typo
* sheet improvements
* rename vector index to vector table
* cleanup
* More work on the vector indexes.
* Add delete for bucket and indexes. Create index should now work.
* chore(studio): vector buckets polish B (#39794)
* loading
* delete modal
* fix modal behaviour
* Add modal for deleting vector buckets.
* Bring back the shimmering loader.
* chore(studio): vector buckets polish C (#39887)
* fix bucketname on deletion dialog
* empty states
* remove extraneous padding
* proper loading and not-found cases
* Minor fixes.
* Minor updates to the vector components.
* Add more instructions to the cursor rules.
* Some refactors and shifting files around
* Refactor analytics buckets detial to improve time to fcp
* Fix
* Refactors + simplify
* Last few nits, just leaving comments
* If the bucket is not empty, delete all indexes first.
---------
Co-authored-by: Danny White <3104761+dnywh@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* First pass of deprecating old storage UI components
* Clean up local-storage flag for new storage UI
* Add coming soon UI for analytics and vector buckets pages
* Set up warning states for analytics buckets
* fix spelling and margin
* Smol fix
* nit
* Surface wrappers upgrade check in UI instead of toast
* Nit clean up
* update tests
---------
Co-authored-by: Danny White <3104761+dnywh@users.noreply.github.com>
* Add custom types for queries, mutations and infinite queries.
* Migrate all queries to use the new type.
* Migrate all infinite queries to useCustomInfiniteQueryOptions.
* Migrate all mutations to use useCustomMutationOptions.
* Add type to all imports in `types` folder.
* Migrate all uses of invalidateQueries to use object syntax.
* Migrate the remainder of useInfiniteQuery.
* Migrate all setQueriesData.
* Migrate all fetchQuery uses.
* Migrate some leftover functions from RQ.
* Fix issues found by Charis.
* Update perms checking in audit logs
* Deprecate useCheckPermissions, useIsPermissionsLoaded and useCheckProjectPermissions as they're no longer used
* Rename useAsyncCheckProjectPermissions to useAsyncCheckPermissions
* Fix TS
* Wrap up remaining swapping of useCheckPermissions
* remove OrganizationSettingsLayout on billing, team, integrations and usage org pages
* Fix missing loading state in usage page
* Add comments for next step
* feat: basic clarity improvements
* feat: all switches
* feat: basic errors
* fix: simplify
* fix: properly separate sections
* fix: remove smart toggle closure
* fix: remember file size unit
* fix: validation
* fix: UpgradeToPro styles
* fix: UpgradeToPro styles for full-width
* feat: create modal match
* fix: tests
* Small fix for error message and desc for file size limit in create and edit bucket modals
* Prettify error message handling for storage settings
* Nit
* set default unit as MB
* dotted underline and default cursor for tooltip
* fix: default MB tests
* minor form style and copy improvements
* remove extraneous divs
* fix mime check on bucket creation
* Clean up + minor refactors including improving perms loading state in storage settings
* EditBucketModal ensure that file size limit unit defaults to MB if bucket doesnt have a file size limit
* Refactor EditBucketModal to use react query mutation
* Refactor
* Fix unit tests for edit bucket modal
* last round of clean up
* last fix
* Tiny unrelated fix
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Replace all usage of useProjectContext with useSelectedProjectQuery
* Replace all usage of useSelectedProject with useSelectedProjectQuery
* Replace all usage of useProjectByRef with useProjectByRefQuery
* Replace all usage of useSelectedOrganization with useSelectedOrganizationQuery
* Deprecate useSelectedProject, useSelectedOrganization, and useProjectByRef hooks
* Deprecate ProjecContext
* fix: use storage endpoint in S3 settings
* Update retrieval of endpoint for analytics bucket related components
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Change the import foreign schema to always create a schema.
* Fix the Iceberg wrapper.
* Refactor the create wrapper to always create a new schema.
* Remove unneeded props.
* Smol fixes
* Prevent double error toasts
* Smol fix
* Fix the wrapper creation to include the correct api key.
* Fix a bug with the new api keys.
* Handle both types of keys when fetching the iceberg namespaces.
* Add a field to all wrappers to hold the schema name which can import foreign schema.
* sq
* When importing a foreign schema, save the schema in a special field.
* Fix a type error.
* Fix importing foreign schema overriding wrapper server options with unencrypted values
* Add comment
* Handle duplicate and empty schemas when importing a foreign schema.
* Update the copy in the foreign schema wrappers.
* Remove unnecessary code.
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Update studio testing setup files
Improves API mocking type safety and polyfills browser APIs necessary to run tests with framer-motion components
* fix missing listen call for msw, resolve test type error
* fix imports
* Update studio testing setup files
Improves API mocking type safety and polyfills browser APIs necessary to run tests with framer-motion components
* fix missing listen call for msw, resolve test type error
* fix imports
* Move non-layout Storage related components to `components/interfaces`
* Fix paths
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
* Deprecate getAPIKeys from project settings v2, use new api keys endpoint instead
* use getPreferredKeys where appropriate
* Prevent usage of secret key for storage and realtime inspector
* Add dashboard API api-keys endpoint
* Simplify
* Disable edge functions test if legacy api keys are disabled
* Revert
* Fix graphiql
* Remove all usage of api keys from project settings, except DisplayApiSettings
* Update
* Fix
* Small fix for an undefined upload url.
* Fix the storage state initialization when the resumable url changes.
---------
Co-authored-by: Ivan Vasilov <vasilov.ivan@gmail.com>
* Migrate the CreateBucketModal to use RHF.
* Minor fixes for the create bucket modal.
* Use the bucket type from the API everywhere.
* Improve the types on some functions. Add "skip" mode to FDWCreateMutation.
* Expand the CreateBucketModal to explain what will happen when creating an iceberg bucket.
* Add a page for iceberg buckets.
* Add error states to the iceberg explorer.
* Fix the names for the FDWs.
* Add a dialog for setting up namespace.
* Restructure the new files.
* Add a download env button.
* Add warning to install the wrappers extension when creating an iceberg bucket.
* Add failover details when the FDW is not setup.
* Fix a lint error.
* Fix a type error.
* Minor fixes.
* Fix the server name.
* Add an icon for iceberg buckets.
* Fix the import foreign schema dialog.
* Make the setup wrapper button functional.
* Fix a bad useMemo dependency.
* Small changes to the iceberg bucket page.
* Minor fix for the edit wrapper sheet.
* Rename the files from kebab-case to PascalCase.
* Rename the files again to include Analytic instead of Iceberg.
* Rename Iceberg type to Analytic.
* Add a switch for creating namespace in the import foreign schema dialog.
* Fix the CreateBucketModal.
* Fix the delete modal feature.
* Fix the S3 keys in the FDW.
* Only create a namespace if the switch is true.
* Regenerate and fix the types.
* Fix the FDW create mutation to handle numbers in the FDW names.
* Make the icon smaller.
* Check whether the namespace exists, if it doesn't create it.
* Hide action from the analytic bucket which don't work.
* Invalidate namespaces when creating them.
* Add small explanation for the creation of namespaces.
* Minor fixes.
* Tons of changes to make the namespace feature work.
* Fix type errors.
* Fix bad import.
* Minor copy fixes.
* Replace the multiple cards with a table of namespaces.
* update copy icon
* tiny copy update
* Fix the empty state for foreign tables.
* Hide the analytics bucket option for self-hosted.
* Minor copy fixes.
* Expand the CTA on no namespaces state.
* More minor fixes.
* More small fixes.
---------
Co-authored-by: Alaister Young <a@alaisteryoung.com>
* Deprecate use of getWithTimeout, refactor BuildingState and RestoringState to use RQ
* Refactor profile-create-mutation to use data/fetchers, and edge-function-status-query to use fetch
* Shift post from lib/common/fetch, refactor bucket-object-download-mutation
* Address feedback
* Minor fix
* Smol fix
* Empty spaces.
* Remove prop drilling for selectedFilePreview and just use it from the store.
* Add a hook for getting a fetch url. Use it in all places where it's needed. Remove the file cache from the store.
* Simplify the onCopyUrl hook further.
* Fix some bugs in the preview pane.
* Update the design of the sonner toasts. Add the close button by default.
* Migrate studio and www apps to use the SonnerToaster.
* Migrate all toasts from studio.
* Migrate all leftover toasts in studio.
* Add a new toast component with progress. Use it in studio.
* Migrate the design-system app.
* Refactor the consent toast to use sonner.
* Switch docs to use the new sonner toasts.
* Remove toast examples from the design-system app.
* Remove all toast-related components and old code.
* Fix the progress bar in the toast progress component. Also make the bottom components vertically centered.
* Fix the width of the toast progress.
* Use text-foreground-lighter instead of muted for ToastProgress text
* Rename ToastProgress to SonnerProgress.
* Shorten the text in sonner progress.
* Use the correct classes for the close button. Add a const var for the default toast duration. Remove the custom width class from sonner.
* Set the position for all progress toasts to bottom right. Set the duration for all toasts to the default (when reusing a toast id from loading/progress toast, the duration is set to infinity).
* Fix the playwright tests.
* Refactor imports to use ui instead of @ui.
* Change all imports of react-hot-toast with sonner. These components were merged since the last commit to this branch.
* Remove react-hot-toast lib.
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
Co-authored-by: Jonathan Summers-Muir <MildTomato@users.noreply.github.com>
* Migrate all API calls to data folder.
* Remove unused mutation.
* MInor leftover type fixes.
* chore: Make the `StorageExplorerStore` fully typed (#23397)
* Migrate all API calls to data folder.
* Convert the poor man enums into proper enums.
* Add types to most of the methods.
* Add types around the methods for uploading files.
* Add types for file cache.
* Add types for buckets.
* Fix bunch of type errors.
* Add types to all store methods. Fix all types in components related to the store.
* Remove all anys from the Storage components.
* Fix the multiple selection.
* Fix the selected checkbox.
* Set all private methods to private and delete some unneeded code.
* Remove unneeded mutation.
* Update all mutations for storage explorer to follow template syntax
* Update apps/studio/components/to-be-cleaned/Storage/StorageExplorer/PreviewPane.tsx
Co-authored-by: Alaister Young <alaister@users.noreply.github.com>
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
Co-authored-by: Alaister Young <alaister@users.noreply.github.com>
Charis