-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Improve user interaction with userfile ExternalStorage #51222
Description
With most of the core work for #47211 checked in we can now shift our focus on getting the interaction with userfile ExternalStorage to a usable state. This involves implementing the following items:
-
Implement
userfile ls userfileBaseURI/globto list all files matching the provided glob pattern. -
Implement
userfile delete userfileBaseURI/globto delete all files matching the provided glob pattern. -
Move the admin only guard on IMPORT to AWS, GCS, nodelocal, Azure with implicit AUTH code paths. This is because if the user has explicit AUTH or is using userfile then they should be able to IMPORT data as they have the required credentials.
Another important modification to the userfile URI was discussed with @mwang1026 and @dt. For all CLI commands, if the user does not provide an absolute userfileURI then we shall default the base URI to userfiles://defaultdb.public.userfiles_$user/.
Some examples of this implicit transformation:
-
upload path/to/a.file => upload path/to/a.file userfiles://defaultdb.public.userfiles_$user/a.file
-
upload path/to/a.file a.file =>
upload path/to/a.file userfiles://defaultdb.public.userfiles_$user/a.file -
upload path/to/a.file path/a.file =>
upload path/to/a.file userfiles://defaultdb.public.userfiles_$user/path/a.file
These transformation rules will apply for upload, ls, and delete. On the flip side, if the user does provide an absolute userfileURI then we shall create the user storage with the specified db.schema.table_name_prefix.
Less urgent feature work:
- BACKUP or EXPORT to userfiles, and then download to client.
- Recursive directory upload/download
- One shot IMPORT