Skip to content

Improve user interaction with userfile ExternalStorage #51222

@adityamaru

Description

@adityamaru

With most of the core work for #47211 checked in we can now shift our focus on getting the interaction with userfile ExternalStorage to a usable state. This involves implementing the following items:

  • Implement userfile ls userfileBaseURI/glob to list all files matching the provided glob pattern.

  • Implement userfile delete userfileBaseURI/glob to delete all files matching the provided glob pattern.

  • Move the admin only guard on IMPORT to AWS, GCS, nodelocal, Azure with implicit AUTH code paths. This is because if the user has explicit AUTH or is using userfile then they should be able to IMPORT data as they have the required credentials.

Another important modification to the userfile URI was discussed with @mwang1026 and @dt. For all CLI commands, if the user does not provide an absolute userfileURI then we shall default the base URI to userfiles://defaultdb.public.userfiles_$user/.

Some examples of this implicit transformation:

  • upload path/to/a.file => upload path/to/a.file userfiles://defaultdb.public.userfiles_$user/a.file

  • upload path/to/a.file a.file =>
    upload path/to/a.file userfiles://defaultdb.public.userfiles_$user/a.file

  • upload path/to/a.file path/a.file =>
    upload path/to/a.file userfiles://defaultdb.public.userfiles_$user/path/a.file

These transformation rules will apply for upload, ls, and delete. On the flip side, if the user does provide an absolute userfileURI then we shall create the user storage with the specified db.schema.table_name_prefix.

Less urgent feature work:

  • BACKUP or EXPORT to userfiles, and then download to client.
  • Recursive directory upload/download
  • One shot IMPORT

Metadata

Metadata

Assignees

Labels

A-disaster-recoveryC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions