ARROW-14917: [C++] Implement GcsFileSystem::CreateDir #11842
Conversation
This PR implements `GcsFileSystem::CreateDir`. I also reviewed the documentation for the class, since now the implementation of the class, particularly how directories are emulated, is more than just notes and ideas.
|
The failure on |
pitrou
left a comment
There was a problem hiding this comment.
Thank you very much. A couple questions and suggestions below.
cpp/src/arrow/filesystem/gcsfs.h
Outdated
| /// directory. | ||
| /// - The class creates marker objects for a directory, using a trailing slash in the | ||
| /// marker names. For debugging purposes, the metadata and contents of these marker | ||
| /// objects indicate that they are markers created by this class. The class does |
There was a problem hiding this comment.
Isn't there any de facto standard for directory metadata markers?
There was a problem hiding this comment.
Kind of? The UI in for Google Cloud creates empty objects ending with /. The command-line utility does no such thing (e.g. gsutil cp -r deep-directory/ gs://my-bucket/foo would not create these markers). The client libraries do not create them either. Whether that amounts to a de facto standard, I cannot say.
Frankly the use of "folder emulation" causes more harm than good. It creates the impression that some things should work (e.g. directory renames, directory permissions, efficient non-recursive listing) when they don't. And the markers are sort of useless. If you want to list objects non-recursively, the API has native support for including any matching prefixes in the results, without the need for these markers 🤷
There was a problem hiding this comment.
Hmm, sorry, my question was about the custom metadata you added to these markers.
There was a problem hiding this comment.
Ahhhhh. No, the UI does not create any metadata, but ignores it too. I think the metadata is harmless and useful for debugging.
|
Please take another look. |
|
Benchmark runs are scheduled for baseline = 77722d9 and contender = 01186fc. 01186fc is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
|
Benchmark runs are scheduled for baseline = 77722d9 and contender = 01186fc. 01186fc is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
This PR implements
GcsFileSystem::CreateDir. I also reviewed thedocumentation for the class, since now the implementation of the class,
particularly how directories are emulated, is more than just notes and
ideas.