feat(storage): capture metadata info in downloads #7694

coryan · 2021-12-04T17:59:00Z

A lot of useful metadata information is included the the metadata
headers for a GCS download over REST, and as an initial Object proto
for gRPC. This information is needed to perform consistent reads from
versioned objects (or detect inconsistent reads from objects without
versioning). I will need this information as part of the changes for
Arrow too.

Fixes #7677

This change is

google-cloud-cpp-bot · 2021-12-04T18:01:30Z

Google Cloud Build Logs
For commit: f31658cdc4b6bb29e802185cfb3b2a372b74ebb2

Public Access Build Logs
Cloud Build Console (requires auth)
Kokoro Logs (requires auth)

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

google-cloud-cpp-bot · 2021-12-04T18:26:13Z

Google Cloud Build Logs
For commit: 402cd0c60c01c96a2b9aa03b29b56fd3c77d013b

Public Access Build Logs
Cloud Build Console (requires auth)
Kokoro Logs (requires auth)

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

google-cloud-cpp-bot · 2021-12-05T13:12:28Z

Google Cloud Build Logs
For commit: 17bae99eb9c31975360c73b1b0ecfaba1e033925

Public Access Build Logs
Cloud Build Console (requires auth)
Kokoro Logs (requires auth)

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

google-cloud-cpp-bot · 2021-12-07T15:09:12Z

Google Cloud Build Logs
For commit: 34dec08e2865d08214097146b71e80a142047c2d

Public Access Build Logs
Cloud Build Console (requires auth)
Kokoro Logs (requires auth)

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

google-cloud-cpp-bot · 2021-12-07T16:20:34Z

Google Cloud Build Logs
For commit: e4a1f9300f520a2b943d2bd14437af5aa2935497

Public Access Build Logs
Cloud Build Console (requires auth)
Kokoro Logs (requires auth)

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

google-cloud-cpp-bot · 2021-12-07T17:53:42Z

Google Cloud Build Logs
For commit: a51f21597819964a8e93e9d5790ecf31801e9bca

Public Access Build Logs
Cloud Build Console (requires auth)
Kokoro Logs (requires auth)

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

google-cloud-cpp-bot · 2021-12-07T20:11:46Z

Google Cloud Build Logs
For commit: 0e01d084a5ba3fcba150a9f803c348448354c59b

Public Access Build Logs
Cloud Build Console (requires auth)
Kokoro Logs (requires auth)

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

google-cloud-cpp-bot · 2021-12-07T21:18:08Z

Google Cloud Build Logs
For commit: 8e57de5e5fafb29a1055f2700411fd2822a04e5b

Public Access Build Logs
Cloud Build Console (requires auth)
Kokoro Logs (requires auth)

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

google-cloud-cpp-bot · 2021-12-07T22:04:27Z

Google Cloud Build Logs
For commit: c08521dc84a71694519fb256170b73820c389c01

Public Access Build Logs
Cloud Build Console (requires auth)
Kokoro Logs (requires auth)

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

codecov · 2021-12-07T22:14:32Z

Codecov Report

Merging #7694 (c08521d) into main (cc5834b) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head c08521d differs from pull request most recent head 3e5ea1e. Consider uploading reports for the commit 3e5ea1e to get more accurate results

@@           Coverage Diff           @@
##             main    #7694   +/-   ##
=======================================
  Coverage   95.29%   95.30%           
=======================================
  Files        1256     1256           
  Lines      113697   113785   +88     
=======================================
+ Hits       108350   108442   +92     
+ Misses       5347     5343    -4

Impacted Files	Coverage Δ
google/cloud/storage/internal/object_read_source.h	`66.66% <ø> (ø)`
...le/cloud/storage/internal/curl_download_request.cc	`90.10% <100.00%> (+0.84%)`	⬆️
.../cloud/storage/internal/grpc_object_read_source.cc	`100.00% <100.00%> (ø)`
...le/cloud/storage/internal/object_read_streambuf.cc	`97.93% <100.00%> (+0.08%)`	⬆️
...gle/cloud/storage/internal/object_read_streambuf.h	`100.00% <100.00%> (ø)`
google/cloud/storage/object_read_stream.h	`100.00% <100.00%> (ø)`
...rage/tests/object_read_headers_integration_test.cc	`95.18% <100.00%> (+8.51%)`	⬆️
...ud/spanner/integration_tests/client_stress_test.cc	`86.18% <0.00%> (+0.65%)`	⬆️
...bigtable/examples/bigtable_hello_instance_admin.cc	`83.51% <0.00%> (+2.19%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc5834b...3e5ea1e. Read the comment docs.

A lot of useful metadata information is included the the metadata headers for a GCS download over REST, and as an initial `Object` proto for gRPC. This information is needed to perform consistent reads from versioned objects (or detect inconsistent reads from objects without versioning). I will need this information as part of the changes for Arrow too.

google-cloud-cpp-bot · 2021-12-09T13:10:30Z

Google Cloud Build Logs
For commit: 3e5ea1ee1e5feac12dc6d7e7ab27ddc826ca22bf

Public Access Build Logs
Cloud Build Console (requires auth)
Kokoro Logs (requires auth)

ℹ️ NOTE: Kokoro logs are linked from "Details" below.

coryan · 2021-12-09T14:50:03Z

After too many iterations this is now ready. PTaL.

dbolduc · 2021-12-09T16:51:45Z

google/cloud/storage/tests/object_read_headers_integration_test.cc

+
+  auto is = client->ReadObject(bucket_name(), object_name,
+                               Generation(insert->generation()));
+  EXPECT_EQ(insert->generation(), is.generation().value_or(0));


why are the .value_or()s necessary?

Nevermind, I see it now.

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Dec 4, 2021

product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Dec 4, 2021

coryan added do not merge Indicates a pull request not ready for merge, due to either quality or timing. and removed cla: yes This human has signed the Contributor License Agreement. api: storage Issues related to the Cloud Storage API. labels Dec 4, 2021

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Dec 4, 2021

product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Dec 5, 2021

coryan force-pushed the feature-storage-expose-metadata-info branch from 402cd0c to 17bae99 Compare December 5, 2021 13:10

coryan force-pushed the feature-storage-expose-metadata-info branch from 17bae99 to 34dec08 Compare December 7, 2021 15:07

coryan force-pushed the feature-storage-expose-metadata-info branch from 34dec08 to e4a1f93 Compare December 7, 2021 16:18

coryan force-pushed the feature-storage-expose-metadata-info branch from e4a1f93 to a51f215 Compare December 7, 2021 17:51

coryan force-pushed the feature-storage-expose-metadata-info branch from a51f215 to 0e01d08 Compare December 7, 2021 20:09

coryan removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Dec 7, 2021

coryan force-pushed the feature-storage-expose-metadata-info branch from c08521d to 3e5ea1e Compare December 9, 2021 13:08

coryan marked this pull request as ready for review December 9, 2021 14:49

coryan requested a review from a team as a code owner December 9, 2021 14:49

dbolduc approved these changes Dec 9, 2021

View reviewed changes

coryan merged commit 9abc956 into googleapis:main Dec 9, 2021

coryan deleted the feature-storage-expose-metadata-info branch December 9, 2021 17:52

feat(storage): capture metadata info in downloads #7694

feat(storage): capture metadata info in downloads #7694

Uh oh!

Conversation

coryan commented Dec 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-cloud-cpp-bot commented Dec 4, 2021

Uh oh!

google-cloud-cpp-bot commented Dec 4, 2021

Uh oh!

google-cloud-cpp-bot commented Dec 5, 2021

Uh oh!

google-cloud-cpp-bot commented Dec 7, 2021

Uh oh!

google-cloud-cpp-bot commented Dec 7, 2021

Uh oh!

google-cloud-cpp-bot commented Dec 7, 2021

Uh oh!

google-cloud-cpp-bot commented Dec 7, 2021

Uh oh!

google-cloud-cpp-bot commented Dec 7, 2021

Uh oh!

google-cloud-cpp-bot commented Dec 7, 2021

Uh oh!

codecov bot commented Dec 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

google-cloud-cpp-bot commented Dec 9, 2021

Uh oh!

coryan commented Dec 9, 2021

Uh oh!

dbolduc Dec 9, 2021

Choose a reason for hiding this comment

Uh oh!

dbolduc Dec 9, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coryan commented Dec 4, 2021 •

edited

Loading

codecov bot commented Dec 7, 2021 •

edited

Loading