Description

This pull request significantly enhances data integrity for object uploads by introducing comprehensive client-side checksum validation. It ensures that data uploaded through the JSON API path remains consistent by verifying both client-provided and client-calculated CRC32C and MD5 hashes against the server's reported hashes. This mechanism proactively detects and prevents silent data corruption, providing a more reliable upload experience.

Impact

The primary impact is a substantial improvement in data integrity and reliability for object uploads, particularly those utilizing the JSON API path (e.g., small single-chunk uploads).

• Prevents Silent Data Corruption: By validating client-side hashes (calculated or user-provided) against the server's reported hashes, the SDK proactively detects and stops data corruption that might otherwise go unnoticed.
• Enhanced Reliability: The stream is immediately destroyed upon hash mismatch, preventing the client from assuming a corrupted file upload was successful.
• Feature Parity: Enables the use of the X-Goog-Hash header, which is essential for ensuring the integrity of the data payload sent to the server.
• API Usage: Introduces new configuration options (clientCrc32c, clientMd5Hash, crc32c, md5) that allow users fine-grained control over checksum validation behavior.

Testing

• Unit and Integration Tests Added? Yes. Extensive test coverage was added and expanded in test/resumable-upload.ts and system-test/kitchen.ts. This includes:

  • Verification of X-Goog-Hash header injection with calculated and client-provided hashes in single and multi-chunk uploads.
  • Validation tests to confirm that the stream is correctly destroyed on both CRC32C and MD5 checksum mismatches between the client and server.
  • A dedicated describe block for Validation of Client Checksums Against Server Response with various success and failure scenarios.

• Were any tests changed? Yes. Tests were expanded and refactored (e.g., checksum application logic tests).

• Are any breaking changes necessary? No. This change introduces new features and configuration options but does not appear to break existing functionality. The changes are largely additive and internal refactoring of hash handling and validation logic.

Additional Information

• Refactoring for Robustness: The HashStreamValidator was improved by adding the md5Digest getter, which centralizes the MD5 calculation and caching logic. The related fix in _flush prevents a potential race condition where calling digest() multiple times would cause a runtime error.
• Internal Consistency: Private helper methods (#validateChecksum, #applyChecksumHeaders) were introduced in resumable-upload.ts to simplify and clarify the hash handling logic, especially ensuring checksum headers are applied correctly across different upload paths.
• Dependency on Configuration: Proper client-side validation requires users to explicitly enable calculation (crc32c: true, md5: true) or provide the pre-calculated hashes.

Checklist

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease
• Appropriate docs were updated
• Appropriate comments were added, particularly in complex areas or places that require background
• No new warnings or issues will be generated from this change

Fixes #