Feat(Storage): Enable full object checksum validation on JSON path

[thiyaguk09]

Enhanced Checksum Validation & Header Logic

This PR implements comprehensive MD5 and CRC32c checksum validation for object uploads, ensuring data integrity via the X-Goog-Hash header, improving data integrity verification. It refactors the upload architecture to handle hashes dynamically across different upload strategies.

Key Technical Changes

1. Core Library Enhancements (google-cloud-core)

• ResumableUploader & StreamableUploader: Added type-safe logic (int)($rangeEnd + 1) === (int)$size to ensure X-Goog-Hash is transmitted only on the final chunk/request, preventing intermediate validation errors.
• MultipartUploader: Standardized header merging to ensure hashes calculated by the connection layer are always included in single-shot uploads.
• Header Integrity: Refactored restOptions merging to ensure custom metadata and encryption headers are preserved alongside checksums.

2. Storage Package Improvements (google-cloud-storage)

• Automatic Hashing: Implemented logic to calculate missing MD5 or CRC32c hashes when the validate option is enabled.
• Validation Logic: Updated Bucket::upload() to honor user-provided checksums and prevent redundant re-calculation.
• Test Coverage: Added unit tests in BucketTest and RestTest to verify hash behavior in resumable, streamable, and multipart scenarios.

---

Note

CI "Lowest Dependencies" Failure: This failure occurs because the CI environment pulls the tagged version of google-cloud-core from Packagist instead of using the local changes in this PR. This will resolve once the Core changes are merged.

[gemini-code-assist]

Summary of Changes

Hello @thiyaguk09, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the storage client's object upload functionality by implementing comprehensive checksum validation. It ensures that both MD5 and CRC32c hashes are calculated for uploaded data and transmitted via the X-Goog-Hash header, improving data integrity verification. The changes also include logic to prevent redundant hash calculations if checksums are already provided, and new tests to cover these behaviors in both resumable and standard upload scenarios.

Highlights

• Checksum Generation: Both MD5 and CRC32c hashes are now calculated for uploaded objects when validation is enabled, ensuring comprehensive data integrity checks.
• X-Goog-Hash Header: A new X-Goog-Hash header is constructed from the calculated MD5 and CRC32c hashes and included in upload requests to Google Cloud Storage.
• Validation Method Enhancement: The chooseValidationMethod now respects pre-existing x-goog-hash headers, allowing users to provide their own checksums and skipping redundant hash calculations.
• Resumable Upload Testing: A new unit test has been added to verify that the X-Goog-Hash header is correctly applied only to the final chunk of resumable uploads.
• Object Insertion Testing: A new unit test confirms the proper calculation and inclusion of the X-Goog-Hash header during standard object insertion.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request enables full object checksum validation by sending both CRC32c and MD5 hashes in the x-goog-hash header for uploads. The implementation correctly calculates and adds these hashes. My review highlights a significant performance issue where the data stream is read twice, which could be problematic for large files. I've also included suggestions for minor code cleanup in both the main logic and the tests. The added tests are comprehensive and effectively validate the new functionality.

[thiyaguk09]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces checksum validation for uploads by adding the X-Goog-Hash header, which is a valuable feature. The changes are implemented across multipart, resumable, and streamable uploaders, with comprehensive tests covering the new functionality. I've identified a couple of areas for improvement: a logic issue with merging custom headers in the multipart uploader, and an inconsistency in the casing of the X-Goog-Hash header. Addressing these points will enhance the robustness and maintainability of the code.

[thiyaguk09]

/gemini summary

[gemini-code-assist]

Summary of Changes

This pull request introduces significant enhancements across two Google Cloud client libraries: Storage and BigQuery. For Google Cloud Storage, it implements comprehensive checksum validation for object uploads, ensuring that MD5 and CRC32c hashes are calculated and transmitted via the 'X-Goog-Hash' header, with careful handling for resumable and streamable uploads to send the header only on the final data chunk. Concurrently, for Google Cloud BigQuery, it integrates the stateless query API, allowing certain queries to bypass traditional job creation for improved efficiency.

Highlights

• Storage Checksum Validation: Implemented calculation of both MD5 and CRC32c hashes for uploaded objects, which are then included in the 'X-Goog-Hash' header.
• Optimized Hash Calculation: Added logic to prevent redundant hash calculations if checksums are already provided by the user, including checking for existing 'X-Goog-Hash' headers.
• Conditional 'X-Goog-Hash' for Uploads: Ensured that the 'X-Goog-Hash' header is only sent with the final chunk in resumable and streamable upload scenarios to optimize network traffic and server processing.
• BigQuery Stateless Query API: Introduced a new execution path in 'BigQueryClient' to leverage the stateless query API for compatible 'QueryJobConfiguration' objects, avoiding job creation overhead for simple queries.
• Query Configuration Enhancements: Added methods to 'QueryJobConfiguration' to determine query statelessness and convert job configurations into the 'QueryRequest' format required by the stateless API.
• Comprehensive Testing: Expanded unit and snippet tests to cover the new checksum validation behaviors in Storage and the stateless query execution in BigQuery, ensuring robustness and correctness.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• BigQuery/src/BigQueryClient.php
  • Refactored the 'runQuery' method to introduce a new stateless query execution path.
  • Implemented conditional logic to use the 'connection->query' endpoint directly for stateless queries.
  • Extracted the job creation logic into a new private 'createJob' method for reusability.
• BigQuery/src/QueryJobConfiguration.php
  • Added a 'JOB_CREATION_MODE_OPTIONAL' constant and an 'isJobIdGenerated' property.
  • Implemented 'isStateless()' method to determine if a query configuration is compatible with the stateless API.
  • Added 'toQueryRequest()' method to convert the job configuration into the format required by the stateless query API.
• BigQuery/src/QueryResults.php
  • Introduced an 'isStateless' property to track the nature of query results.
  • Modified the 'reload' method to return cached information directly if the query was stateless, avoiding unnecessary API calls.
• BigQuery/tests/Snippet/BigQueryClientTest.php
  • Updated snippet tests to use the 'connection->query' method instead of 'insertJob' for stateless query scenarios.
  • Adjusted expected arguments for the 'query' method to match the new stateless API request format.
• BigQuery/tests/Unit/BigQueryClientTest.php
  • Added new unit tests for stateless query execution, including cases where the query endpoint returns a complete job or a job reference.
  • Introduced a data provider to test various non-stateless query options that should fall back to the stateful path.
  • Added a test to verify that queries with pre-defined job IDs use the stateful path.
• Core/src/Upload/MultipartUploader.php
  • Merged custom headers provided in 'restOptions' into the request headers for multipart uploads.
• Core/src/Upload/ResumableUploader.php
  • Implemented logic to include the 'X-Goog-Hash' header only on the final chunk of a resumable upload.
• Core/src/Upload/StreamableUploader.php
  • Implemented logic to include the 'X-Goog-Hash' header only on the final request of a streamable upload.
• Core/tests/Unit/Upload/MultipartUploaderTest.php
  • Added a test case to verify that custom headers are correctly applied during multipart uploads.
• Core/tests/Unit/Upload/ResumableUploaderTest.php
  • Added tests to confirm that the 'X-Goog-Hash' header is sent only on the final chunk of resumable uploads and not on intermediate chunks.
• Core/tests/Unit/Upload/StreamableUploaderTest.php
  • Added tests to confirm that the 'X-Goog-Hash' header is sent only on the final step of streamable uploads.
• Storage/src/Connection/Rest.php
  • Implemented calculation of both MD5 and CRC32c hashes for uploaded data when validation is enabled.
  • Constructed the 'X-Goog-Hash' header from the calculated hashes.
  • Integrated the 'X-Goog-Hash' header and other custom headers into the uploader options.
  • Updated 'chooseValidationMethod' to also check for pre-existing 'X-Goog-Hash' headers to prevent redundant hash calculations.
• Storage/tests/Unit/BucketTest.php
  • Added a test to verify that resumable uploads initiated via the Bucket client correctly send the 'X-Goog-Hash' header only on the final chunk.
• Storage/tests/Unit/Connection/RestTest.php
  • Added a test for 'insertObject' to verify the correct calculation and inclusion of the 'X-Goog-Hash' header.
  • Updated the 'validationMethod' data provider to include test cases where 'X-Goog-Hash' is provided in headers.
  • Added a helper proxy method 'getCrcFromStreamForTest' to expose a private method for testing purposes.

Activity

• gemini-code-assist[bot] posted an initial summary of the pull request.
• gemini-code-assist[bot] provided a high-priority review comment regarding the efficiency of hash calculation, suggesting a single-pass approach.
• gemini-code-assist[bot] provided a medium-priority review comment about a redundant null-coalescing operator in 'Rest.php'.
• gemini-code-assist[bot] provided a medium-priority review comment about a duplicate assertion in 'RestTest.php'.
• gemini-code-assist[bot] provided a medium-priority review comment about a PHPDoc typo in 'RestTest.php'.
• thiyaguk09 requested a review from gemini-code-assist[bot].
• gemini-code-assist[bot] provided a high-priority review comment about an incorrect 'array_merge' order in 'MultipartUploader.php', which could overwrite custom headers.
• gemini-code-assist[bot] provided a medium-priority review comment about inconsistent casing for the 'X-Goog-Hash' header in 'Rest.php'.
• gemini-code-assist[bot] provided a medium-priority review comment about inconsistent casing for the 'X-Goog-Hash' header check in 'Rest.php'.
• gemini-code-assist[bot] provided a medium-priority review comment about inconsistent casing for the 'X-Goog-Hash' header in a test case in 'RestTest.php'.
• thiyaguk09 requested a summary from gemini-code-assist[bot].