At Google Cloud, we know that you count on us to maintain the durability and integrity of your data at all times, both at rest and in transit. And now we’re making it easier for developers to take advantage of native data integrity features in Cloud Storage, by enabling end-to-end checksumming by default in all the Cloud Storage SDKs.
Like in any disk-based storage system, bits can flip anywhere in their journey, from the application all the way down to the disk. Cloud Storage has always let clients provide a checksum of the object data being uploaded, and receive a checksum of the data being downloaded. Also since its inception, Cloud Storage stores a checksum for every object in its metadata, regardless of how the object was uploaded into Cloud Storage.
But until recently, ensuring end-to-end data integrity required extra work on the part of developers to calculate and provide checksums to Cloud Storage. Cloud Storage always calculates the crc32 (32-bit cyclic redundancy check) of data it receives and ensures data stored on disk matches this checksum. When a client request includes the object’s checksum, Cloud Storage ensures that this checksum also matches. However, when an upload request doesn’t include a checksum, that upload is vulnerable to a bit flip while the data is in-flight, prior to the server-side checksum computation. Not all customers and clients enable client-side checksums by default, leaving data in this phase unprotected.
To address this gap, the latest version of all Cloud Storage SDKs now internally checksums data being uploaded and passes this checksum to Cloud Storage, if it’s not provided by the application. The SDKs also support verifying the object’s checksum when an object is being downloaded.
Finally, there are many use-cases where applications download select ranges of objects instead of the full object. When using Cloud Storage SDKs with our gRPC API to perform a range read, the SDKs take advantage of gRPC’s built-in end-to-end range checksum, using it to verify the data it receives.
We highly recommend updating to our latest SDK versions to take advantage of these important integrity features.
And now, let’s peek under the hood
Ensuring continuous “chain-of-custody” between the data and its associated checksum from your application down to the disk platter, with no gap where a bit flip could go unnoticed, is quite challenging. And it’s critical to get this right: at our current scale of hundreds of thousands of Cloud Storage frontends, bit flips aren’t theoretical and do happen from time to time.
For instance, consider this simple example: when Cloud Storage receives your data in its frontend, this data gets encrypted with per-object encryption keys. This involves a data copy: the plaintext data is passed through an encryptor into a new memory buffer containing ciphertext. Extremely rarely, a bit in the source or destination memory buffer flips during this process. However, at our scale, extremely rare things happen routinely.
In this situation, we maintain chain-of-custody by reversing the whole process: after encrypting the data (1), we calculate a checksum that protects the ciphertext. Then we decrypt the ciphertext (2), and if the resulting plaintext doesn’t match the original (3), we throw everything away and start over. This adds up to a lot of extra CPU time spent on encryption and checksumming, but it’s a necessary step to ensure data integrity.
