blob/gcsblob: support the gRPC API and Rapid Storage (zonal) buckets

[stanhu]

This pull request adds support for the Cloud Storage gRPC API to gcsblob, and on top of it, support for Rapid Storage (zonal) buckets.

Two new fields on Options, each with a URL query parameter:

Option	URL parameter	Effect
UseGRPC	grpc=true	Builds the client with storage.NewGRPCClient instead of storage.NewClient.
UseZonalAPIs	zonal=true	Adds experimental.WithZonalBucketAPIs(). Implies UseGRPC.

b, err := blob.OpenBucket(ctx, "gs://my-rapid-bucket?zonal=true")

Why Rapid Storage needs more than a transport switch

Zonal buckets accept only appendable object uploads. Every ordinary write is rejected, on both transports:

gs://bucket?grpc=true
  InvalidArgument: This bucket type only supports appendable objects
gs://bucket
  googleapi: Error 400: This bucket requires appendable objects

An appendable object is uploaded over a bidirectional stream. It becomes visible as soon as the first bytes are flushed and stays open for further writes until something finalizes it. Ordinary uploads, one-shot or resumable, produce an object only once the whole payload has been sent. Zonal buckets support the appendable form and nothing else.

So two things are needed:

1. experimental.WithZonalBucketAPIs() on the client. It makes ObjectHandle.NewWriter default Writer.Append to true, which selects the appendable upload path, and switches reads to the bidirectional API.

2. Writer.FinalizeOnClose on the writer. An appendable object stays open by default, so Close leaves an unfinalized object exposing only the bytes that happened to be flushed. For a payload small enough to fit in one buffer that is a zero-length object, even though Close returned no error.

Reads already worked on both transports without any change.

Notes for review

FinalizeOnClose is set unconditionally in NewTypedWriter. This looked risky to me too, so I verified that:

• The field never appears in http_client.go, so the JSON/HTTP writer ignores it.
• In grpc_writer.go the only read is gRPCAppendBidiWriteBufferSender.send, and pickBufferSender returns that sender only when Writer.Append is set. It is therefore dead code on every path gcsblob uses today.

Setting it unconditionally rather than gating it on UseZonalAPIs is deliberate: it also fixes a pre-existing issue, where passing experimental.WithZonalBucketAPIs() through Options.ClientOptions already reached the client but produced zero-length objects. Callers who genuinely want an object left open for later appends can set it back to false from WriterOptions.BeforeWrite.

gRPC is not a free speedup. Measured from an e2-standard-4 in us-central1-c, 1 KiB objects, each read exactly once, n=200:

Bucket	JSON/HTTP p50	JSON/HTTP p99	gRPC p50	gRPC p99
Rapid Storage (zonal), zonal=true	N/A	N/A	16.9 ms	33.5 ms
NAM4 dual-region	39.0 ms	55.3 ms	43.3 ms	99.9 ms
US multi-region	52.3 ms	115.7 ms	63.1 ms	136.3 ms

Plain gRPC was slower than JSON/HTTP on both standard buckets, so the UseGRPC docs say to measure before enabling it. Against the zonal bucket it is 2.3x faster than NAM4 and 3.1x faster than multi-region, with a tighter tail.

Testing

go test ./blob/gcsblob/ passes; the existing replay tests are unaffected by the writer change. New unit tests cover the zonal parameter, the invalid value, and the zonal=true plus grpc=false conflict.

Beyond that I exercised the branch against a real Rapid Storage bucket in the same zone as the client. Writes, reads, attributes, range reads and 1 MiB writes all succeed with zero errors, where they fail outright on master. I
also confirmed with ObjectAttrs that objects written through gs://bucket?zonal=true come back finalized (size=23, non-zero Finalized), against size=0 and a zero Finalized without the FinalizeOnClose line.

What I could not test: there is no replay coverage for the gRPC path. gcsblob's harness is httpreplay only. The repository does have grpcreplay wired up in internal/testing/setup.NewGCPgRPCConn, used by gcpkms, gcppubsub and gcpfirestore, so the machinery exists, but plumbing it into the gcsblob conformance tests is a larger piece of work.

What this PR does not do

Sub-millisecond is possible, but under certain conditions. You only get it on later reads of one particular object, from a process that has kept a bidirectional read stream open to that object. The first read of any given object costs 10-20 ms no matter which API you use.

Getting there means keeping state alive between reads, and that requires significantly more changes.

[vangent]

Can you merge with HEAD? I think that might fix the golangci-lint problem.

[vangent]

So, the "client *gcp.HTTPClient" passed in to the constructor here is getting ignored? That seems odd.

Maybe enforce that client is nil to make that more clear?

[vangent]

OpenKeeper for KMS has a constructor that takes a client (so that the caller can do whatever with it); maybe that's a better pattern here?

I.e., the grpc=true URL option is fine, and controls what the URL opener does, but there's no "UseGRPC" Option; instead, there are two separate OpenBucket constructors, one for HTTP and one for gRPC, where the latter takes a storage.Client, and we provide a Dial to create it pre-wrapped similar to KMS ("cloudkms.NewKeyManagementClient(ctx, option.WithTokenSource(ts), useragent.ClientOption("secrets"))").

[vangent]

I don't want to merge this without running it through the conformance test.

You should be able to make a new function here, TestConformanceGRPC (and maybe another one, TestConformanceGRPCZonal), that uses a different newHarness-equivalent function (or refactor newHarness) that creates a gRPC client etc.

To generate the golden files locally you'll need to update the constants at the top of the file and run with --record. I'll ask you to revert the constant changes before merging, and I'll re-generate the golden files with our bucket after that.

[stanhu]

TestConformanceGRPCZonal currently fails for a number of reasons:

TestCopy/Works
  got unexpected error copying blob: (code=InvalidArgument):
  Rapid storage class objects do not support rewrite.

TestListDelimiters/backslash
  (code=InvalidArgument): Invalid argument.        # non-"/" delimiter on an HNS bucket

TestWrite/write_with_explicit_ContentType_overrides_discovery
  NewWriter or Close got err (code=ResourceExhausted):
  The object <rapid bucket>/blob-for-reading exceeded the rate limit for object
  mutation operations (create, update, and delete).

https://docs.cloud.google.com/storage/docs/rapid/rapid-bucket mentions that object rewrites are not supported (https://docs.cloud.google.com/storage/docs/json_api/v1/objects/rewrite).

Rapid Storage also requires / as the delimeter (https://cloud.google.com/blog/products/storage-data-transfer/understanding-new-cloud-storage-hierarchical-namespace), so those tests fail as well.

For now I'll omit TestConformanceGRPCZonal until there's a better way to selectively disable conformance tests.