Client

Using cloud.google.com/go/spanner v1.73.0.

Environment

GKE, go 1.23.8

Code and Dependencies

We run a go web server which makes calls back to Cloud Spanner using the client library.

Our application performs around 5-12 kQPS (i.e. 5000-12000 QPS) of the StreamingRead and 1-4k QPS of the other RPCs on our production deployment. This load originates over many docker containers (i.e. many instances of the client library).

Expected behavior

No errors logged to Cloud Logging.

Actual behavior

We receive many errors (around 10-30 per second), similar to:
One or more TimeSeries could not be written: timeSeries[0-23] (example metric.type="spanner.googleapis.com/internal/client/operation_latencies", metric.labels={"client_uid": "<omitted>", "directpath_used": "false", "directpath_enabled": "false", "database": "<omitted>", "status": "OK", "method": "Spanner.ExecuteStreamingSql", "client_name": "spanner-go/1.73.0"}): write for resource=spanner_instance_client{instance_config:unknown,client_hash:000171,instance_id:prod,location:us-central1} failed with: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: spanner.googleapis.com/internal/client/operation_latencies, Timestamps: {Youngest Existing: '2025/04/30-18:56:30.964', New: '2025/04/30-18:57:00.232'}}

Referencing methods:

• Spanner.ExecuteStreamingSql
• Spanner.StreamingRead
• Spanner.BeginTransaction
• and others.

And metrics:

• spanner.googleapis.com/internal/client/operation_count
• spanner.googleapis.com/internal/client/operation_latencies
• spanner.googleapis.com/internal/client/attempt_latencies

Additional context

This error has been occuring for at least 30 days, but we did not have it in the past. I have not gone back to investigate which client library version introduced it.

My Google LDAP is meiring@.