spanner: Close causes /google.monitoring.v3.MetricService/CreateServiceTimeSeries error internally

## Client

Spanner

## Environment

Alpine Docker on GKE
Go version: 1.24

## Code and Dependencies

```go
package main

import (
	"context"
	"flag"
	"log"
	"time"

	"cloud.google.com/go/spanner"
	"google.golang.org/api/option"
	"google.golang.org/grpc"
)

var db string

func init() {
	flag.StringVar(&db, "db", "", "db uri.")
}
func main() {
	flag.Parse()
	ctx := context.Background()
	cli, err := spanner.NewClientWithConfig(
		ctx,
		db,
		spanner.ClientConfig{SessionPoolConfig: spanner.DefaultSessionPoolConfig, DisableRouteToLeader: false},
		option.WithGRPCDialOption(grpc.WithUnaryInterceptor(uinterceptor)),
		option.WithGRPCDialOption(grpc.WithStreamInterceptor(sinterceptor)),
	)
	if err != nil {
		log.Fatal(err)
	}
	go func(ctx context.Context, cli *spanner.Client) {
		for {
			select {
			case <-ctx.Done():
				break
			default:
				singerColumns := []string{"SingerId", "FirstName", "LastName"}
				m := []*spanner.Mutation{
					spanner.InsertOrUpdate("Singers", singerColumns, []interface{}{"a", "Melissa", "Garcia"}),
				}
				_, err := cli.Apply(ctx, m)
				if err != nil {
					log.Printf("apply failed. %v\n", err)
				}
				time.Sleep(10 * time.Second)
			}
		}
	}(ctx, cli)
	time.Sleep(1*time.Minute + 1*time.Second)
	log.Print("call Close.\n")
	cli.Close()
	time.Sleep(5 * time.Second)
}

func uinterceptor(
	ctx context.Context,
	method string,
	req, reply interface{},
	cc *grpc.ClientConn,
	invoker grpc.UnaryInvoker,
	opts ...grpc.CallOption,
) error {
	log.Printf("intercept unary %s\n", method)
	err := invoker(ctx, method, req, reply, cc, opts...)
	if err != nil && method == "/google.monitoring.v3.MetricService/CreateServiceTimeSeries" {
		log.Printf("error occuerd. %v\n", err)
	}
	return err
}

func sinterceptor(ctx context.Context, desc *grpc.StreamDesc, cc *grpc.ClientConn, method string, streamer grpc.Streamer, opts ...grpc.CallOption) (grpc.ClientStream, error) {
	log.Printf("intercept stream:%s\n", method)
	s, err := streamer(ctx, desc, cc, method, opts...)
	if err != nil {
		return nil, err
	}
	return &wrappedStream{s, method}, nil
}

type wrappedStream struct {
	grpc.ClientStream

	method string
}

func (w *wrappedStream) RecvMsg(m interface{}) error {
	log.Printf("RecvMsg:%s\n", w.method)
	return w.ClientStream.RecvMsg(m)
}
func (w *wrappedStream) SendMsg(m interface{}) error {
	log.Printf("SendMsg:%s\n", w.method)
	return w.ClientStream.SendMsg(m)
}
```

<details>
  <summary>go.mod</summary>

```text
module github.com/nktks/test/spanbuiltinmetrics

go 1.24.1

require cloud.google.com/go/spanner v1.79.0

require (
	cel.dev/expr v0.19.2 // indirect
	cloud.google.com/go v0.120.0 // indirect
	cloud.google.com/go/auth v0.15.0 // indirect
	cloud.google.com/go/auth/oauth2adapt v0.2.7 // indirect
	cloud.google.com/go/compute/metadata v0.6.0 // indirect
	cloud.google.com/go/monitoring v1.24.1 // indirect
	github.com/GoogleCloudPlatform/grpc-gcp-go/grpcgcp v1.5.2 // indirect
	github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.27.0 // indirect
	github.com/cespare/xxhash/v2 v2.3.0 // indirect
	github.com/cncf/xds/go v0.0.0-20250121191232-2f005788dc42 // indirect
	github.com/envoyproxy/go-control-plane/envoy v1.32.4 // indirect
	github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect
	github.com/felixge/httpsnoop v1.0.4 // indirect
	github.com/go-logr/logr v1.4.2 // indirect
	github.com/go-logr/stdr v1.2.2 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/google/s2a-go v0.1.9 // indirect
	github.com/google/uuid v1.6.0 // indirect
	github.com/googleapis/enterprise-certificate-proxy v0.3.6 // indirect
	github.com/googleapis/gax-go/v2 v2.14.1 // indirect
	github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
	go.opencensus.io v0.24.0 // indirect
	go.opentelemetry.io/auto/sdk v1.1.0 // indirect
	go.opentelemetry.io/contrib/detectors/gcp v1.35.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.59.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.59.0 // indirect
	go.opentelemetry.io/otel v1.35.0 // indirect
	go.opentelemetry.io/otel/metric v1.35.0 // indirect
	go.opentelemetry.io/otel/sdk v1.35.0 // indirect
	go.opentelemetry.io/otel/sdk/metric v1.35.0 // indirect
	go.opentelemetry.io/otel/trace v1.35.0 // indirect
	golang.org/x/crypto v0.36.0 // indirect
	golang.org/x/net v0.37.0 // indirect
	golang.org/x/oauth2 v0.28.0 // indirect
	golang.org/x/sync v0.12.0 // indirect
	golang.org/x/sys v0.31.0 // indirect
	golang.org/x/text v0.23.0 // indirect
	golang.org/x/time v0.11.0 // indirect
	google.golang.org/api v0.227.0 // indirect
	google.golang.org/genproto v0.0.0-20250303144028-a0af3efb3deb // indirect
	google.golang.org/genproto/googleapis/api v0.0.0-20250313205543-e70fdf4c4cb4 // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20250313205543-e70fdf4c4cb4 // indirect
	google.golang.org/grpc v1.71.0 // indirect
	google.golang.org/protobuf v1.36.6 // indirect
)
```

</details>

<details><summary>Execute result (Click me)</summary>

```
> go run main.go -db "projects/${PROJECT}/instances/${INSTANCE}/databases/${DATABASE}"
2025/04/20 21:35:55 intercept unary /google.spanner.v1.Spanner/BatchCreateSessions
2025/04/20 21:35:55 intercept unary /google.spanner.v1.Spanner/BatchCreateSessions
2025/04/20 21:35:55 intercept unary /google.spanner.v1.Spanner/BatchCreateSessions
2025/04/20 21:35:55 intercept unary /google.spanner.v1.Spanner/BatchCreateSessions
2025/04/20 21:35:56 intercept unary /google.spanner.v1.Spanner/BeginTransaction
2025/04/20 21:35:56 intercept unary /google.spanner.v1.Spanner/Commit
2025/04/20 21:36:06 intercept unary /google.spanner.v1.Spanner/BeginTransaction
2025/04/20 21:36:06 intercept unary /google.spanner.v1.Spanner/Commit
2025/04/20 21:36:16 intercept unary /google.spanner.v1.Spanner/BeginTransaction
2025/04/20 21:36:16 intercept unary /google.spanner.v1.Spanner/Commit
2025/04/20 21:36:26 intercept unary /google.spanner.v1.Spanner/BeginTransaction
2025/04/20 21:36:26 intercept unary /google.spanner.v1.Spanner/Commit
2025/04/20 21:36:36 intercept unary /google.spanner.v1.Spanner/BeginTransaction
2025/04/20 21:36:36 intercept unary /google.spanner.v1.Spanner/Commit
2025/04/20 21:36:46 intercept unary /google.spanner.v1.Spanner/BeginTransaction
2025/04/20 21:36:46 intercept unary /google.spanner.v1.Spanner/Commit
2025/04/20 21:36:55 intercept unary /google.monitoring.v3.MetricService/CreateServiceTimeSeries
2025/04/20 21:36:56 call Close.
2025/04/20 21:36:56 intercept unary /google.monitoring.v3.MetricService/CreateServiceTimeSeries
2025/04/20 21:36:56 intercept unary /google.spanner.v1.Spanner/BeginTransaction
2025/04/20 21:36:56 intercept unary /google.spanner.v1.Spanner/Commit
2025/04/20 21:36:56 error occuerd. rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: timeSeries[0-11]: write for resource=spanner_instance_client{location:global,instance_id:xxxx,client_hash:00011c,instance_config:unknown} failed with: One or more points were written more frequently than the maximum sampling period configured for the metric.
2025/04/20 21:36:56 intercept unary /google.monitoring.v3.MetricService/CreateServiceTimeSeries
2025/04/20 21:36:57 error occuerd. rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: timeSeries[0-11]: write for resource=spanner_instance_client{location:global,client_hash:00011c,instance_id:xxxx,instance_config:unknown} failed with: One or more points were written more frequently than the maximum sampling period configured for the metric.
```

</details>

## Expected behavior

- No error happens when we call `Close`.
- Pending telemetry is exported even during `Close`.


## Actual behavior

- If we intercept gRPC rpc calls, error shown as above `Code and Dependencies`.
- Some builtin metrics aren't exported to Cloud Monitoring.


## Additional context

- This issue is due to `go.opentelemetry.io/otel/sdk/metric.(*PeriodicReader)` side implementation.
  -  Cloud Monitoring has some quota like `Rate at which data can be written to a single time series`, `one point each 5 seconds`.
  - https://cloud.google.com/monitoring/quotas#custom_metrics_quotas
  - (But I'm not sure error message in `Execute result` is related to this quota)
  - `*PeriodicReader.Shutdown` and `*PeriodicReader.ForceFlush` called at [here](https://github.com/googleapis/google-cloud-go/blob/spanner/v1.78.0/spanner/metrics.go#L245-L246) forces export pending metrics, so it hits above quotas.
  - I created otel side issue to wait some seconds in `Shutdown`. https://github.com/open-telemetry/opentelemetry-go/issues/6677
    - But they said https://github.com/open-telemetry/opentelemetry-go/issues/6677#issuecomment-2817228311
- [The export on the Spanner side](https://github.com/googleapis/google-cloud-go/blob/spanner/v1.79.0/spanner/metrics.go#L245-L246) also needs to be modified.
  - It calls both `ForceFlush` and `Shutdown`, both methods force export pending metrics. [ref](https://github.com/open-telemetry/opentelemetry-go/blob/sdk/metric/v1.35.0/sdk/metric/periodic_reader.go#L275-L347)
  - After otel side issue is fixed, I think we only need to call `Shutdown` to shut down and export pending metrics.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

spanner: Close causes /google.monitoring.v3.MetricService/CreateServiceTimeSeries error internally #12017

Client

Environment

Code and Dependencies

Expected behavior

Actual behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

spanner: Close causes /google.monitoring.v3.MetricService/CreateServiceTimeSeries error internally #12017

Description

Client

Environment

Code and Dependencies

Expected behavior

Actual behavior

Additional context

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions