Skip to content

Subscriber maxes out CPU after exactly 60 minutes on GKE #890

@ctavan

Description

@ctavan

Environment details

  • OS: GKE
  • Node.js version: 12.14.1
  • npm version: –
  • @google-cloud/pubsub version: pubsub@1.5.0 / grpc-js@0.6.16

Steps to reproduce

  1. Start a pod that subscribes to an idle subscription
  2. Wait 60 minutes
  3. Observe node process eat up all available CPU

Screenshot 2020-02-19 12 54 29

I have a pod that subscribes to a subscription which is 100% idle (i.e. no messages published on that topic) during the observation period.

Exactly 60 minutes after the pod started it eats up all available CPU until the pod is deleted manually to force a restart.

This behavior does not appear when using the C++ gGRPC bindings as suggested in #850 (comment)

I have also enabled debug output through

GRPC_TRACE=all
GRPC_VERBOSITY=DEBUG

I've put the full debug output (as fetched from Google Cloud Logging) of the entire lifetime of the pod in this gist.

There is no further output after the last line even though the pod kept running for a while as can be seen form the graph above: the bump in CPU usage is the time where the pod eats up all available CPU and doesn't seem to log anything. (Times in the logs correspond to the times in the graph above with 1h timezone difference, so 10:51 in the logs = 11:51 in the graph).

I can reproduce this problem with any of my subscriber pods.

I believe it is very likely that this problem also falls into the category of #868.

Metadata

Metadata

Assignees

Labels

api: pubsubIssues related to the googleapis/nodejs-pubsub API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions