Environment details

• Hardware: on-prem 8-core compute node
• OS: RHEL 6.9
• Node.js version: 8.11.3
• npm version: 6.4.0
• @google-cloud/pubsub 0.23.0-0.24.1:

Steps to reproduce

1. Publish a moderate number of messages to a topic (~4k in our case)
2. Use a single subscriber that processes each message within 1-2 min, e.g. with the following code:

const subscription = pubSub.subscription(SUBSCRIPTION_NAME, {
  flowControl: {
    maxMessages: 2,
    allowExcessMessages: false,
  },
});
subscription.on("message", ({ ack }) => {
  setTimeout(ack, 60000 * 2 * Math.random());
});
subscription.on("error", console.error);

3. After ~30 min of churning through messages, Stackdriver shows that:
   - num_undelivered_messages stops going down
   - pull_ack_message_operation_count marks every message ack as expired
   
   - streaming_pull_message_operation_count shows that 10-min pull interval stops.
   

As a net result, we can't seem to ack any more messages and have to restart the subscriber.

We varied settings like flowControl.maxMessages (1-4) and ackDeadline (10, 300, 7200, 720000) and that didn't seem to have an effect.

Downgrading to 0.22.2 seems to resolve this issue. We still get occasional expired acks, but not in 100% of cases.

Thanks!