Skip to content

Problems with reliability of Pub/Sub subscriptions in different Redis clients #7855

@adrianpasternak

Description

@adrianpasternak

Describe the bug

I'm not sure if this is right place to report this issue, because it seems like a problem with Redis clients. But the same issue is present in all clients that I've checked (Lettuce, Redisson, Jedis, go-redis).

In a case of a sudden connection loss Redis clients are not able detect network problems, and will be listening for Pub/Sub messages on a broken TCP connection for hours, making Pub/Sub unusable.

To reproduce

  1. Start a Redis on Host A
  2. Connect to a Pub/Sub using one of the Redis clients from Host B
  3. Block all traffic on Host A to a Redis server using iptables or other tool
  4. Redis client will not discover that the connection is lost.
  5. Now restart Redis on Host A, and restore network traffic.
  6. Redis client will be listening on connection that no longer exist on the server-side.

I've managed to reproduce this behavior using three different Java clients, and go-redis. Ticket for Lettuce with more details: redis/lettuce#1428

Expected behavior

Redis clients subscribed to a Pub/Sub should be able to detect a broken network connection, and reconnect when necessary.

Additional information

The undocumented workaround for this issue is to tweak OS parameters on a client's host: SO_KEEPALIVE, TCP_KEEPIDLE, TCP_KEEPINTVL and TCP_KEEPCNT.
It's similar to what redis-cli client is doing in application layer:

anetKeepAlive(NULL, context->fd, REDIS_CLI_KEEPALIVE_INTERVAL);

int anetKeepAlive(char *err, int fd, int interval)

Is there is any other way of making reliable Pub/Sub subscriptions without changing OS parameters?
Shouldn't all Redis clients change socket parameters in application layer like redis-cli?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions