Skip to content

Envoy proxy stuck in PRE_INITIALIZING state for upstream Encrypted Redis Cluster (Elasticache) #8223

@sabiurr

Description

@sabiurr

Title: Envoy proxy stuck in PRE_INITIALIZING state for upstream Encrypted Redis Cluster (Elasticache)

Description:

I have enabled encryption (Encryption in-transit (TLS)) on my elasticache cluster and now it seems that envoy is stuck in PRE_INITIALIZING state. I believe this is during cluster discovery/initialization. I believe I have to enable using SSL with the redis protocol (rediss). Could someone help my with this configuration?

I am using envoy (v1.11.1) as a redis cluster proxy and running as a sidecar on the client host

Config:

  listeners:
  - name: redis_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 1999
    filter_chains:
    - filters:
      - name: envoy.redis_proxy
        config:
          stat_prefix: egress_redis
          prefix_routes:
            catch_all_route:
               cluster: redis_cluster
          settings:
            op_timeout: 5s
  clusters:
  - name: redis_cluster
    connect_timeout: 0.25s
    dns_lookup_family: V4_ONLY
    lb_policy: CLUSTER_PROVIDED
    upstream_connection_options:
      tcp_keepalive:
        keepalive_time: 60
        keepalive_probes: 1
        keepalive_interval: 5
    hosts:
      - socket_address:
          address: clustercfg.test-encrypted.use1.cache.amazonaws.com
          port_value: 6379
    cluster_type:
      name: envoy.clusters.redis
      typed_config:
        "@type": type.googleapis.com/google.protobuf.Struct
        value:
          cluster_refresh_rate: 360s
          cluster_refresh_timeout: 4s
admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

Logs:

[2019-09-13 17:27:45.871][6][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:839] adding TLS initial cluster redis_cluster
[2019-09-13 17:27:45.871][6][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:75] cm init: adding: cluster=redis_cluster primary=1 secondary=0
[2019-09-13 17:27:45.871][6][info][config] [source/server/configuration_impl.cc:71] loading 1 listener(s)
[2019-09-13 17:27:45.871][6][debug][config] [source/server/configuration_impl.cc:73] listener #0:
[2019-09-13 17:27:45.871][6][debug][config] [source/server/listener_manager_impl.cc:485] begin add/update listener: name=redis_listener hash=8613620242285538431
[2019-09-13 17:27:45.871][6][debug][config] [source/server/listener_manager_impl.cc:57]   filter #0:
[2019-09-13 17:27:45.871][6][debug][config] [source/server/listener_manager_impl.cc:58]     name: envoy.redis_proxy
[2019-09-13 17:27:45.871][6][debug][config] [source/server/listener_manager_impl.cc:61]   config: {"prefix_routes":{"catch_all_route":{"cluster":"redis_cluster"}},"stat_prefix":"egress_redis","settings":{"op_timeout":"5s"}}
[2019-09-13 17:27:45.873][6][debug][config] [source/server/listener_manager_impl.cc:376] add active listener: name=redis_listener, hash=8613620242285538431, address=0.0.0.0:1999
[2019-09-13 17:27:45.873][6][info][config] [source/server/configuration_impl.cc:96] loading tracing configuration
[2019-09-13 17:27:45.873][6][info][config] [source/server/configuration_impl.cc:116] loading stats sink configuration
[2019-09-13 17:27:45.873][6][info][main] [source/server/server.cc:516] starting main dispatch loop
[2019-09-13 17:27:45.887][6][debug][connection] [source/common/network/connection_impl.cc:704] [C0] connecting to <IP>:6379
[2019-09-13 17:27:45.887][6][debug][connection] [source/common/network/connection_impl.cc:713] [C0] connection in progress
[2019-09-13 17:27:45.888][6][debug][connection] [source/common/network/connection_impl.cc:552] [C0] connected
[2019-09-13 17:27:49.891][6][debug][connection] [source/common/network/connection_impl.cc:101] [C0] closing data_to_write=0 type=1
[2019-09-13 17:27:49.891][6][debug][connection] [source/common/network/connection_impl.cc:190] [C0] closing socket: 1
{
 "version": "e349fb6139e4b7a59a9a359be0ea45dd61e589c5/1.11.1/Clean/RELEASE/BoringSSL",
 "state": "PRE_INITIALIZING",
 "command_line_options": {
  "base_id": "0",
  "concurrency": 4,
  "config_path": "/etc/envoy.yaml",
  "config_yaml": "",
  "allow_unknown_fields": false,
  "admin_address_path": "",
  "local_address_ip_version": "v4",
  "log_level": "debug",
  "component_log_level": "",
  "log_format": "[%Y-%m-%d %T.%e][%t][%l][%n] %v",
  "log_path": "/tmp/envoy.log",
  "hot_restart_version": false,
  "service_cluster": "",
  "service_node": "",
  "service_zone": "",
  "mode": "Serve",
  "max_stats": "0",
  "max_obj_name_len": "0",
  "disable_hot_restart": false,
  "enable_mutex_tracing": false,
  "restart_epoch": 0,
  "cpuset_threads": false,
  "file_flush_interval": "10s",
  "drain_time": "600s",
  "parent_shutdown_time": "900s"
 },
 "uptime_current_epoch": "11s",
 "uptime_all_epochs": "11s"
}```

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionQuestions that are neither investigations, bugs, nor enhancements

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions