server: TLS handshake log spam from health checks #32102
Description
Summary
We are using k8s on AWS and experience the following errors despite cockroachdb working without issues.
2018-11-01 10:38:34 | ERROR | http: TLS handshake error from 10.0.9.140:36238: EOF | server.go:2921
-- | -- | -- | --
2018-11-01 10:38:36 | WARNING | grpc: Server.Serve failed to complete security handshake from "10.0.9.140:62179": EOF | vendor/google.golang.org/grpc/server.go:603
2018-11-01 10:38:37 | ERROR | http: TLS handshake error from 10.0.9.66:4587: EOF | server.go:2921
2018-11-01 10:38:38 | WARNING | grpc: Server.Serve failed to complete security handshake from "10.0.9.66:1101": EOF
k8s configs -> https://github.com/cockroachdb/cockroach/pull/27921/files
Load Balancer configs:
apiVersion: v1
kind: Service
metadata:
# This service is meant to be used by clients of the database. It exposes a ClusterIP that will
# automatically load balance connections to the different database pods.
name: cockroachdb-external
labels:
app: cockroachdb
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "4000"
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
type: LoadBalancer
ports:
# The main port, served by gRPC, serves Postgres-flavor SQL, internode
# traffic and the cli.
- port: 26257
targetPort: 26257
name: grpc
# The secondary port serves the UI as well as health and debug endpoints.
- port: 8080
targetPort: 8080
name: http
selector:
app: cockroachdb
Steps to reproduce
- Setup k8s with the above configurations
- View error logs
Expected Result
No error log entry
Actual Result
Error log entry
Log files/version
Node 1
2018-11-01 10:22:43 | INFO | [config] clusterID: 0e8897c8-ce0f-498b-95c1-e56892697e47 | util/log/clog.go:1067
-- | -- | -- | --
2018-11-01 10:22:43 | INFO | [config] arguments: [/cockroach/cockroach start --logtostderr --certs-dir /cockroach/cockroach-certs --advertise-host cockroachdb-0.cockroachdb.default.svc.cluster.local --http-host 0.0.0.0 --join cockroachdb-0.cockroachdb,cockroachdb-1.cockroachdb,cockroachdb-2.cockroachdb --cache 25% --max-sql-memory 25%] | util/log/clog.go:1067
2018-11-01 10:22:43 | INFO | [config] binary: CockroachDB CCL v2.1.0 (x86_64-unknown-linux-gnu, built 2018/10/30 12:32:34, go1.10.3) | util/log/clog.go:1067
2018-11-01 10:22:43 | INFO | [config] running on machine: cockroachdb-0 | util/log/clog.go:1067
2018-11-01 10:22:43 | INFO | [config] file created at: 2018/11/01 10:22:43
---
2018-11-01 10:38:34 | WARNING | grpc: Server.Serve failed to complete security handshake from "10.0.9.66:50273": EOF | vendor/google.golang.org/grpc/server.go:603
-- | -- | -- | --
2018-11-01 10:38:34 | WARNING | grpc: Server.Serve failed to complete security handshake from "10.0.9.66:56280": EOF | vendor/google.golang.org/grpc/server.go:603
2018-11-01 10:38:34 | ERROR | http: TLS handshake error from 10.0.9.140:36238: EOF | server.go:2921
2018-11-01 10:38:36 | WARNING | grpc: Server.Serve failed to complete security handshake from "10.0.9.140:62179": EOF | vendor/google.golang.org/grpc/server.go:603
2018-11-01 10:38:37 | ERROR | http: TLS handshake error from 10.0.9.66:4587: EOF | server.go:2921
2018-11-01 10:38:38 | WARNING | grpc: Server.Serve failed to complete security handshake from "10.0.9.66:1101": EOF
Node 2
2018-11-01 10:07:28 | INFO | [config] clusterID: 0e8897c8-ce0f-498b-95c1-e56892697e47 | util/log/clog.go:1067
-- | -- | -- | --
2018-11-01 10:07:28 | INFO | [config] arguments: [/cockroach/cockroach start --logtostderr --certs-dir /cockroach/cockroach-certs --advertise-host cockroachdb-2.cockroachdb.default.svc.cluster.local --http-host 0.0.0.0 --join cockroachdb-0.cockroachdb,cockroachdb-1.cockroachdb,cockroachdb-2.cockroachdb --cache 25% --max-sql-memory 25%] | util/log/clog.go:1067
2018-11-01 10:07:28 | INFO | [config] binary: CockroachDB CCL v2.1.0 (x86_64-unknown-linux-gnu, built 2018/10/30 12:32:34, go1.10.3) | util/log/clog.go:1067
2018-11-01 10:07:28 | INFO | [config] running on machine: cockroachdb-2 | util/log/clog.go:1067
2018-11-01 10:07:28 | INFO | [config] file created at: 2018/11/01 10:07:28
---
2018-11-01 10:22:37 | ERROR | http: TLS handshake error from 10.0.9.140:7586: EOF | server.go:2921
-- | -- | -- | --
2018-11-01 10:22:41 | WARNING | grpc: Server.Serve failed to complete security handshake from "10.0.9.140:18280": EOF | vendor/google.golang.org/grpc/server.go:603
2018-11-01 10:22:42 | WARNING | grpc: Server.Serve failed to complete security handshake from "10.0.9.66:59662": EOF | vendor/google.golang.org/grpc/server.go:603
2018-11-01 10:22:44 | WARNING | grpc: Server.Serve failed to complete security handshake from "10.0.9.140:10379": EOF
Epic: CRDB-549