This repository was archived by the owner on Jul 19, 2025. It is now read-only.
This repository was archived by the owner on Jul 19, 2025. It is now read-only.
ksync's syncthing container crashloop due to failure in listening to service at :22000 #411
Description
Version:
$ ksync version
ksync:
Version: Release
Go Version: go1.13.6
Git Commit: b2ca492
Git Tag: 0.4.4
Built: Thu Feb 20 23:22:28 +0000 2020
OS/Arch: darwin/amd64
service:
Version: corrupted-version
Go Version: go1.13.6
Git Commit: fbee236
Git Tag: 0.4.3
Built: Wed Feb 19 01:35:35 +0000 2020
Issue
Some of the ksync pods get stuck in Crashloop everyday. This happens because the synching container process is not able to start and is stuck in restart loop due to the following error.
kubectl get pods -n kube-system | grep ksync | grep Cras
ksync-p7826 1/2 CrashLoopBackOff 392 45h
ksync-xwbcc 1/2 CrashLoopBackOff 345 47h
Logs of the failing synching container
k logs ksync-xwbcc -n kube-system -c syncthing
[monitor] 03:39:53 INFO: Starting syncthing
[start] 03:39:53 INFO: syncthing v1.3.4 "Fermium Flea" (go1.13.7 linux-amd64) teamcity@build.syncthing.net 2020-01-14 07:01:03 UTC
[start] 03:39:53 INFO: Generating ECDSA key and certificate for syncthing...
[start] 03:39:53 INFO: Archiving a copy of old config file format at: /var/syncthing/config/config.xml.v26
[TK3XU] 03:39:53 INFO: My ID: TK3XUXT-X6EVETU-LIB3GSR-CVTDG4G-HDEPGBK-R3GGMXA-BFFNSBO-X7XRNQ4
[TK3XU] 03:39:54 INFO: Single thread SHA256 performance is 282 MB/s using crypto/sha256 (280 MB/s using minio/sha256-simd).
[TK3XU] 03:39:54 VERBOSE: Starting up (/var/syncthing/config)
[TK3XU] 03:39:54 INFO: Hashing performance is 235.72 MB/s
[TK3XU] 03:39:54 INFO: Overall send rate is unlimited, receive rate is unlimited
[TK3XU] 03:39:54 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:54 INFO: Listen (BEP/tcp): listen tcp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:54 INFO: c.S.listenerSupervisor: Failed service 'quic://0.0.0.0:22000' (1.000000 failures of 2.000000), restarting: true, error: "{quic://0.0.0.0:22000 quic://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:54 INFO: c.S.listenerSupervisor: Failed service 'tcp://0.0.0.0:22000' (1.999998 failures of 2.000000), restarting: true, error: "{tcp://0.0.0.0:22000 tcp://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:54 INFO: Listen (BEP/tcp): listen tcp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:54 INFO: Entering the backoff state.
[TK3XU] 03:39:54 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:54 INFO: c.S.listenerSupervisor: Failed service 'tcp://0.0.0.0:22000' (2.999992 failures of 2.000000), restarting: false, error: "{tcp://0.0.0.0:22000 tcp://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:54 INFO: Entering the backoff state.
[TK3XU] 03:39:54 INFO: c.S.listenerSupervisor: Failed service 'quic://0.0.0.0:22000' (3.999987 failures of 2.000000), restarting: false, error: "{quic://0.0.0.0:22000 quic://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:54 INFO: Loading HTTPS certificate: open /var/syncthing/config/https-cert.pem: no such file or directory
[TK3XU] 03:39:54 INFO: Creating new HTTPS certificate
[TK3XU] 03:39:54 WARNING: Failed starting API: listen tcp 0.0.0.0:8384: bind: address already in use
[monitor] 03:39:54 INFO: Syncthing exited: exit status 1
[monitor] 03:39:55 INFO: Starting syncthing
[start] 03:39:55 INFO: syncthing v1.3.4 "Fermium Flea" (go1.13.7 linux-amd64) teamcity@build.syncthing.net 2020-01-14 07:01:03 UTC
[TK3XU] 03:39:55 INFO: My ID: TK3XUXT-X6EVETU-LIB3GSR-CVTDG4G-HDEPGBK-R3GGMXA-BFFNSBO-X7XRNQ4
[TK3XU] 03:39:56 INFO: Single thread SHA256 performance is 287 MB/s using crypto/sha256 (280 MB/s using minio/sha256-simd).
[TK3XU] 03:39:56 VERBOSE: Starting up (/var/syncthing/config)
[TK3XU] 03:39:57 INFO: Hashing performance is 236.08 MB/s
[TK3XU] 03:39:57 INFO: Overall send rate is unlimited, receive rate is unlimited
[TK3XU] 03:39:57 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:57 INFO: c.S.listenerSupervisor: Failed service 'quic://0.0.0.0:22000' (1.000000 failures of 2.000000), restarting: true, error: "{quic://0.0.0.0:22000 quic://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:57 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:57 INFO: c.S.listenerSupervisor: Failed service 'quic://0.0.0.0:22000' (1.999996 failures of 2.000000), restarting: true, error: "{quic://0.0.0.0:22000 quic://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:57 INFO: Listen (BEP/tcp): listen tcp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:57 INFO: Entering the backoff state.
[TK3XU] 03:39:57 INFO: c.S.listenerSupervisor: Failed service 'tcp://0.0.0.0:22000' (2.999993 failures of 2.000000), restarting: false, error: "{tcp://0.0.0.0:22000 tcp://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:57 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:57 INFO: Entering the backoff state.
[TK3XU] 03:39:57 INFO: c.S.listenerSupervisor: Failed service 'quic://0.0.0.0:22000' (3.999986 failures of 2.000000), restarting: false, error: "{quic://0.0.0.0:22000 quic://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:57 WARNING: Failed starting API: listen tcp 0.0.0.0:8384: bind: address already in use
[monitor] 03:39:57 INFO: Syncthing exited: exit status 1
[monitor] 03:39:58 INFO: Starting syncthing
[start] 03:39:58 INFO: syncthing v1.3.4 "Fermium Flea" (go1.13.7 linux-amd64) teamcity@build.syncthing.net 2020-01-14 07:01:03 UTC
[TK3XU] 03:39:58 INFO: My ID: TK3XUXT-X6EVETU-LIB3GSR-CVTDG4G-HDEPGBK-R3GGMXA-BFFNSBO-X7XRNQ4
[TK3XU] 03:39:59 INFO: Single thread SHA256 performance is 285 MB/s using crypto/sha256 (282 MB/s using minio/sha256-simd).
[TK3XU] 03:39:59 VERBOSE: Starting up (/var/syncthing/config)
[TK3XU] 03:39:59 INFO: Hashing performance is 240.57 MB/s
[TK3XU] 03:39:59 INFO: Overall send rate is unlimited, receive rate is unlimited
[TK3XU] 03:39:59 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:59 INFO: c.S.listenerSupervisor: Failed service 'quic://0.0.0.0:22000' (1.000000 failures of 2.000000), restarting: true, error: "{quic://0.0.0.0:22000 quic://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:59 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:59 INFO: c.S.listenerSupervisor: Failed service 'quic://0.0.0.0:22000' (1.999997 failures of 2.000000), restarting: true, error: "{quic://0.0.0.0:22000 quic://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:59 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:59 INFO: Listen (BEP/tcp): listen tcp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:39:59 INFO: Entering the backoff state.
[TK3XU] 03:39:59 INFO: c.S.listenerSupervisor: Failed service 'quic://0.0.0.0:22000' (2.999992 failures of 2.000000), restarting: false, error: "{quic://0.0.0.0:22000 quic://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:59 INFO: Entering the backoff state.
[TK3XU] 03:39:59 INFO: c.S.listenerSupervisor: Failed service 'tcp://0.0.0.0:22000' (3.999979 failures of 2.000000), restarting: false, error: "{tcp://0.0.0.0:22000 tcp://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:39:59 WARNING: Failed starting API: listen tcp 0.0.0.0:8384: bind: address already in use
[monitor] 03:39:59 INFO: Syncthing exited: exit status 1
[monitor] 03:40:00 INFO: Starting syncthing
[start] 03:40:00 INFO: syncthing v1.3.4 "Fermium Flea" (go1.13.7 linux-amd64) teamcity@build.syncthing.net 2020-01-14 07:01:03 UTC
[TK3XU] 03:40:00 INFO: My ID: TK3XUXT-X6EVETU-LIB3GSR-CVTDG4G-HDEPGBK-R3GGMXA-BFFNSBO-X7XRNQ4
[TK3XU] 03:40:01 INFO: Single thread SHA256 performance is 286 MB/s using crypto/sha256 (279 MB/s using minio/sha256-simd).
[TK3XU] 03:40:01 VERBOSE: Starting up (/var/syncthing/config)
[TK3XU] 03:40:02 INFO: Hashing performance is 237.58 MB/s
[TK3XU] 03:40:02 INFO: Overall send rate is unlimited, receive rate is unlimited
[TK3XU] 03:40:02 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:40:02 INFO: c.S.listenerSupervisor: Failed service 'quic://0.0.0.0:22000' (1.000000 failures of 2.000000), restarting: true, error: "{quic://0.0.0.0:22000 quic://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:40:02 INFO: Listen (BEP/tcp): listen tcp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:40:02 INFO: Listen (BEP/quic): listen udp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:40:02 INFO: c.S.listenerSupervisor: Failed service 'tcp://0.0.0.0:22000' (1.999997 failures of 2.000000), restarting: true, error: "{tcp://0.0.0.0:22000 tcp://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:40:02 INFO: Entering the backoff state.
[TK3XU] 03:40:02 INFO: c.S.listenerSupervisor: Failed service 'quic://0.0.0.0:22000' (2.999994 failures of 2.000000), restarting: false, error: "{quic://0.0.0.0:22000 quic://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:40:02 INFO: Listen (BEP/tcp): listen tcp 0.0.0.0:22000: bind: address already in use
[TK3XU] 03:40:02 INFO: Entering the backoff state.
[TK3XU] 03:40:02 INFO: c.S.listenerSupervisor: Failed service 'tcp://0.0.0.0:22000' (3.999984 failures of 2.000000), restarting: false, error: "{tcp://0.0.0.0:22000 tcp://0.0.0.0:22000} returned unexpectedly", stacktrace: [unknown stack trace]
[TK3XU] 03:40:02 WARNING: Failed starting API: listen tcp 0.0.0.0:8384: bind: address already in use
[monitor] 03:40:02 INFO: Syncthing exited: exit status 1
[monitor] 03:40:03 WARNING: 4 restarts in 9.736738931s; not retrying further
Hack Resolution
Deleting the ksync pod fixes it, if it recovered on restart that would be much better than this.