Issues: Storage node is failing to replicate files to other servers.
Storage node running a docker.
joystream ~/joystream # docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
postgres 12 10f11c61d788 6 weeks ago 373MB
prom/alertmanager latest c8568f914cd2 6 weeks ago 65.1MB
node 14 0f2c7f307e4d 7 weeks ago 916MB
prom/prometheus latest 932c2dbe7d3e 7 weeks ago 231MB
joystream/node latest b595ebd99376 7 weeks ago 201MB
grafana/grafana latest 83f377cc32a0 8 weeks ago 317MB
redis 6.0-alpine f30aed8202b4 2 months ago 26.3MB
prom/node-exporter latest 0da6a335fe13 2 months ago 22.5MB
caddy 2.6 006d393a4e6a 3 months ago 46.8MB
joystream/hydra-indexer-gateway 4.0.0-alpha.7 da55b0b55e4c 3 months ago 1.28GB
joystream/hydra-indexer v4.0.0-alpha.7 4c4fbd0bc505 3 months ago 960MB
gcr.io/cadvisor/cadvisor latest 68c29634fe49 2 years ago 163MB
Error log
joystream ~/joystream # docker-compose logs -f --tail 10000 colossus-1
Attaching to colossus-1
colossus-1 | yarn run v1.22.19
colossus-1 | $ /joystream/node_modules/.bin/storage-node server --worker=0 --port=3333 --uploads=/data --sync --syncInterval=1 --queryNodeEndpoint=http://graphql-server:8081/graphql --apiUrl=ws://joystream-node:9944/ --keyFile=/keystore/storage-role-key.json --password=******* --elasticSearchEndpoint=https://elastic.joystreamstats.live/ --logFilePath=/logs
colossus-1 | 2023-02-08 22:41:29:4129 info: Initialized runtime connection: ws://joystream-node:9944/
colossus-1 | 2023-02-08 22:41:29:4129 info: Waiting for chain to be synced before proceeding.
colossus-1 | 2023-02-08 22:41:29:4129 info: Removing temp directory ...
colossus-1 | 2023-02-08 22:41:30:4130 info: Creating temp directory ...
colossus-1 | 2023-02-08 22:41:30:4130 debug: Local ID cache loaded.
colossus-1 | 2023-02-08 22:41:30:4130 info: Query node endpoint set: http://graphql-server:8081/graphql
colossus-1 | 2023-02-08 22:41:30:4130 info: Synchronization enabled.
colossus-1 | 2023-02-08 22:41:30:4130 info: Sync paused for 1 minute(s).
colossus-1 | 2023-02-08 22:41:30:4130 debug: Query - storageBucketsConnection
colossus-1 | 2023-02-08 22:41:30:4130 debug: Max file size runtime parameter: 64424509440
colossus-1 | 2023-02-08 22:41:30:4130 info: Listening on http://localhost:3333
colossus-1 | 2023-02-08 22:41:48:4148 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:41:49:4149 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:41:58:4158 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:42:00:420 http: HTTP GET /api/v1/state/data
colossus-1 | 2023-02-08 22:42:22:4222 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:42:23:4223 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:42:24:4224 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:42:30:4230 http: HTTP GET /api/v1/state/data
colossus-1 | 2023-02-08 22:42:30:4230 info: Resume syncing....
colossus-1 | 2023-02-08 22:42:30:4230 info: Started syncing...
colossus-1 | 2023-02-08 22:42:30:4230 debug: Query - storageBucketsConnection
colossus-1 | 2023-02-08 22:42:30:4230 debug: Sync - getting all storage buckets: offset = 0, limit = 1000
colossus-1 | 2023-02-08 22:42:30:4230 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:42:31:4231 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:42:32:4232 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:42:32:4232 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:42:33:4233 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:42:33:4233 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:42:34:4234 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:42:34:4234 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:42:35:4235 debug: Query - storageDataObjectsConnection
colossus-1 | 2023-02-08 22:42:35:4235 error: Critical sync error: Error: Unexpected token < in JSON at position 0
colossus-1 | 2023-02-08 22:42:35:4235 info: Sync paused for 1 minute(s).
colossus-1 | 2023-02-08 22:42:48:4248 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:42:49:4249 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:42:59:4259 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:43:00:430 http: HTTP GET /api/v1/state/data
colossus-1 | 2023-02-08 22:43:22:4322 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:43:23:4323 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:43:26:4326 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:43:30:4330 http: HTTP GET /api/v1/state/data
colossus-1 | 2023-02-08 22:43:35:4335 info: Resume syncing....
colossus-1 | 2023-02-08 22:43:35:4335 info: Started syncing...
colossus-1 | 2023-02-08 22:43:35:4335 debug: Query - storageBucketsConnection
colossus-1 | 2023-02-08 22:43:35:4335 debug: Sync - getting all storage buckets: offset = 0, limit = 1000
colossus-1 | 2023-02-08 22:43:35:4335 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:43:36:4336 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:43:36:4336 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:43:37:4337 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:43:37:4337 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:43:38:4338 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:43:39:4339 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:43:39:4339 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:43:40:4340 debug: Query - storageDataObjectsConnection
colossus-1 | 2023-02-08 22:43:40:4340 error: Critical sync error: Error: Unexpected token < in JSON at position 0
colossus-1 | 2023-02-08 22:43:40:4340 info: Sync paused for 1 minute(s).
colossus-1 | 2023-02-08 22:43:48:4348 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:43:59:4359 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:44:00:440 http: HTTP GET /api/v1/state/data
colossus-1 | 2023-02-08 22:44:08:448 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:44:22:4422 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:44:23:4423 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:44:27:4427 http: HTTP GET /api/v1/version
colossus-1 | 2023-02-08 22:44:30:4430 http: HTTP GET /api/v1/state/data
colossus-1 | 2023-02-08 22:44:40:4440 info: Resume syncing....
colossus-1 | 2023-02-08 22:44:40:4440 info: Started syncing...
colossus-1 | 2023-02-08 22:44:40:4440 debug: Query - storageBucketsConnection
colossus-1 | 2023-02-08 22:44:40:4440 debug: Sync - getting all storage buckets: offset = 0, limit = 1000
colossus-1 | 2023-02-08 22:44:40:4440 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:44:40:4440 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:44:41:4441 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:44:42:4442 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:44:42:4442 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:44:43:4443 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:44:43:4443 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:44:44:4444 debug: Query - storageBagsConnection
colossus-1 | 2023-02-08 22:44:44:4444 debug: Query - storageDataObjectsConnection
colossus-1 | 2023-02-08 22:44:44:4444 error: Critical sync error: Error: Unexpected token < in JSON at position 0
colossus-1 | 2023-02-08 22:44:44:4444 info: Sync paused for 1 minute(s).
Issues: Storage node is failing to replicate files to other servers.
Storage node running a docker.
Error log