Skip to content

colossus failing to sync files #4615

@yasiryagi

Description

@yasiryagi

Issues: Storage node is failing to replicate files to other servers.

Storage node running a docker.

joystream ~/joystream # docker images
REPOSITORY                        TAG              IMAGE ID       CREATED        SIZE
postgres                          12               10f11c61d788   6 weeks ago    373MB
prom/alertmanager                 latest           c8568f914cd2   6 weeks ago    65.1MB
node                              14               0f2c7f307e4d   7 weeks ago    916MB
prom/prometheus                   latest           932c2dbe7d3e   7 weeks ago    231MB
joystream/node                    latest           b595ebd99376   7 weeks ago    201MB
grafana/grafana                   latest           83f377cc32a0   8 weeks ago    317MB
redis                             6.0-alpine       f30aed8202b4   2 months ago   26.3MB
prom/node-exporter                latest           0da6a335fe13   2 months ago   22.5MB
caddy                             2.6              006d393a4e6a   3 months ago   46.8MB
joystream/hydra-indexer-gateway   4.0.0-alpha.7    da55b0b55e4c   3 months ago   1.28GB
joystream/hydra-indexer           v4.0.0-alpha.7   4c4fbd0bc505   3 months ago   960MB
gcr.io/cadvisor/cadvisor          latest           68c29634fe49   2 years ago    163MB

Error log

joystream ~/joystream # docker-compose logs -f --tail 10000 colossus-1
Attaching to colossus-1
colossus-1               | yarn run v1.22.19
colossus-1               | $ /joystream/node_modules/.bin/storage-node server --worker=0 --port=3333 --uploads=/data --sync --syncInterval=1 --queryNodeEndpoint=http://graphql-server:8081/graphql --apiUrl=ws://joystream-node:9944/ --keyFile=/keystore/storage-role-key.json --password=******* --elasticSearchEndpoint=https://elastic.joystreamstats.live/ --logFilePath=/logs
colossus-1               | 2023-02-08 22:41:29:4129 info: Initialized runtime connection: ws://joystream-node:9944/
colossus-1               | 2023-02-08 22:41:29:4129 info: Waiting for chain to be synced before proceeding.
colossus-1               | 2023-02-08 22:41:29:4129 info: Removing temp directory ...
colossus-1               | 2023-02-08 22:41:30:4130 info: Creating temp directory ...
colossus-1               | 2023-02-08 22:41:30:4130 debug: Local ID cache loaded.
colossus-1               | 2023-02-08 22:41:30:4130 info: Query node endpoint set: http://graphql-server:8081/graphql
colossus-1               | 2023-02-08 22:41:30:4130 info: Synchronization enabled.
colossus-1               | 2023-02-08 22:41:30:4130 info: Sync paused for 1 minute(s).
colossus-1               | 2023-02-08 22:41:30:4130 debug: Query - storageBucketsConnection
colossus-1               | 2023-02-08 22:41:30:4130 debug: Max file size runtime parameter: 64424509440
colossus-1               | 2023-02-08 22:41:30:4130 info: Listening on http://localhost:3333
colossus-1               | 2023-02-08 22:41:48:4148 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:41:49:4149 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:41:58:4158 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:42:00:420 http: HTTP GET /api/v1/state/data
colossus-1               | 2023-02-08 22:42:22:4222 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:42:23:4223 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:42:24:4224 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:42:30:4230 http: HTTP GET /api/v1/state/data
colossus-1               | 2023-02-08 22:42:30:4230 info: Resume syncing....
colossus-1               | 2023-02-08 22:42:30:4230 info: Started syncing...
colossus-1               | 2023-02-08 22:42:30:4230 debug: Query - storageBucketsConnection
colossus-1               | 2023-02-08 22:42:30:4230 debug: Sync - getting all storage buckets: offset = 0, limit = 1000
colossus-1               | 2023-02-08 22:42:30:4230 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:42:31:4231 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:42:32:4232 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:42:32:4232 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:42:33:4233 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:42:33:4233 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:42:34:4234 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:42:34:4234 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:42:35:4235 debug: Query - storageDataObjectsConnection
colossus-1               | 2023-02-08 22:42:35:4235 error: Critical sync error: Error: Unexpected token < in JSON at position 0
colossus-1               | 2023-02-08 22:42:35:4235 info: Sync paused for 1 minute(s).
colossus-1               | 2023-02-08 22:42:48:4248 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:42:49:4249 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:42:59:4259 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:43:00:430 http: HTTP GET /api/v1/state/data
colossus-1               | 2023-02-08 22:43:22:4322 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:43:23:4323 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:43:26:4326 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:43:30:4330 http: HTTP GET /api/v1/state/data
colossus-1               | 2023-02-08 22:43:35:4335 info: Resume syncing....
colossus-1               | 2023-02-08 22:43:35:4335 info: Started syncing...
colossus-1               | 2023-02-08 22:43:35:4335 debug: Query - storageBucketsConnection
colossus-1               | 2023-02-08 22:43:35:4335 debug: Sync - getting all storage buckets: offset = 0, limit = 1000
colossus-1               | 2023-02-08 22:43:35:4335 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:43:36:4336 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:43:36:4336 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:43:37:4337 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:43:37:4337 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:43:38:4338 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:43:39:4339 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:43:39:4339 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:43:40:4340 debug: Query - storageDataObjectsConnection
colossus-1               | 2023-02-08 22:43:40:4340 error: Critical sync error: Error: Unexpected token < in JSON at position 0
colossus-1               | 2023-02-08 22:43:40:4340 info: Sync paused for 1 minute(s).
colossus-1               | 2023-02-08 22:43:48:4348 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:43:59:4359 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:44:00:440 http: HTTP GET /api/v1/state/data
colossus-1               | 2023-02-08 22:44:08:448 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:44:22:4422 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:44:23:4423 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:44:27:4427 http: HTTP GET /api/v1/version
colossus-1               | 2023-02-08 22:44:30:4430 http: HTTP GET /api/v1/state/data
colossus-1               | 2023-02-08 22:44:40:4440 info: Resume syncing....
colossus-1               | 2023-02-08 22:44:40:4440 info: Started syncing...
colossus-1               | 2023-02-08 22:44:40:4440 debug: Query - storageBucketsConnection
colossus-1               | 2023-02-08 22:44:40:4440 debug: Sync - getting all storage buckets: offset = 0, limit = 1000
colossus-1               | 2023-02-08 22:44:40:4440 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:44:40:4440 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:44:41:4441 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:44:42:4442 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:44:42:4442 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:44:43:4443 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:44:43:4443 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:44:44:4444 debug: Query - storageBagsConnection
colossus-1               | 2023-02-08 22:44:44:4444 debug: Query - storageDataObjectsConnection
colossus-1               | 2023-02-08 22:44:44:4444 error: Critical sync error: Error: Unexpected token < in JSON at position 0
colossus-1               | 2023-02-08 22:44:44:4444 info: Sync paused for 1 minute(s).

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions