Skip to content

DP ES crash on debug log level #3877

@traumschule

Description

@traumschule

reporting response times to ES on log levels debug or silly values have wrong type, the request fails and DP crashes:
mapper [fields.responseTimes] cannot be changed from type [text] to [long]

Error should be handled and not crash the node.

STR

  1. set up DP provider
  2. enable elastic reporting with level debug
    (silly is another option)
  elastic:
    level: debug
    endpoint: https://elastic.joystreamstats.live/
  1. restart the node

Errors

Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: elasticsearch indexing error {
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: type: 'illegal_argument_exception',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: reason: 'mapper [fields.responseTimes] cannot be changed from type [text] to [long]'
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: } {
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: '@timestamp': '2022-06-07T23:13:01.580Z',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: message: 'Mean response times updated',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: severity: 'verbose',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: fields: {
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: label: 'NetworkingManager',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: responseTimes: [ [Array], [Array], [Array], [Array], [Array], [Array], [Array], [Array] ],
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: [Symbol(level)]: 'verbose',
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: [Symbol(splat)]: [ [Object] ],
Jun 07 23:13:08 us.joystreamstats.live yarn[979590]: [Symbol(message)]: '{"@timestamp":"2022-06-07T23:13:01.580Z","log.level":"verbose","message":"Mean response times updated","ecs":{"version":"1.6.0"},"label":"NetworkingManager","responseTimes":[["https://new.xjames.xyz/storage/api/v1%22,371],[%22https://jssp.cryptostarter.info/storage/api/v1%22,361],[%22https://sp1.nodex.club/storage/api/v1%22,360],[%22https://joystream2.yyagi.cloud/storage/api/v1%22,367],[%22https://joystream.godshunter.su/storage/api/v1",368],["https://ipfs.joystreamstats.live/storage/api/v1%22,412],[%22https://whitesocks.ru/storage/api/v1%22,415],[%22https://ru.joystreamstats.live/storage/api/v1%22,491]]%7D'

On another occasion error wasn't handled either:

Jun 07 14:34:56 ipfs.joystreamstats.live yarn[316636]: (node:316636) UnhandledPromiseRejectionWarning: ResponseError: security_exception: [security_exception] Reason: missing authentication credentials for REST request [/_bulk?wait_for_active_shards=1&timeout=5000ms]
Jun 07 14:34:56 ipfs.joystreamstats.live yarn[316636]: at onBody (/home/joystream-storage/joystream/node_modules/@elastic/elasticsearch/lib/Transport.js:355:23)
Jun 07 14:34:56 ipfs.joystreamstats.live yarn[316636]: at IncomingMessage.onEnd (/home/joystream-storage/joystream/node_modules/@elastic/elasticsearch/lib/Transport.js:279:11)
Jun 07 14:34:56 ipfs.joystreamstats.live yarn[316636]: (node:316636) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch().

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions