Skip to content

netbox: reduce spam in netbox logs#11727

Merged
sergepetrenko merged 1 commit intotarantool:masterfrom
mrForza:mrforza/gh-10506-spam-in-netbox-logs
Nov 20, 2025
Merged

netbox: reduce spam in netbox logs#11727
sergepetrenko merged 1 commit intotarantool:masterfrom
mrForza:mrforza/gh-10506-spam-in-netbox-logs

Conversation

@mrForza
Copy link
Contributor

@mrForza mrForza commented Jul 30, 2025

Before this patch a lot of "Connection refused" records were printed in
log files during netbox connection. This happened in two general
situations:

  1. Current instance tries to connect to non-existent instance.
  2. Current instance loses a succeeded connection to another instance.

The spam in logs was appearing because we didn't have any logic in
netbox_worker_f which would control periodic errors' logging during
netbox connection.

Now, we fix this issue by passing a whole error object into lua stack
instead of errmsg and changing logic of errors' displaying in new_sm
netbox lua function. We display error in "warning" mode only if

  1. This error is the first one in netbox session.
  2. Current and previous error has different types (codes).
  3. Current and previous error has different messages and different types
    of parent error.

Otherwise, current error will be repeated in "verbose" mode. Also we print
an information about how many seconds it will take to reconnect (after
Connection refused error).

Closes #10506

NO_DOC=bugfix

@mrForza mrForza requested a review from a team as a code owner July 30, 2025 15:29
@mrForza mrForza force-pushed the mrforza/gh-10506-spam-in-netbox-logs branch 2 times, most recently from 53b6054 to 1658801 Compare July 31, 2025 14:12
@coveralls
Copy link

coveralls commented Jul 31, 2025

Coverage Status

coverage: 87.643% (+0.003%) from 87.64%
when pulling 3bd36fa on mrForza:mrforza/gh-10506-spam-in-netbox-logs
into cc77a6e
on tarantool:master
.

Copy link
Contributor

@Serpentian Serpentian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the patch, finally vshard users will stop complaining about spam in logs!

Here we balance between code readability and functionality, and I'm on the side of the readability, since we can afford to loose a real error in logs for now, IMHO. But I'd rather create a ticket after that patch for introducing different error codes in the net.box, one error code for all errors is definitely not enough

@Serpentian Serpentian assigned mrForza and locker and unassigned Serpentian Aug 13, 2025
@locker locker assigned mrForza and unassigned locker and mrForza Aug 13, 2025
@locker locker removed their assignment Aug 15, 2025
@mrForza mrForza force-pushed the mrforza/gh-10506-spam-in-netbox-logs branch 3 times, most recently from c5cbb43 to 828d4cd Compare August 20, 2025 09:26
@mrForza mrForza assigned Serpentian and sergepetrenko and unassigned mrForza Aug 20, 2025
@mrForza mrForza force-pushed the mrforza/gh-10506-spam-in-netbox-logs branch from 828d4cd to de7e650 Compare August 25, 2025 07:44
Copy link
Contributor

@Serpentian Serpentian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the new logic of error handling, but we need to simplify the pushing of an error to Lua, since we already have all this code written and should not reimplement it)

@mrForza mrForza force-pushed the mrforza/gh-10506-spam-in-netbox-logs branch 2 times, most recently from adf63aa to f4e5435 Compare November 12, 2025 08:13
@mrForza mrForza requested a review from Serpentian November 12, 2025 09:38
@mrForza mrForza assigned Serpentian and unassigned mrForza Nov 12, 2025
@Serpentian Serpentian assigned mrForza and unassigned Serpentian Nov 12, 2025
Before this patch a lot of "Connection refused" records were printed in
log files during netbox connection. This happened in two general
situations:
1) Current instance tries to connect to non-existent instance.
2) Current instance loses a succeeded connection to another instance.

The spam in logs was appearing because we didn't have any logic in
`netbox_worker_f` which would control periodic errors' logging during
netbox connection.

Now, we fix this issue by passing a whole error object into lua stack
instead of errmsg and changing logic of errors' displaying in `new_sm`
netbox lua function. We display error in "warning" mode only if
1) This error is the first one in netbox session.
2) Current and previous error has different types (codes).
3) Current and previous error has different messages and different types
   of parent error.

Otherwise, current error will be repeated in "verbose" mode.

Closes tarantool#10506

NO_DOC=bugfix
@mrForza mrForza force-pushed the mrforza/gh-10506-spam-in-netbox-logs branch from f4e5435 to 3bd36fa Compare November 14, 2025 13:21
Copy link
Contributor

@Serpentian Serpentian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the patch! It'll finally stop clogging logs. Good work

@Serpentian Serpentian assigned sergepetrenko and unassigned mrForza Nov 14, 2025
@sergepetrenko sergepetrenko added backport/3.2 Automatically create a 3.2 backport PR backport/3.3 Automatically create a 3.3 backport PR backport/3.4 Automatically create a 3.4 backport PR backport/3.5 Automatically create a 3.5 backport PR labels Nov 19, 2025
Copy link
Collaborator

@sergepetrenko sergepetrenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

@sergepetrenko sergepetrenko added full-ci Enables all tests for a pull request and removed full-ci Enables all tests for a pull request labels Nov 19, 2025
@sergepetrenko sergepetrenko merged commit 2c27c9e into tarantool:master Nov 20, 2025
131 of 136 checks passed
@TarantoolBot
Copy link
Collaborator

Successfully created backport PR for release/3.2:

@TarantoolBot
Copy link
Collaborator

Successfully created backport PR for release/3.3:

@TarantoolBot
Copy link
Collaborator

Successfully created backport PR for release/3.4:

@TarantoolBot
Copy link
Collaborator

Successfully created backport PR for release/3.5:

@TarantoolBot
Copy link
Collaborator

Backport summary

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport/3.2 Automatically create a 3.2 backport PR backport/3.3 Automatically create a 3.3 backport PR backport/3.4 Automatically create a 3.4 backport PR backport/3.5 Automatically create a 3.5 backport PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"Connection refused" spam in logs, when storage is unreachable

7 participants