[LogAnalyzer]: Accumulate syslog files rather than force logrotate by sihuihan88 · Pull Request #590 · sonic-net/sonic-mgmt

sihuihan88 · 2018-04-27T01:46:05Z

Signed-off-by: Sihui Han sihan@microsoft.com

Description of PR

Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Approach

How did you do it?
Problem: In current design, before loganalyzer analyzes syslog file, it will force logrotate. Since we have default logrotate cron job every 1min, it can result in following two issues:

When default logrotate and force logrotate happen at the same time, due to the conflict, it will return a middle-stage result i.e. creating syslog.1.gz (or swss.1.gz/sairedis.1.gz), without actually rotating it. Afterwards, all logrotate attempt will fail.
Loganalyzer place start_marker and end_marker in the syslog file so that it knows which part should be analyzed. Currently it only analyze the /var/log/syslog file and that is the reason why it forces logrotate before anything. However, when the default logrotate happens within this period, it could rotate start_marker or end_marker to other files, so that loganalyzer will fail as it could not find the markers.

Solution:
Instead of force logrotate and only analyze the latest one, we change the logic to following:

Place start_marker before taking any action.
Extract all syslogs after latest start_marker and store them to /tmp/syslog for future analyze.
Analyze /tmp/syslog so that there is no need to force logrotate, and there will be no conflict with default logrotate cron job.

How did you verify/test it?
Tested on DUT.
Any platform specific information?
Supported testbed topology if it's a new test case?

Documentation

stcheng · 2018-04-27T03:54:50Z

ansible/roles/test/tasks/pfc_wd.yml

any references to this 60 seconds number?

60 seconds is current logrotate cron job configuration

lguohan · 2018-04-27T07:53:38Z

ansible/roles/test/tasks/pfc_wd.yml

this should not be just for pfcwd, it should generic.

lguohan · 2018-04-30T18:04:36Z

also there is a catch here, since we disable log rotate, it might happen that we will get disk full duration the test.

stcheng · 2018-04-30T18:11:19Z

ansible/roles/test/files/tools/loganalyzer/loganalyzer_init.yml

put into /tmp?

sihuihan88 · 2018-04-30T18:26:14Z

That should not be an issue, as we only disable cron job when we want to force logrotate, and enable default cron job right after that.

jleveque · 2018-04-30T18:32:05Z

ansible/roles/test/files/tools/loganalyzer/loganalyzer_init.yml

I feel like we should also check to make sure no logrotate processes are running before proceeding. Even though you disable the cron job, there's a possibility the previous cron job is still running.

Good point. I updated the logic to wait 1min before proceed as cron checks configuration update every 1min. https://www.freebsd.org/cgi/man.cgi?cron

Cron will definitely pick up the config change before the next run, as I have scheduled logrotate to run every minute. Therefore, the next time cron wakes up, it will reload its config and will not kick off logrotate.

My concern is not when cron will pick up the configuration change, but rather that we need to wait for any already-running instances of logrotate from previous cron jobs to exit before starting another process.

Signed-off-by: Sihui Han <sihan@microsoft.com>

…590)" This reverts commit 0454ca1.

…590)" (#641) This reverts commit 0454ca1.

…onic-net#590)" This reverts commit 0454ca1.

…t#590) Signed-off-by: Sihui Han <sihan@microsoft.com>

…rotate (sonic-net#590)" (sonic-net#641)" This reverts commit c225eb5.

* Revert "Revert "[logrotate]: Accumulate syslogs rather than force logrotate (#590)" (#641)" This reverts commit c225eb5. * [loganalyze] Make sure logrotate will not run during logs accumulation Signed-off-by: stepanb <stepanb@mellanox.com> * Improve "extract_log" ansible module performance In most scenarios of using this module it is reasonable to search 'start_string' in the most recent log file instead of searching all appearence of 'start_string' in all logs in /var/log/syslog*. This improves performance of tests which use loganalyze frequently (e.g crm, pfc wd) Signed-off-by: stepanb <stepanb@mellanox.com>

…rotate (sonic-net#590)" (sonic-net#641)" This reverts commit c225eb5. Signed-off-by: stepanb <stepanb@mellanox.com> [logrotate] Make sure logrotate will not interfere with extract logs task for loganalyzer Signed-off-by: stepanb <stepanb@mellanox.com>

…er (sonic-net#590)  ### Description of PR  Summary: Fixes # (issue) On t2 topo we observed the following failure: `Failed: Not all routes flushed from nexthop 10.0.0.25 on asic 0 on cmp210-4` in tests: ``` pc/test_po_update.py::test_po_update::test_po_update_io_no_loss pc/test_po_voq.py::test_po_voq::test_voq_po_member_update ``` Increasing the timeout of the check_no_routes_from_nexthop call, resolves these issues. ### Type of change  - [x] Bug fix - [ ] Testbed and Framework(new/improvement) - [ ] New Test case - [ ] Skipped for non-supported platforms - [ ] Test case improvement ### Back port request - [ ] 202012 - [ ] 202205 - [ ] 202305 - [ ] 202311 - [x] 202405 - [ ] 202411 ### Approach #### What is the motivation for this PR? #### How did you do it? #### How did you verify/test it? #### Any platform specific information? #### Supported testbed topology if it's a new test case? ### Documentation

stcheng reviewed Apr 27, 2018

View reviewed changes

lguohan reviewed Apr 27, 2018

View reviewed changes

ansible/roles/test/tasks/pfc_wd.yml Outdated

Copy link
Copy Markdown

Contributor

lguohan Apr 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should not be just for pfcwd, it should generic.

sihuihan88 force-pushed the dev/sihan/logrotate branch from 736c429 to e2371cd Compare April 27, 2018 21:06

sihuihan88 changed the title ~~[pfcwd]: disable logrotate during the test~~ [LogAnalyzer]: disable logrotate cron job during loganalyzer Apr 27, 2018

sihuihan88 changed the title ~~[LogAnalyzer]: disable logrotate cron job during loganalyzer~~ [LogAnalyzer]: disable logrotate cron job during the test Apr 27, 2018

stcheng reviewed Apr 30, 2018

View reviewed changes

ansible/roles/test/files/tools/loganalyzer/loganalyzer_init.yml Outdated

Copy link
Copy Markdown

Contributor

stcheng Apr 30, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put into /tmp?

stcheng added the Enhancement label Apr 30, 2018

jleveque reviewed Apr 30, 2018

View reviewed changes

[logrotate]: Accumulate syslogs rather than force logrotate

f1fd1cd

Signed-off-by: Sihui Han <sihan@microsoft.com>

sihuihan88 force-pushed the dev/sihan/logrotate branch from 9cb5a55 to f1fd1cd Compare May 17, 2018 21:55

sihuihan88 changed the title ~~[LogAnalyzer]: disable logrotate cron job during the test~~ [LogAnalyzer]: Accumulate syslog files rather than force logrotate May 17, 2018

sihuihan88 merged commit 0454ca1 into sonic-net:master Jun 6, 2018

sihuihan88 added a commit that referenced this pull request Jun 11, 2018

Revert "[logrotate]: Accumulate syslogs rather than force logrotate (#…

114ca32

…590)" This reverts commit 0454ca1.

sihuihan88 mentioned this pull request Jun 11, 2018

Revert "[LogAnalyzer]: Accumulate syslog files rather than force logrotate" #641

Merged

sihuihan88 added a commit that referenced this pull request Jun 11, 2018

Revert "[logrotate]: Accumulate syslogs rather than force logrotate (#…

c225eb5

…590)" (#641) This reverts commit 0454ca1.

stepanblyschak pushed a commit to stepanblyschak/sonic-mgmt that referenced this pull request Jul 4, 2018

Revert "[logrotate]: Accumulate syslogs rather than force logrotate (s…

be8f3ab

…onic-net#590)" This reverts commit 0454ca1.

stepanblyschak pushed a commit to stepanblyschak/sonic-mgmt that referenced this pull request Jul 4, 2018

[logrotate]: Accumulate syslogs rather than force logrotate (sonic-ne…

763ed74

…t#590) Signed-off-by: Sihui Han <sihan@microsoft.com>

stepanblyschak pushed a commit to stepanblyschak/sonic-mgmt that referenced this pull request Jul 19, 2018

Revert "Revert "[logrotate]: Accumulate syslogs rather than force log…

f6a981b

…rotate (sonic-net#590)" (sonic-net#641)" This reverts commit c225eb5.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[LogAnalyzer]: Accumulate syslog files rather than force logrotate#590

[LogAnalyzer]: Accumulate syslog files rather than force logrotate#590
sihuihan88 merged 1 commit intosonic-net:masterfrom
sihuihan88:dev/sihan/logrotate

sihuihan88 commented Apr 27, 2018 •

edited

Loading

Uh oh!

stcheng Apr 27, 2018

Uh oh!

sihuihan88 Apr 27, 2018

Uh oh!

lguohan Apr 27, 2018

Uh oh!

lguohan commented Apr 30, 2018

Uh oh!

stcheng Apr 30, 2018

Uh oh!

sihuihan88 commented Apr 30, 2018 •

edited

Loading

Uh oh!

jleveque Apr 30, 2018

Uh oh!

sihuihan88 Apr 30, 2018

Uh oh!

jleveque May 1, 2018 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

sihuihan88 commented Apr 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Approach

Documentation

Uh oh!

stcheng Apr 27, 2018

Choose a reason for hiding this comment

Uh oh!

sihuihan88 Apr 27, 2018

Choose a reason for hiding this comment

Uh oh!

lguohan Apr 27, 2018

Choose a reason for hiding this comment

Uh oh!

lguohan commented Apr 30, 2018

Uh oh!

stcheng Apr 30, 2018

Choose a reason for hiding this comment

Uh oh!

sihuihan88 commented Apr 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jleveque Apr 30, 2018

Choose a reason for hiding this comment

Uh oh!

sihuihan88 Apr 30, 2018

Choose a reason for hiding this comment

Uh oh!

jleveque May 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sihuihan88 commented Apr 27, 2018 •

edited

Loading

sihuihan88 commented Apr 30, 2018 •

edited

Loading

jleveque May 1, 2018 •

edited

Loading