Skip to content

[7.x](backport #23457) Add support for copytruncate method when rotating input logs with an external tool in filestream input#26475

Merged
kvch merged 1 commit into7.xfrom
mergify/bp/7.x/pr-23457
Jun 24, 2021
Merged

[7.x](backport #23457) Add support for copytruncate method when rotating input logs with an external tool in filestream input#26475
kvch merged 1 commit into7.xfrom
mergify/bp/7.x/pr-23457

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify bot commented Jun 24, 2021

This is an automatic backport of pull request #23457 done by Mergify.


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.io/

…external tool in `filestream` input (#23457)

## What does this PR do?

The PR makes the `filestream` log rotation aware to make sure Filebeat can cooperate better with external log rotation tools. The first supported strategy is `copytruncate`.

When `logrotate` rotates e.g. `boot.log` with `copytruncate` the following things happen:
1. all archived files are renamed e.g. `boot.log.2` is renamed `boot.log.3` until `boot.log.1` no longer exists
2. `boot.log` is copied to `boot.log.1`
3. `boot.log` is truncated

You can see my tests on my machine:

Before rotation:
```
root@sleipnir:/home/n# ls -lisaht /var/log/boot.log*
130476 30K -rw------- 1 root root 28K Jan 29 08:59 /var/log/boot.log
130577 36K -rw------- 1 root root 34K Jan 29 08:59 /var/log/boot.log.1
130657 60K -rw------- 1 root root 57K Jan  7 09:51 /var/log/boot.log.2
```
After rotation:
```
root@sleipnir:/home/n# ls -lisaht /var/log/boot.log*
130476   0 -rw------- 1 root root   0 May 25 12:41 /var/log/boot.log
130430 30K -rw------- 1 root root 28K May 25 12:41 /var/log/boot.log.1
130577 36K -rw------- 1 root root 34K Jan 29 08:59 /var/log/boot.log.2
130657 60K -rw------- 1 root root 57K Jan  7 09:51 /var/log/boot.log.3
```

On rotation, the active file is continued and archived files are kept open until EOF is reached.

### Configuration

```yaml
rotation.external.strategy.copytruncate:
  suffix_regex: \.\d$
  count: 10
```

Note: when Filebeat will be able to rotate input logs, its configuration will be under `rotation.internal.*`.

## Why is it important?

Previously, Filebeat was not able to cooperate with external log rotation tools that used `copytruncate` method.

(cherry picked from commit 1f5198e)
@mergify mergify bot added the backport label Jun 24, 2021
@mergify mergify bot assigned kvch Jun 24, 2021
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 24, 2021
@botelastic
Copy link
Copy Markdown

botelastic bot commented Jun 24, 2021

This pull request doesn't have a Team:<team> label.

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26475 opened

  • Start Time: 2021-06-24T13:59:38.315+0000

  • Duration: 113 min 40 sec

  • Commit: c15c0e9

Test stats 🧪

Test Results
Failed 0
Passed 14186
Skipped 2325
Total 16511

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 14186
Skipped 2325
Total 16511

@kvch kvch merged commit 22f647a into 7.x Jun 24, 2021
@mergify mergify bot deleted the mergify/bp/7.x/pr-23457 branch June 24, 2021 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport needs_team Indicates that the issue/PR needs a Team:* label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants