Skip to content

Commit c15c0e9

Browse files
kvchmergify-bot
kvch
authored and
mergify-bot
committed
Add support for copytruncate method when rotating input logs with an external tool in filestream input (#23457)
## What does this PR do? The PR makes the `filestream` log rotation aware to make sure Filebeat can cooperate better with external log rotation tools. The first supported strategy is `copytruncate`. When `logrotate` rotates e.g. `boot.log` with `copytruncate` the following things happen: 1. all archived files are renamed e.g. `boot.log.2` is renamed `boot.log.3` until `boot.log.1` no longer exists 2. `boot.log` is copied to `boot.log.1` 3. `boot.log` is truncated You can see my tests on my machine: Before rotation: ``` root@sleipnir:/home/n# ls -lisaht /var/log/boot.log* 130476 30K -rw------- 1 root root 28K Jan 29 08:59 /var/log/boot.log 130577 36K -rw------- 1 root root 34K Jan 29 08:59 /var/log/boot.log.1 130657 60K -rw------- 1 root root 57K Jan 7 09:51 /var/log/boot.log.2 ``` After rotation: ``` root@sleipnir:/home/n# ls -lisaht /var/log/boot.log* 130476 0 -rw------- 1 root root 0 May 25 12:41 /var/log/boot.log 130430 30K -rw------- 1 root root 28K May 25 12:41 /var/log/boot.log.1 130577 36K -rw------- 1 root root 34K Jan 29 08:59 /var/log/boot.log.2 130657 60K -rw------- 1 root root 57K Jan 7 09:51 /var/log/boot.log.3 ``` On rotation, the active file is continued and archived files are kept open until EOF is reached. ### Configuration ```yaml rotation.external.strategy.copytruncate: suffix_regex: \.\d$ count: 10 ``` Note: when Filebeat will be able to rotate input logs, its configuration will be under `rotation.internal.*`. ## Why is it important? Previously, Filebeat was not able to cooperate with external log rotation tools that used `copytruncate` method. (cherry picked from commit 1f5198e)
1 parent 8fe3d21 commit c15c0e9

20 files changed

Lines changed: 1128 additions & 116 deletions

‎CHANGELOG.next.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -588,6 +588,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
588588
- RFC 5424 and UNIX socket support in the Syslog input are now GA {pull}26293[26293]
589589
- Update grok patterns for HA Proxy module {issue}25827[25827] {pull}25835[25835]
590590

591+
- Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457]
592+
591593
*Heartbeat*
592594

593595
- Bundle synthetics deps with heartbeat docker image. {pull}23274[23274]

‎filebeat/_meta/config/filebeat.inputs.reference.yml.tmpl‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -292,6 +292,16 @@ filebeat.inputs:
292292
# original for harvesting but will report the symlink name as source.
293293
#prospector.scanner.symlinks: false
294294
295+
### Log rotation
296+
297+
# When an external tool rotates the input files with copytruncate strategy
298+
# use this section to help the input find the rotated files.
299+
#rotation.external.strategy.copytruncate:
300+
# Regex that matches the rotated files.
301+
# suffix_regex: \.\d$
302+
# If the rotated filename suffix is a datetime, set it here.
303+
# dateformat: -20060102
304+
295305
### State options
296306
297307
# Files for the modification data is older then clean_inactive the state from the registry is removed

‎filebeat/docs/inputs/input-filestream-file-options.asciidoc‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -482,3 +482,56 @@ Set the location of the marker file the following way:
482482
----
483483
file_identity.inode_marker.path: /logs/.filebeat-marker
484484
----
485+
486+
=== Log rotation
487+
488+
As log files are constantly written, they must be rotated and purged to prevent
489+
the logger application from filling up the disk. Rotation is done by an external
490+
application, thus, {beatname_uc} needs information how to cooperate with it.
491+
492+
When reading from rotating files make sure the paths configuration includes
493+
both the active file and all rotated files.
494+
495+
By default, {beatname_uc} is able to track files correctly in the following strategies:
496+
* create: new active file with a unique name is created on rotation
497+
* rename: rotated files are renamed
498+
499+
However, in case of copytruncate strategy, you should provide additional configuration
500+
to {beatname_uc}.
501+
502+
[float]
503+
==== rotation.external.strategy.copytruncate
504+
505+
experimental[]
506+
507+
If the log rotating application copies the contents of the active file and then
508+
truncates the original file, use these options to help {beatname_uc} to read files
509+
correctly.
510+
511+
Set the option `suffix_regex` so {beatname_uc} can tell active and rotated files apart. There are
512+
two supported suffix types in the input: numberic and date.
513+
514+
==== Numeric suffix
515+
516+
If your rotated files have an incrementing index appended to the end of the filename, e.g.
517+
active file `apache.log` and the rotated files are named `apache.log.1`, `apache.log.2`, etc,
518+
use the following configuration.
519+
520+
[source,yaml]
521+
---
522+
rotation.external.strategy.copytruncate:
523+
suffix_regex: \.\d$
524+
---
525+
526+
==== Date suffix
527+
528+
If the rotation date is appended to the end of the filename, e.g. active file `apache.log` and the
529+
rotated files are named `apache.log-20210526`, `apache.log-20210527`, etc. use the following configuration:
530+
531+
[source,yaml]
532+
---
533+
rotation.external.strategy.copytruncate:
534+
suffix_regex: \-\d{6}$
535+
dateformat: -20060102
536+
---
537+

‎filebeat/filebeat.reference.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -699,6 +699,16 @@ filebeat.inputs:
699699
# original for harvesting but will report the symlink name as source.
700700
#prospector.scanner.symlinks: false
701701

702+
### Log rotation
703+
704+
# When an external tool rotates the input files with copytruncate strategy
705+
# use this section to help the input find the rotated files.
706+
#rotation.external.strategy.copytruncate:
707+
# Regex that matches the rotated files.
708+
# suffix_regex: \.\d$
709+
# If the rotated filename suffix is a datetime, set it here.
710+
# dateformat: -20060102
711+
702712
### State options
703713

704714
# Files for the modification data is older then clean_inactive the state from the registry is removed

‎filebeat/input/filestream/config.go‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ type config struct {
4141
HarvesterLimit uint32 `config:"harvester_limit" validate:"min=0"`
4242
IgnoreOlder time.Duration `config:"ignore_older"`
4343
IgnoreInactive ignoreInactiveType `config:"ignore_inactive"`
44+
Rotation *common.ConfigNamespace `config:"rotation"`
4445
}
4546

4647
type closerConfig struct {
@@ -78,6 +79,17 @@ type backoffConfig struct {
7879
Max time.Duration `config:"max" validate:"nonzero"`
7980
}
8081

82+
type rotationConfig struct {
83+
Strategy *common.ConfigNamespace `config:"strategy" validate:"required"`
84+
}
85+
86+
type commonRotationConfig struct {
87+
SuffixRegex string `config:"suffix_regex" validate:"required"`
88+
DateFormat string `config:"dateformat"`
89+
}
90+
91+
type copyTruncateConfig commonRotationConfig
92+
8193
func defaultConfig() config {
8294
return config{
8395
Reader: defaultReaderConfig(),

0 commit comments

Comments
 (0)