The awscloudwatch input can skip data in its target log groups, with severity depending on configuration and log size. This seems to apply to all platforms and all versions at least since 8.0.
Easy reproduction:
- Set
number_of_workers to 1 (the default)
- Set
start_position to beginning (the default)
- Set
log_group_name_prefix to a value matching 2 or more log groups
The first matching log group will ingest data starting from the beginning, but all other log groups will only include data from after ingestion began.
This loss continues during ingestion: events from any time span will only include data from at most one log group at a time.
More finicky reproduction with a single log group:
- Set
number_of_workers to 1 (the default)
- Set
start_position to beginning (the default)
- Target a single log group with a significant amount of past data (enough to require significantly longer than the
scan_frequency to ingest -- optionally set scan_frequency to 1s to make this easier)
Data added to the log group between the start of ingestion and the completion of the first scan will be skipped.
The
awscloudwatchinput can skip data in its target log groups, with severity depending on configuration and log size. This seems to apply to all platforms and all versions at least since 8.0.Easy reproduction:
number_of_workersto 1 (the default)start_positiontobeginning(the default)log_group_name_prefixto a value matching 2 or more log groupsThe first matching log group will ingest data starting from the beginning, but all other log groups will only include data from after ingestion began.
This loss continues during ingestion: events from any time span will only include data from at most one log group at a time.
More finicky reproduction with a single log group:
number_of_workersto 1 (the default)start_positiontobeginning(the default)scan_frequencyto ingest -- optionally setscan_frequencyto1sto make this easier)Data added to the log group between the start of ingestion and the completion of the first scan will be skipped.