Cherry-pick of PR #15422 to 7.x branch. Original message:

Type of change

• Bug fix

What does this PR do?

Introduce CLI flags -environment to control the default logging settings
the Beat should use if no logging is configured. The behavior of -e does
not change. By replacing -e with -environment system in the system
unit file we continue to log to stdout/stderr by default, but users are
still able to overwrite settings.

For now (I'm targeting 7.x) I tried to not introduce any breaking changes one way or the other on the config file or CLI level. So the idea is to replace the -e flag with the -environment flag in the systemd unit file. If -environment systemd is given the default log output is stdout/stderr, while we keep file based logging as default for other environments.

Why is it important?

If beats are started via systemd unit files the -e CLI flag is added to the list of CLI arguments. This was introduced in #8942. The -e flag its purpose is to disable the configured logging output and always log to stdout/stderr. By to enforcing the -e flag the users logging configurations is not honored (for example see the reports in #12024).

Checklist

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation

How to test this PR locally

The PR changes how beats are started as Windows Service, macOS service (installed via launchctl), docker container, and via systemd (deb/rpm). For testing Beats must be installed in a few different environment types to check if logs are correctly written to the default target and if log settings can be overwritten.

Running beat locally via command line:

• Running a beat (e.g. metricbeat with system module) locally without configuring logging output should log to stdout/stderr if -environment systemd is given on the command line
• Running beat without -environment flag, without logging configured should create log files as usual

Running a beat via systemd:

• The Beat should log to journald by default
• Logs should be written to /var/log/<beatname> when setting logging.to_stderr: false and logging.to_file: true

Running beat using old init scripts:

• Install Beat on system still using init scripts
• Logs should be written to /var/log/<beatname> by default

Running as container:

• The Beat should log to stdout/stderr by default. Check via docker log is this is really the case.
• Configure to_file in the config file and log to a mounted directory. Check the Beat now writes logs to the configured path.

Running on windows

• Install the windows package zip file on windows:
  • check the service is correctly installed
  • check the service can be started
  • check the service writes logs to the configured files

Running as service on macOS

• Install the packaged Beat on macOS. Including launchd files and preference pane:
  • check the service is correctly installed and can be restarted
  • check the service writes logs to the configured files

Related issues

• Closes: [all beats] Remove BEAT_LOG_OPTS from the systemd unit file #12024
• Closes: Default CMD does not allow syslog/file output for logging #12667