Logrotate Ubuntu Tutorial: A Comprehensive Guide for Developers

As a full-stack developer and Linux systems administrator, log management is one of my essential daily tasks. Applications can easily generate gigabytes of log data every day – sometimes every hour! Without a way to automatically control log sizes, I‘d constantly battle storage capacity issues and obscure bugs caused by gigantic log files.

That‘s where the venerable logrotate tool comes in. Logrotate is included in essentially every Linux distro and runs as a cron job to combat bloated log files. In this comprehensive tutorial I‘ll cover all facets of configuring logrotate on Ubuntu to expertly wrangle logs.

The Growing Threat of Uncontrolled Log Volumes

Industry analysts predict log volumes doubling every 2-3 years. And a 2021 Docker report found that 25% of all file storage consists of log data. Dealing with such explosive growth manually would be a nightmare!

Thankfully Linux distros include logrotate to save us from being buried under an avalanche of log files. It‘s been rotating and compressing sysadmin‘s logs for over 20 years.

An Introduction to Logrotate Capabilities

Logrotate allows you to:

• Automatically rotate logs on a scheduled basis
• Compress rotated logs to save space
• Delete old logs based on time periods or number of rotations
• Run custom scripts before/after rotating logs
• Ensure applications smoothly handle log rotations

This prevents any one log file from consuming too much space. Plus log history is retained for troubleshooting needs.

The main logrotate config file is /etc/logrotate.conf containing global defaults. Additional application-specific rules get defined under /etc/logrotate.d.

Now let‘s dive into mastering your logs with logrotate on Ubuntu!

Checking Your Logrotate Version

See what version of logrotate is installed by running:

logrotate -V

On Ubuntu 18.04 you‘ll likely see logrotate 3.11.0. Ensure your major version is at least 3+ for stable long term support.

Setting Global Logrotate Defaults

Open the main configuration file to edit logrotate‘s global behavior:

sudo vim /etc/logrotate.conf

Some key settings include:

Directive	Description
weekly	Rotate logs weekly by default unless overridden elsewhere
rotate 4	Retain 4 weeks of rotated logs before deletion
create	Instantly create a new empty log file when rotating old

Customize these defaults to meet your specific log management needs. Just take care not to accidentally remove logs before dependent applications finish writing!

Viewing Service-Specific Logrotate Configs

Most well-behaved Linux services and applications include a logrotate config file under /etc/logrotate.d – over 100 different configs are possible!

For example let‘s examine nginx and mysql:

sudo vim /etc/logrotate.d/nginx

sudo vim /etc/logrotate.d/mysql

These contain log paths, rotation frequency, rotation count, compression format, and other options tailored for each application‘s logging behavior.

Creating a Custom Logrotate Config

Let‘s walk through building a custom logrotate configuration from scratch to manage our application‘s logs.

We‘ll setup rotation rules for a hypothetical Django app called my-project writing logs to /var/log/my-project/*.log.

First, create the new config file:

sudo vim /etc/logrotate.d/my-project  

Add these logrotate rules:

/var/log/my-project/*.log {
    weekly
    rotate 12
    compress
    delaycompress
    missingok
    notifempty
}

This rotates logs weekly, keeping 12 rotated copies before removal. Compression helps minimize storage usage.

Save the file then manually test it by running:

sudo logrotate -f /etc/logrotate.d/my-project

Verify your /var/log/my-project/*.log files correctly rotated. The -f forces an immediate rotation.

Now our Django app logs will automatically rotate weekly!

Setting Log Rotation Frequency

By default logrotate rotates logs weekly according to /etc/logrotate.conf. But it‘s easy to customize:

Frequency	Directive	Notes
Daily	daily or rotate 1	Useful for very chatty logs
Weekly	weekly	The default
Monthly	monthly	Good for intermittent logs
Yearly	yearly or annually	Rotate yearly for archival logs
N Days	rotate 5	Rotate every X days
Size Based	size 10M	Rotate when over 10MB vs time

Simply declare an alternative frequency in any config file. The more verbose logs your app produces, the more often you‘ll want to rotate.

I prefer two week rotations for most services. But applications like Docker or MySQL may benefit from larger weekly or even daily rotations.

Limiting Number of Retained Rotated Logs

By default logrotate retains 4 weeks (28 days) of rotated logs before deletion. You can lower this number to save disk space by configuring the rotate directive:

rotate 5

The above setting keeps only the 5 most recent rotated logs before removing oldest files.

Another option is using maxage to expire old logs after a set number of days:

maxage 30 

The above deletes any logs older than 30 days, regardless of how many rotations occurred.

When setting limits, balance keeping logs long enough to troubleshoot issues against running low on disk space!

Compressing Rotated Logs

Compressing rotated logs drastically reduces the storage space logs consume. To enable compression across all services and apps, open /etc/logrotate.conf and uncomment:

compress

By default gzip provides general compression, but other algorithms are available:

Method	Savings	Speed	Support
gzip	65-90%	Fast	Universal support
bzip2	70-90%	Slower	Common support
lzop	65-75%	Faster than gzip	Less support
xz	80-90%	Very slow	Increasing support

Or for more selective compression, declare it in per-application configs:

compress 
compresscmd xz

This enables xz specifically for that app‘s logs only.

But beware – previously declaring nocompress overrides individual compress directives!

Archiving Logs Instead of Deleting

If regulations require keeping all log history beyond normal rotation policies, use the create directive instead:

create 0660 root adm

This archives rotated logs with a timestamp like /var/log/my-app-20201205.log instead of removing logs.

The arguments set permissions and ownership matching most log files under /var/log.

There‘s also a dateext option to use more granular timestamps for archives. And the dateyesterday extension variant names logs based on the date prior to rotation rather than current date.

See the logrotate man page for additional archival configuration details.

Adding Pre/Post Rotation Scripts

The prerotate and postrotate directives allow you to trigger custom commands before and after rotating logs:

prerotate
    /bin/systemctl kill -SIGUSR1 my-app
endscript

postrotate
    /bin/systemctl start my-app
endscript

Here we reload the app prior to log rotating, then restart it after. This ensures the app transitions smoothly across log file changes.

The script directives accept either inline commands or script files to include more complex log handling tasks:

prerotate
    /opt/scripts/pre-log-rotate.sh
endscript

You‘re only limited by your imagination on useful logrotate automation!

Notifying Services of Log Rotations

The create group directive activates real-time logging notifications:

create 660 root adm group log-notify

This sets the SGID permission on new empty log files after rotation. Applications that are members of the log-notify group automatically get signals to reopen logs.

For example, many Systemd services belong to the adm group and handle logrotate notifications gracefully.

See the logrotate man page for more notify options like shared scripts and using syslog.

Mailing Logs Upon Rotation

To receive an email containing your newly rotated application logs, use the mail directives:

mail support@mycompany.com
mailfirst

This emails the rotated log file to the defined address after the first rotation event occurs. The maillast option instead sends only the final log before deletion.

You can also configure a custom mailing script:

mail /opt/scripts/email-log.sh

Handy for compliance reporting needs or reviewing logs from remote systems!

Ensuring Log File Integrity

By default, logrotate renaming log files and lack of application notification can lead to log entries getting lost or duplicated between rotations.

The copytruncate option eliminates this risk:

copytruncate

It safely rotates logs by copying contents to a new file, then truncating original log back to 0 size. This guarantees no log entries get lost across all services.

Troubleshooting Logrotate Issues

If you suspect issues with logrotate not properly managing logs, use verbose debug mode:

sudo logrotate -d -f /etc/logrotate.d/mysql

The -d flag shows detailed output and -f forces an immediate rotation to test.

Look for any errors about opening, rotating, compressing, mailing logs. Permissions problems are common culprits!

Also check time intervals match need – chatty logs may need daily instead of weekly turns.

Conclusion

I hope this guide provided a solid grasp of logrotate‘s purpose along with expert-level knowledge on configuring log rotation. Proper log management is essential for maintaining high performing Linux systems and meeting compliance needs.

While logrotate‘s defaults work well, don‘t be afraid to customize rotation frequency, retention limits, compression style, and append custom scripts to fit your application‘s logging patterns. Automating log upkeep beats manual cleanups any day!

Now you have the power to stop log file overload from ruining your day. Go forth and rotate those logs!