A Complete Guide to Tomcat Logging in Linux

As an experienced full-stack developer and Linux systems administrator, logging is an area I cannot emphasize enough. Robust logging and monitoring provides invaluable observability into your applications and servers. This enables you to troubleshoot issues faster, understand usage patterns to improve performance, lock down security, and much more.

In this comprehensive 2600+ word guide, I will impart the logging best practices and techniques I‘ve learned from over a decade of running Java applications on Tomcat across dozens of Linux servers. By the end, you will have an in-depth understanding of the intricacies of Tomcat logging and be equipped to build secure, optimized Tomcat deployments.

An Introduction to Tomcat Logging

Apache Tomcat is one of the most popular open-source Java web application servers. At its core Tomcat is a servlet container that handles HTTP requests and serves dynamic content powered by Java and JVM-based languages like Kotlin and Scala.

A standard Tomcat environment has two primary log types:

Catalina Logs: Text-based logs that record internal Tomcat operations, errors, and diagnostic messages.

Access Logs: HTTP transaction logs containing granular request details like IP addresses, URLs, response times, status codes and more.

Additionally there are rolled over history logs for each file, plus stdout logs and JVM platform logs – totaling over a dozen log files from a typical Tomcat instance!

Having clear insight across all these dispersed logs is critical for monitoring the health and security of your Tomcat applications. Next let‘s understand where these logs reside…

Tomcat Log File Locations

Tomcat‘s logs location depends on how you have installed the Tomcat binaries:

Package Installs (apt, yum, etc)

/var/log/tomcat/
├── catalina.out
├── catalina.err
├── localhost_access_log.txt
└── host-manager_access_log.txt

When using Linux package managers, the distro maintains Tomcat as a system service – so logs go to standard /var/log directories.

Binary Installs

$CATALINA_HOME
└── logs
     ├── catalina.out
     ├── catalina.err      
     ├── localhost_access_log.txt
     └── host-manager_access_log.txt

If you maintain your own Tomcat binary install, then logs are written inside the logs/ subdirectory. $CATALINA_HOME points to root install directory.

Custom Log Locations

You can customize Catalina log paths by setting CATALINA_OUT in startup.sh:

CATALINA_OUT=/opt/tomcat/tomcat.log

Access logs stay in default logs/ location, configured via server.xml.

Now that you know where logs reside, let‘s discuss approaches for accessing them…

Viewing & Tailing Tomcat Log Files

Here I cover my favorite Linux commands for reviewing Tomcat logs:

1. Tail Logs in Real-time

My most oft-used trick is tailing log files continuously with tail -f:

tail -f /opt/tomcat/tomcat.log

This prints live logs as they are written – invaluable for monitoring ongoing activity and issues.

2. Less for Inspection

Get a static view of entire log contents with less:

less /opt/tomcat/tomcat.log

Then navigate up/down, search keywords (/warning), and more. Press q to quit.

3. Filtering Logs via grep

Extract matching lines using the versatile grep tool:

grep "SEVERE" /opt/tomcat/*.log

This reveals all SEVERE entries across Tomcat logs – making it easy to pinpoint errors.

I utilize these three simple commands daily for all my Tomcat debugging. Now let‘s examine the primary Catalina logs…

Tomcat‘s Catalina Log Files

The catalina.out file provides Tomcat‘s central text logs – your go-to resource for troubleshooting or monitoring internal operations.

Meanwhile catalina.err contains Java stack traces and exceptions.

Here is a sample excerpt from a catalina.out log:

Note the log format includes timestamp, log level (INFO, WARN), thread name, plus log message body.

Understanding Catalina Logging Levels is key…

Tomcat Catalina Log Levels

Tomcat utilizes Apache Commons Logging for its internal logging format and infrastructure. Commons Logging defines the following log levels by order of severity:

1. FATAL – Unrecoverable crashes
2. ERROR – Critical errors
3. WARN – Concerning issues
4. INFO – Status information
5. DEBUG – Diagnostic debugging
6. TRACE – Extremely detailed traces

By default, Tomcat sets Catalina catalina.out logging to INFO level. This provides a baseline of operational awareness without overwhelming volume.

Configuring additional levels like DEBUG or WARN provides more/less verbose tracing as needed. I‘ll cover that ahead…

But first, let‘s explore Tomcat‘s vital Access Logs.

Tomcat Access Logs

While Catalina logs provide internal visibility – the access logs deliver exterior visibility by tracking all incoming HTTP requests. The central file is imaginatively named:

localhost_access_log.txt

Access logs add crucial context around:

• Number of visitors
• Most requested pages
• Traffic volumes
• Slowest endpoints
• Error rates
• Bots vs humans
• Security threats

Here is a sample access log entry:

127.0.0.1 - john [10/Oct/2022:13:55:36 -0700] "GET /app/ HTTP/1.0" 200 12281

This reveals the:

• Client IP – Visitor IP address
• Identity – Mainly used for proxies
• Timestamp – Date & time
• Request – HTTP method, URL path, protocol
• Status – Response code (200 OK, 404 NotFound, etc)
• Bytes – Content size in bytes

Fun Fact: This common format is actually called the "Common Log Format" – in use since the early days of the web!

Now let‘s explore customizing access logs further…

Customizing Tomcat Access Logs

One issue with the default access log format is it‘s limited to only a few salient fields. Moreover, timestamps only have one second resolution limiting utility for detailed performance analysis.

Luckily you can fully customize access logs by editing Tomcat‘s server.xml configuration file:

vim $CATALINA_HOME/conf/server.xml  

Then search for the AccessLogValve section:

<Valve className="org.apache.catalina.valves.AccessLogValve"  
       directory="logs"
       prefix="localhost_access_log."  
       suffix=".txt" 
       pattern="%h %l %u %t "%r" %s %b" />

These attributes dictate access logging policies:

• directory – Where to write access log files
• prefix/suffix – Filename start/end
• pattern – Layout of access log entries

The pattern value supports syntax from the Common Log Format linked above.

For example, to add timestamps with milliseconds, Java thread name, and query string:

%t %D %I %{my-thread-name}r %q

Now Tomcat will log higher-fidelity access logs enabling better monitoring and profiling!

Next let‘s explore managing these rapidly growing log volumes…

Log Rotation for Tomcat Logs

A chronic challenge with active applications is log files quickly bloat filling up disk space. I‘ve seen 100GB+ Tomcat logs bringing servers to their knees!

Thus intelligently rotating and archiving logs is essential.

Tomcat itself does not handle log rotation – instead rely on Linux‘s logrotate utility.

First install logrotate if needed:

apt install logrotate

Then edit /etc/logrotate.conf and add a Tomcat config section:

/var/log/tomcat/*.log {

    weekly

    rotate 12

    compress

    delaycompress

    missingok

    notifempty

}

This configures logrotate to:

• Rotate logs weekly
• Keep 12 archived log copies
• Gzip compress old logs
• Don‘t error if missing logs
• Don‘t rotate empty logs

Enable other options like mail alerts on rotation errors.

Now your Tomcat logs will neatly rotate weekly – avoiding excessive expansion.

Next I‘ll share how to tune those verbose Catalina log levels…

Configuring Tomcat‘s Catalina Log Levels

We discussed earlier how Tomcat‘s catalina.out uses Apache Commons Logging for internal event reporting. By default this is minimum INFO logs.

You can enrich logging by enabling additional verbosity levels like DEBUG and WARN.

Edit Tomcat‘s $CATALINA_BASE/conf/logging.properties:

cd $CATALINA_HOME/conf
vim logging.properties

Then search for the AsyncFileHandler sections:

1catalina.org.apache.juli.AsyncFileHandler.level = FINE
2localhost.org.apache.juli.AsyncFileHandler.level = FINE 

Add DEBUG, WARN or other levels you wish to enable:

1catalina.org.apache.juli.AsyncFileHandler.level = FINE, DEBUG, WARN
2localhost.org.apache.juli.AsyncFileHandler.level = FINE, DEBUG, WARN

Save changes and restart Tomcat to apply richer logging!

Finally, let‘s discuss fully disabling Tomcat logging…

Disabling Tomcat Logging

In select scenarios like production systems or performance testing, you may need to completely disable all Tomcat logging.

I advise doing this only in narrow testing windows as logs provide so much operational insight.

But to switch off logs across Catalina and access logs:

Catalina Logging

Edit $CATALINA_HOME/conf/logging.properties and comment out the handlers:

#handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler  

Access Logging

In Tomcat‘s server.xml, comment out the AccessLogValve:

<!--
<Valve className="org.apache.catalina.valves.AccessLogValve"  
   ...
/>
-->

Save changes and restart Tomcat to disable all logging output.

Be sure to uncomment when your testing is complete!

Conclusion

In closing, I hope this guide has provided both breadth and depth on managing Tomcat logging in Linux environments. We covered:

• The basics of Catalina and Access logging
• Log file locations across various installs
• Core techniques like tailing, filtering, inspecting
• Customizing access logs for enhanced monitoring
• Configuring log rotation to prevent clogged disks
• Tuning Catalina log verbosities as needed
• Temporarily disabling logging for performance tests

Getting a handle around logs is crucial for unlocking the full potential of Tomcat‘s capabilities in powering Java application environments. Mastering these logging best practices will equip you to develop robust and optimized Tomcat deployments.

Now that you have these skills for parsing, analyzing and configuring Tomcat logs – leverage them to monitor, secure, and enhance the performance of your Java applications!