How to Enable Debug Logs in Nginx

Enabling the debug log level in Nginx provides tremendous visibility into the real-time operations and internal workings of the high-performance web server. Detailed debug message capture can be invaluable for troubleshooting complex application, connectivity, and web traffic routing issues. However, careful consideration of storage requirements, security implications, and log analysis capabilities is necessary to fully leverage the power of Nginx debugging.

Nginx Logging Fundamentals

As a web server, Nginx maintains logs to record important operational events, user activity, errors, and notifications. This serves administrative purposes for security, analytics, and troubleshooting. There are two primary log types in Nginx:

• Error Log – Captures diagnostic messages about runtime problems, failures, and significant events.
• Access Log – Tracks detailed metadata about every client request received by the server.

These can be enabled in the main Nginx config file nginx.conf using the error_log and access_log directives. By default, minimal "notice" level details are captured.

Logging Levels

The verbosity and types of messages that Nginx records can be controlled by setting the log level. From least to most verbose, the log levels are:

• Emerg: System unavailable. Requires immediate attention.
• Alert: Critical issues like database outages. Need intervention.
• Crit: Serious but manageable errors like failed disk writes.
• Error: All standard runtime errors. Default level.
• Warn: Unusual but non-critical conditions.
• Notice: Significant normal events like service reloads.
• Info: Status messages that can assist tracing request flow.
• Debug: Very detailed debugging traces and connection metrics.

Day-to-day operations typically run at "error" logging level to capture only major runtime problems and events. Debug level enables expansive message capture for diagnosing complex issues.

Enabling Debug Logging

Here are the steps to configure debug level logging in Nginx:

1. Edit the top-level config file:
   sudo vim /etc/nginx/nginx.conf
2. Within the context of a specific website server { } block, include the error_log directive:
   
   error_log /var/log/nginx/example.com.error.log debug;
3. Save changes and test for syntax errors:
   sudo nginx -t
4. Reload the Nginx service:
   sudo systemctl reload nginx

This will enable debug level logging specifically for that website server context, without cluttering logs from other hosted sites.

Global Debug Logging

To turn on debug traces for all Nginx servers and traffic, place the error_log directive at the top-level outside any server blocks:

error_log /var/log/nginx/nginx.error.log debug;

Just be aware this will rapidly fill up log storage.

Key Debug Log Contents

With debug level enabled, some specific message types to expect in the Nginx error logs include:

• Configuration Parsing: Details about parsing the Nginx config files at startup or following reloads.
• Server Initialization: Status messages about setting listener sockets, ports, and SSL certificates.
• Connection Handling: Metrics on active connections, requests per second, SSL handshakes, etc.
• Request Processing: Detailed headers, rewrite rules evaluation, proxy behavior, response codes sent.
• Access Control: Specific logins, CORS policy checks, denial details.
• Upstream Communications: Performance stats and connectivity issues to origin servers and datastores.

These rich debug messages expose the internals of how Nginx handles traffic, applies configuration, routes requests, and interacts with application infrastructure.

Contrasted with Info Log Level

The "info" log level provides less detail than debug tracing. Info level captures status messages on service health metrics, configuration reloads, ssl certificate status, and high-level connection details. It outputs steady-state notifications rather than voluminous call tracing. Info logs can establish baseline visibility in between debug troubleshooting sessions.

Storage and Retention Implications

The verbosity of debug logs necessitates planning for much larger storage capacity compared to standard Nginx logging. Exact growth rates will depend heavily on web traffic volume. As a starting guideline:

Log Level	Typical Size Per Day
Error logging	~50 MB
Debug logging	~2 GB

For a high traffic site, debug logs can quickly accumulate gigabytes or terabytes unless archived. Log rotation should be configured to periodically compress and move older logs to secondary storage. For example:

error_log /var/log/nginx/site.error.log debug daily;

To keep debug logs for analytics or forensics after rotation, they should be archived on a centralized log server or durable object storage. Prominent open source options include:

• ELK Stack: Elasticsearch for storage, Logstash for collection, Kibana for reporting.
• Graylog: Aggregates logs across servers for search and alerts.
• S3 Buckets: Provides massive scalable cloud object storage.

Multi-Server Considerations

For Nginx server farms, a centralized logging layer is crucial to aggregate the debug traces from all web nodes into an indexable archive. Rather than local disk logs, agents like Filebeat or Logstash can forward Nginx logs continuously to a central platform. Common aggregators include:

• Elasticsearch
• Splunk
• Apache Hadoop
• InfluxDB

This provides holistic visibility across the server landscape for debugging complicated interdependent issues.

Security Considerations

Since verbose debug logs can reveal IP addresses, request parameters, headers, and backend communications, safeguards are necessary for sensitive information.

Ensure debug level logging only captures essential data. Mask security and compliance fields like API keys that could expose vulnerabilities.

Lock down log file permissions and transfer channels. Encrypt flows and obfuscate logs at rest when handling personal data. Limit query access to authorized teams. Debug trails provide powerful forensic analysis if mishandled.

Advanced Configuration

Beyond a global debug log level, Nginx enables very precise control over trace capture using conditional logging filters and output streams:

Conditional Logging

Filter debug logging to specific client IP ranges, request types, domains, response codes, etc. For example, to selectively target debugging a site backend integration:


map $request_uri $logdetail {
default info;
~^/api/v1 debug;
}

error_log /var/log/nginx/debug.log $logdetail;


Multiple Log Sinks

Split debug traces by website or application area into separate logs using named output streams:


http {

log_format payments ‘$remote_addr - $remote_user ...‘;

error_log /var/log/nginx/errors.log;

access_log /var/log/nginx/payments.log payments;

...
}

This segments payment related entries away from the main error logs for easier analysis.

Downstream Integration

For tracking high debug log volumes, integrate Nginx with infrastructure monitoring like Prometheus. Capture metrics on log size and growth rate. Plot this over time and trigger alerts on excessive increments:

• job_name: ‘nginx_log_monitoring‘
  metrics_path: /nginx_status
  static_configs:
  • targets:
    • nginx.logs.example.com


Feed this into Grafana dashboards for convenient visualization alongside other system health metrics.

Log data can also be streamed via Fluentd or Apache Kafka queues to feed analytics platforms, SIEMs, and full text search engines to empower more advanced debugging capabilities.

Getting the Most From Debug Logs

While enabling debug logging in Nginx provides low-level visibility into web transactions, additional tools significantly improve troubleshooting efficiency:

Central Log Analysis Platform

The raw textual logs can be hard to navigate. Integrating the debug trails with a specialized analysis solution like an ELK stack or Graylog enables interactive search, statistics, filtering, annotations, and shareable reporting to accelerate root cause discovery.

Application Performance Monitoring

APMs like New Relic tie code-level application diagnostics together with Nginx web performance metrics and infrastructure health data. This connects debug logs with the broader environment.

Packet Capture Analysis

For tricky network issues like traffic routing failures or SSL handshake errors, packet sniffing with tcpdump or Wireshark provides network-level debugging to complement Nginx logs.

Alternatives to Debug Logging

Other approaches beyond detailed logging for troubleshooting Nginx include:

• Built-in Status Module – Provides performance metrics on active connections, requests, network I/O, etc.
• Watchdog Monitoring – External health checks test site availability from all geo regions.
• Synthetic User Testing – Automated crawlers mimic visitor workflows to detect end-user-facing issues.
• Profiling – Instruments key functions to measure CPU and memory burn during transactions.

These run on a continuous basis for identifying problems proactively rather than forensically digging through dense debug logs.

Conclusion

Nginx debug level logs provide tremendous internal system visibility at the cost of more elaborate storage and analysis capabilities. For intermittent troubleshooting, debug tracing is very useful for diagnosing complex connectivity and request routing failures based on the detailed event capture. Production logging at info or warn levels with brief debug bursts enables sustainable oversight into Nginx web operations. By centrally aggregating logs and correlating with other signals, observability improves greatly without excessive log volumes. The rich introspection exposed by Nginx debug logging proves invaluable for technically savvy web administrators when taming performance issues or instability of high traffic sites.