Monitoring Linux Network Statistics

As a Linux system administrator, having visibility into network activity and performance is critical for troubleshooting issues, planning capacity, detecting security threats, and optimizing infrastructure. There are several simple yet powerful open-source tools available for monitoring interface and traffic stats on Linux systems. This comprehensive guide covers the most popular options along with best practices for leveraging them effectively.

Why Network Monitoring Matters

In any non-trivial network, actively monitoring bandwidth utilization, latency, errors and traffic flows provides invaluable visibility and diagnostics. By collecting and analyzing network statistics over time, administrators can:

• Establish expected baselines – What is the typical bandwidth range for a given network segment? How low is the average latency? How many errors per minute are normal? Defining normal behavior makes it faster and easier to detect problems when they emerge.

• Capacity planning – Reviewing peak usage times and growth trends allows properly sizing pipes to meet demand without over-provisioning and overspending.

• Detect DoS attacks – Surges in traffic or unusually high bandwidth utilization may indicate a denial of service attempt or other security breach.

• Identify bandwidth hogs – If capacity nears saturation during certain hours, drilling down can help locate particularly heavy users or applications so limits can be enforced.

• Troubleshoot issues – Unusually high latency, errors and traffic flows can pinpoint physical problems like faulty equipment as well as misconfigurations needing address.

• Optimize applications – Tracking usage by protocol and port can help administrators better allocate resources based on service priority and load.

Key Metrics to Monitor

Some core network metrics that should be monitored over time to establish normal thresholds include:

• Utilization – incoming/outgoing bandwidth used on each interface
• Saturation % – utilization as a percentage of total link capacity
• Latency – lag between sending and receiving traffic
• Errors – discards, overruns, frame/checksum errors
• Traffic flows – breakdown of usage by protocol, IP address, TCP/UDP port

Looking at historical trends allows sane alerting thresholds to be defined – for example triggering alarms when saturation exceeds 80% or latency doubles the normal average.

Network Monitoring Tools Overview

There are many excellent open source tools available for gathering network statistics on Linux. This guide covers some of the most popular and capable options:

Iptraf – Interactive, customizable CLI-based monitoring with break downs and filters

vnStat – Background collector for historical interface usage reporting

IfTop – Real-time display of current bandwidth usage by connection

ntopng – Advanced web-based monitoring and analysis with enterprise capabilities

Understanding the strengths of each tool is key to leveraging them effectively. For optimal visibility, they are often used together to provide complementary dataset.

Iptraf

Iptraf is an interactive, customizable tool for gathering a variety of network statistics in real-time at the CLI. It works by putting network interfaces into promiscuous mode to collect traffic data across the wire.

Features include:

• Breakdowns by TCP/IP connection, protocol, packet size
• Logging and customizable reporting
• Traffic segmentation with filtering by IP, subnet, port, protocol
• Supports Ethernet, FDDI, Token Ring, Wireless (802.11) interfaces

To install on Ubuntu/Debian:

sudo apt install iptraf-ng

Launching iptraf brings up the main menu:

The IP traffic monitor provides a live, sortable view of bandwidth usage with breakouts by IPv4 vs IPv6 as well as TCP vs UDP:

Drilling down further, the Statistical breakdowns menu displays elegant segmentation of traffic by packet size, IP protocol, TCP/UDP port usage, and source/destination IP address:

Examining bandwidth usage by protocol helps determine what applications may be driving higher network utilization. Tracking usage by source/dest IP can identify particularly heavy users which helps with capacity planning.

Custom filters provide a powerful mechanism to isolate suspicious traffic for further diagnostics – for example, any connections from IP blocks registered abroad. Define criteria under Filters:

vnStat

Unlike iptraf which shows live data, vnStat is oriented around historical reporting. Interface statistics get collected in the background and summarized hourly, daily or monthly.

Key features include:

• Lightweight data collector for long-term monitoring
• Traffic reports with tables for hourly, daily and monthly summaries
• Handy visualization of historical trends and growth via CLI plots
• Configurable interface, logging, data retention

Installation is straightforward on most Debian/Ubuntu systems:

sudo apt install vnstat 

By default vnStat will begin tracking the primary network interface, typically eth0. This can be overridden by editing /etc/vnstat.conf.

Daily usage reports help identify peak hours and days with particularly heavy bandwidth demand:

vnstat -d  

                               rx      /      tx      /     total    /   estimated
2020-10
Mon                 129.05 MiB /   90.63 MiB / 219.68 MiB  /   75.31 %

Tue                 357.31 MiB /  279.47 MiB / 636.79 MiB  /   75.76 %

Wed                 966.55 MiB /    1.23 GiB /    2.19 GiB  /   75.29 %
                      ...

Monthly outputs provide a broader overview of trends which helps planning capacity:

vnstat -m  

                         rx      /      tx      /     total    /   estimated
2020-10     12.04 GiB /  10.55 GiB /   22.60 GiB /   75.58 %  

2020-09     11.10 GiB /  9.20 GiB  / 20.30 GiB  / 75.23 %
                      ...

The estimated percentage gives a confidence measure of accuracy based on the sample rate configured.

Integrating vnstat with monitoring stacks like Graphite or Grafana allows visualizing usage growth over longer periods. This helps spot abnormal changes in traffic requiring investigation.

IfTop

While vnstat and iptraf provide cumulative bandwidth statistics, iftop offers visibility into current utilization on a per-connection basis. It essentially provides a real-time view of active sockets using send/receive bandwidth.

Typical iftop output displays:

• Sorted list of connections by recent send/receive throughput
• Total send/receive for all connections in the sample period
• Breakdown of cumulative bandwidth per protocol

iftop makes it very easy to immediately identify clients and servers driving high bandwidth utilization which helps locate usage spikes.

Filtering capabilities allow isolating traffic from specific subnets, hosts etc. For example, to only show traffic to/from a MySQL server:

sudo iftop -F 10.20.30.5/32

Like most monitoring tools, iftop supports both IPv4 and IPv6 environments.

ntopng

While the other solutions covered excel at exposing detailed network metrics at the CLI, ntopng provides a feature-packed web-based interface with advanced enterprise capabilities.

Benefits include:

• Modern GUI with sorting/filtering/searching to quickly locate traffic
• Customizable dashboards
• Geographical maps to easily identify traffic locality
• Sophisticated flow analytics – sflow/netflow inspection, application breakdowns, autonomous system visualization etc
• Role-based access control, REST APIs etc for complex environments

ntopng allows much easier data exploration compared to CLI tools while still providing very sophisticated diagnostics leveraging netflow metadata as well as heuristics like DNS analysis for classifying traffic. The web interface facilitates consuming meaningful network statistics in larger environments with many servers/network devices compared to combing through raw text logs.

Integrating Monitoring into Alerts

While actively watching dashboards can help spot anomalies, configuring alert notifications is critical for timely detection and rapid diagnostics. Alerting criteria can vary based on specific environment and risk tolerance, but some useful triggers include:

• Links exceeding 80% average utilization (risk of congestion)
• Traffic volume exceeding 2 standard deviations above typical peak
• Error rates increasing 2x above normal baseline
• Any detected DoS attack patterns

Most monitoring systems provide flexible options for alarm notifications via email, SMS, chat bots, support tickets etc. Triggering automatic alerts when abnormal network patterns emerge allows rapid investigation and mitigation.

Sample Grafana Dashboard

Here is an example Grafana dashboard correlating interface metrics from multiple Linux servers to provide unified network visualization:

This consolidates key bandwidth, error and traffic flow metrics to quickly highlight systems experiencing abnormalities like usage spikes or latency. Consolidating statistics from complementary tools gives fuller picture – for example overlaying real-time iftop connections on historical vnstat interface utilization charts.

Troubleshooting Using Statistics

Network utilization reports and graphs provide clues for troubleshooting performance issues or suspicious traffic patterns:

• Bandwidth saturation at peak hours – Adding capacity or setting QoS priorities can help
• Unusually high overnight bandwidth – Could indicate a cryptojacker malware infection
• Latency doubling baseline – Potential faulty equipment or WiFi interference
• Traffic drops over time – Network misconfiguration like incorrect MTU

Here are some real-world examples and investigative techniques:

Case 1: Website loading slow – Leverage geographic IP maps to localize latencies. Review traffic by application protocol and port. Checks DNS response times. If isolated to extarnal hosting provider network, open support ticket for investigation.

Case 2: Bandwidth spike alerts triggering – Leverage real-time connections view in iftop to identify top talkers by IP. Further isolate suspected ip blocks with filters. Run scans on systems using excessive bandwidth to check for malware or misconfigurations. View historical monthly bandwidth trends to determine if growth related.

Case 3: Packet loss / errors – Interface error counters can localize physical port or equipment issues. Loss on WiFi indicates potential interference. Leverage iperf, mtr tests to differentiate network hardware vs application code issues.

Final Thoughts

As discussed, actively monitoring key Linux network metrics provides invaluable visibility for performance tuning, diagnostics and planning. This guide provided an overview of common CLI monitoring tools – iptraf for granular real-time segmentation, vnstat for historical interface logging, iftop for live bandwidth usage by connection, and ntopng for advanced and elegant visualization leveraging netflow.

These tools work best together to provide well-rounded analytics. vmstat exposes long-term trends, while iptraf segments traffic flows and iftop highlights current talkers. Feeding stats into centralized time-series DBs allows powerful dashboards and alerts. Visualizing network history spotlights developing issues like usage surges or loss spikes before they escalate. Mastering open-source traffic analysis unlocks deeper understanding.