Network monitoring tools

ToolDescription
ssSocket statistics
ipNetwork interface and route statistics
ifconfigNetwork interface statistics
nstatNetwork stack statistics
netstatVarious network stack and interface statistics
sarHistorical statistics
nicstatNetwork interface throughput and utilization
ethtoolNetwork interface driver statistics
tcplifeTrace TCP session lifespans with connection details
tcptopShow TCP throughput by host and process
tcpretransTrace TCP retransmits with address and TCP state
bpftraceTCP/IP stack tracing: connections, packets, drops, latency
tcpdumpNetwork packet sniffer
WiresharkGraphical network packet inspection
ToolDescription
offcputimeOff-CPU profiling can show network I/O
sockstatHigh-level socket statistics
sofamilyCount address families for new sockets, by process
soprotocolCount transport protocols for new sockets, by process
soconnectTrace socket IP-protocol connections with details
soacceptTrace socket IP-protocol accepts with details
socketioSummarize socket details with I/O counts
socksizeShow socket I/O sizes as per-process histograms
sormemShow socket receive buffer usage and overflows
soconnlatSummarize IP socket connection latency with stacks
so1stbyteSummarize IP socket first byte latency
tcpconnectTrace TCP active connections (connect())
tcpacceptTrace TCP passive connections (accept())
tcpwinTrace TCP send congestion window parameters
tcpnagleTrace TCP Nagle usage and transmit delays
udpconnectTrace new UDP connections from localhost
gethostlatencyTrace DNS lookup latency via library calls
ipecnTrace IP inbound explicit congestion notification
superpingMeasure ICMP echo times from the network stack
qdisc-fq (…)Show FQ qdisc queue latency
netsizeShow net device I/O sizes
nettxlatShow net device transmission latency
skbdropTrace sk_buff drops with kernel stack traces
skblifeLifespan of sk_buff as inter-stack latency
ieee80211scanTrace IEEE 802.11 WiFi scanning
pcharpathchar is similar to traceroute but includes the bandwidth between hops.

Wazuh: Issues encountered and solutions

 

  1. logstash service does not find config files in /etc/logstash/conf.d
    I installed logstash via centos rpm and placed a valid logstash configuration file into /etc/logstash/conf.d. Starting the service it fails with following error message:

    {:timestamp=>"2015-10-21T08:11:06.939000+0000", :message=>"Error: No config files found: /etc/logstash/conf.d/pipe.conf\nCan you make sure this path is a logstash config file?", :file=>"logstash/agent.rb", :line=>"159", :method=>"execute"}
    {:timestamp=>"2015-10-21T08:11:06.948000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system.", :file=>"logstash/agent.rb", :line=>"161", :method=>"execute"}
    

    there is definitely a file in this location read/write to anyone

    [root@logserv logstash]# ls -al /etc/logstash/conf.d/
    total 12
    drwxr-xr-x. 2 root root 4096 Oct 21 08:00 .
    drwx------. 4 root root 4096 Oct 21 07:54 ..
    -rwxrwxrwx. 1 root root  234 Oct 21 07:56 pipe.conf

    and its content is valid

    [root@logserv logstash]# bin/logstash -f /etc/logstash/conf.d/pipe.conf --configtest
    Configuration OK

    Solution:

    chown -R logstash:root /etc/logstash/conf.d
    chmod 0750 /etc/logstash/conf.d
    chmod 0640 /etc/logstash/conf.d/*
  2.  If in the Wazuh UI you see data in wazuh-alerts but not in any of the wazuh dashboards, check if the data is getting pushed to Elasticsearch first:

    curl localhost:9200/_cat/indices

    The output should look like below:

    green open wazuh-alerts-3.x-2019.07.19 GIPOTyJuSxSZgVtsdkouxg 3 0 131 0 424.7kb 424.7kb
    green open .kibana_task_manager cCFAzTqIQ6GuhVtJsfuUrQ 1 0 2 0 29.5kb 29.5kb
    yellow open .wazuh tgqyhP1rQHqRk4bnfvjivg 1 1 1 0 11kb 11kb
    green open wazuh-alerts-3.x-2019.07.20 vbSs-0TRRRKihI3vo67C0w 3 0 10 0 79.7kb 79.7kb
    green open wazuh-alerts-3.x-2019.07.21 GYbynBOLTsedyuxIVfSmig 3 0 9 0 80.7kb 80.7kb
    green open .kibana_1 24p2awqCTFafufPXuTkM_A 1 0 6 2 110.6kb 110.6kb
    green open wazuh-monitoring-3.x-2019.07.18 GtPTclhVS6CveIoTB9s88w 2 0 192 0 174.7kb 174.7kb
    green open wazuh-monitoring-3.x-2019.07.20 skX7aKIMTNa20VKvZdG-gg 2 0 192 0 210.9kb 210.9kb
    green open wazuh-monitoring-3.x-2019.07.17 fERZ9LMeQheBUDo4CFZgbw 2 0 98 0 215.1kb 215.1kb
    green open wazuh-monitoring-3.x-2019.07.21 fDT71M7bSNawPplieIEXRg 2 0 46 0 208.6kb 208.6kb
    yellow open .wazuh-version 2TPSH17YQ4e_n6NiWDhQqQ 1 1 1 0 5.2kb 5.2kb
    green open wazuh-monitoring-3.x-2019.07.19 8RDIhk0EQIOxNIYOOh6VXA 2 0 198 0 140.1kb 140.1kb

    Check if wazuh-alerts-3.x-* index is present with current date. If yes, then check if data is present in the index:

    curl localhost:9200/<INDEX_NAME>/_search?pretty=true&size=1000
    Example:
    localhost:9200/wazuh-alerts-3.x-2019.07.19/_search?pretty=true&size=1000

    This should return first 1000 entries.
    Check if latest entries are present. Also check if manager.name matches the hostname of the manager. If not, then change the hostname of the manager by executing command:

    hostname <HOSTNAME>
    Example:
    hostname abc.example.com

    The alerts in the Elasticsearch index will start coming in with the manager.hostname as abc.example.com

  3. Get details of a Wazuh agent:

    curl -u foo:bar localhost:55000/agents/<AGENT_ID>

    foo:bar is the credentials for wazuh api.

  4. Get details of nodes in Wazuh cluster:
    curl -u foo:bar localhost:55000/cluster/nodes?pretty
  5. Rules not getting reflected aftere change in /var/ossec/ruleset/rules
    Restarting the wazuh-manger should reload the rules:

    systemctl restart wazuh-manager

  6. No events in wazuh-monitoring index:
    Check if index wazuh-monitoring-3.x-* is present with today’s date:

    curl elastic:9200/_cat/indices/wazuh-monitoring*

    Check if there is any error in wazuhapp for wazuh-monitoring:

    cat /usr/share/kibana/optimize/wazuh-logs/wazuhapp-plain.log | grep monitoring

    Execute:

    curl -XGET "http://elastic:9200/_cat/templates/wazuh-agent"

    If you get something like:
    wazuh-agent [wazuh-monitoring*, wazuh-monitoring-3.x-*] 0
    You probably have a template issue. Execute the following to resolve it:

    Stop Kibana: systemctl stop kibana
    
    Delete the monitoring template, curl -XDELETE elastic:9200/_template/wazuh-agent
    
    Restart Kibana: systemctl restart kibana, it should insert the monitoring template and the Kibana UI should start working shortly
  7. Change wazuh app to debug mode:
    Edit /usr/share/kibana/plugins/wazuh/config.yml
    Replace #logs.level: info with logs.level: debug, then restart Kibana service (systemctl restart kibana)
  8. Wazuh UI Error: “Saved field parameter is now invalid” OR “Error in Visualisation: field is a required parameter”
    You will require a cleanup. Execute the following commands:
    Delete wazuh-alerts-3.x-* index with today’s date:

    curl -XDELETE localhost:9200/wazuh-alerts-3.x-2019.07.18
    systemctl restart wazuh-manager
    curl -XDELETE localhost:9200/.kiban*
    systemctl restart kibana
    rm -f /var/ossec/queue/db/0*
    systemctl restart wazuh-manager
  9. Generate SCA alerts:
    rm -f /var/ossec/queue/db/0*
    systemctl restart wazuh-manager
  10. Error in visualisation: Expected numeric type on field on data.sca.score got numeric:
    You most probably have wrong template. Just install the template according to your wazuh version from their github repo. To install latest (3.9.3) execute the following:

    curl https://raw.githubusercontent.com/wazuh/wazuh/v3.9.3/extensions/elasticsearch/6.x/wazuh-template.json | curl -X PUT "http://localhost:9200/_template/wazuh" -H 'Content-Type: application/json' -d @-
  11. java.lang.IllegalArgumentException: Rejecting mapping update to [wazuh-alerts-3.x] as the final mapping would have more than 1 type:
    This would most possibly mean you have wrong template. Install the latest one with above step.
    If already done you might have an issue with your logstash configuration:

    curl -so /etc/logstash/conf.d/01-wazuh.conf https://raw.githubusercontent.com/wazuh/wazuh/v3.9.3/extensions/logstash/7.x/01-wazuh-remote.conf

    Delete wazuh-alerts-3.x-* index with today’s date:

    curl -XDELETE localhost:9200/wazuh-alerts-3.x-2019.07.18systemctl restart wazuh-manager
    curl -XDELETE localhost:9200/.kiban*
    systemctl restart kibana
    rm -f /var/ossec/queue/db/0*
    systemctl restart wazuh-manager
  12. Version mismatch:
    If you get the above error on Wazuh UI, execute the following commands:

    service kibana stop
    curl -XDELETE localhost:9200/.wazuh
    curl -XDELETE localhost:9200/.wazuh_version
    service kibana start
  13. Other useful commands:
    Check documents in an index:
    curl elastic:9200/_cat/indices/wazuh-monitoring*
    
    Check wazuhapp logs:
    cat /usr/share/kibana/optimize/wazuh-logs/wazuhapp-plain.log | grep monitoring
    
    Check wazuh config:
    cat /usr/share/kibana/plugins/wazuh/config.yml
    
    Get Wazuh id:
    curl elastic:9200/.wazuh/_search?pretty -s | grep "_id"
    
    Templates:
    curl elastic:9200/_cat/templates/wazuh
    
    Version:
    cat /usr/share/kibana/plugins/wazuh/package.json | grep version
    cat /etc/ossec-init.conf | grep -i version
    curl -u foo:bar localhost:55000/version
    
    Monitoring settings:
    cat /usr/share/kibana/plugins/wazuh/config.yml | grep monitoring
    
    Search monitoring index:
    curl elastic:9200/wazuh-monitoring*/_search
    
    List of agents:
    curl -u foo:bar "localhost:55000/agents?q=id!=000"
    
    Index settings:
    curl  "elastic:9200/wazuh-monitoring-3.x-2019.07.17/_settings"
    
    Get details of template:
    curl -XGET "http://elastic:9200/_cat/templates/wazuh-agent"
    
    Check if filebeat is configured correctly:
    filebeat test output

Install and configure Wazuh with ELK 6.x

Wazuh helps you to gain deeper security visibility into your infrastructure by monitoring hosts at an operating system and application level. This solution, based on lightweight multi-platform agents, provides the following capabilities:

  • File integrity monitoring

Wazuh monitors the file system, identifying changes in content, permissions, ownership, and attributes of files that you need to keep an eye on.

 

  • Intrusion and anomaly detection

Agents scan the system looking for malware, rootkits or suspicious anomalies. They can detect hidden files, cloaked processes or unregistered network listeners, as well as inconsistencies in system call responses.

 

  • Automated log analysis

Wazuh agents read operating system and application logs, and securely forward them to a central manager for rule-based analysis and storage. The Wazuh rules help make you aware of application or system errors, misconfigurations, attempted and/or successful malicious activities, policy violations and a variety of other security and operational issues.

 

  • Policy and compliance monitoring

Wazuh monitors configuration files to ensure they are compliant with your security policies, standards and/or hardening guides. Agents perform periodic scans to detect applications that are known to be vulnerable, unpatched, or insecurely configured.

This diverse set of capabilities is provided by integrating OSSEC, OpenSCAP and Elastic Stack into a unified solution and simplifying their configuration and management.

Execute the following commands to install and configure Wazuh:

  1. apt-get update

  2. apt-get install curl apt-transport-https lsb-release gnupg2

  3. curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add –

  4. echo “deb https://packages.wazuh.com/3.x/apt/ stable main” | tee -a /etc/apt/sources.list.d/wazuh.list

  5. apt-get update

  6. apt-get install wazuh-manager

  7. systemctl status wazuh-manager

  8. curl -sL https://deb.nodesource.com/setup_8.x | bash –

  9. apt-get install gcc g++ make

  10. apt-get install -y nodejs

  11. curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add –

  12. echo “deb https://dl.yarnpkg.com/debian/ stable main” | sudo tee /etc/apt/sources.list.d/yarn.list

  13. sudo apt-get update && sudo apt-get install yarn

  14. apt-get install nodejs

  15. apt-get install wazuh-api

  16. systemctl status wazuh-api

  17. sed -i “s/^deb/#deb/” /etc/apt/sources.list.d/wazuh.list

  18. apt-get update

  19. curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add –

  20. echo “deb https://artifacts.elastic.co/packages/6.x/apt stable main” | tee /etc/apt/sources.list.d/elastic-6.x.list

  21. apt-get update

  22. apt-get install filebeat

  23. curl -so /etc/filebeat/filebeat.yml https://raw.githubusercontent.com/wazuh/wazuh/v3.9.3/extensions/filebeat/6.x/filebeat.yml

  24. Edit the file /etc/filebeat/filebeat.yml and replace  YOUR_ELASTIC_SERVER_IP with the IP address or the hostname of the Logstash server.

  25. apt search elasticsearch

  26. apt-get install elasticsearch

  27. systemctl daemon-reload

  28. systemctl enable elasticsearch.service

  29. systemctl start elasticsearch.service

  30. curl https://raw.githubusercontent.com/wazuh/wazuh/v3.9.3/extensions/elasticsearch/6.x/wazuh-template.json | curl -X PUT “http://localhost:9200/_template/wazuh&#8221; -H ‘Content-Type: application/json’ -d @-

  31. curl -X PUT “http://localhost:9200/*/_settings?pretty&#8221; -H ‘Content-Type: application/json’ -d’
    “settings”: {
    “number_of_replicas” : 0
    }

  32. sed -i ‘s/#bootstrap.memory_lock: true/bootstrap.memory_lock: true/’ /etc/elasticsearch/elasticsearch.yml

  33. sed -i ‘s/^-Xms.*/-Xms12g/;s/^-Xmx.*/-Xmx12g/’ /etc/elasticsearch/jvm.options

  34. mkdir -p /etc/systemd/system/elasticsearch.service.d/

  35. echo -e “[Service]\nLimitMEMLOCK=infinity” > /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf

  36. systemctl daemon-reload

  37. systemctl restart elasticsearch

  38. apt-get install logstash

  39. curl -so /etc/logstash/conf.d/01-wazuh.conf https://raw.githubusercontent.com/wazuh/wazuh/master/extensions/logstash/6.x/01-wazuh-local.conf

  40. systemctl daemon-reload

  41. systemctl enable logstash.service

  42. systemctl start logstash.service

  43. systemctl status filebeat

  44. systemctl start filebeat

  45. apt-get install kibana

  46. export NODE_OPTIONS=”–max-old-space-size=3072″

  47. sudo -u kibana /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.9.3_6.8.1.zip

  48. Kibana will only listen on the loopback interface (localhost) by default. To set up Kibana to listen on all interfaces, edit the file /etc/kibana/kibana.yml uncommenting the setting server.host. Change the value to:
    server.host: “0.0.0.0”

  49. systemctl enable kibana.service

  50. systemctl start kibana.service

  51. cd /var/ossec/api/configuration/auth

  52. Create a username and password for Wazuh API. When prompted, enter the password:
    node htpasswd -c user admin

  53. systemctl restart wazuh-api

Then in the agent machine execute the following commands:

  1. apt-get install curl apt-transport-https lsb-release gnupg2
  2. curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add –
  3. echo “deb https://packages.wazuh.com/3.x/apt/ stable main” | tee /etc/apt/sources.list.d/wazuh.list
  4. apt-get update
  5. You can automate the agent registration and configuration using variables. It is necessary to define at least the variable WAZUH_MANAGER_IP. The agent will use this value to register and it will be the assigned manager for forwarding events.
    WAZUH_MANAGER_IP=“10.0.0.2” apt-get install wazuh-agent
  6. sed -i “s/^deb/#deb/” /etc/apt/sources.list.d/wazuh.list
  7. apt-get update

In this section, we’ll register the Wazuh API (installed on the Wazuh server) into the Wazuh App in Kibana:

  1. Open a web browser and go to the Elastic Stack server’s IP address on port 5601 (default Kibana port). Then, from the left menu, go to the Wazuh App.

    ../../_images/kibana_app.png

  1. Click on Add new API.
    ../../_images/connect_api.png


  2. Fill Username and Password with appropriate credentials you created in previous step. Enter http://MANAGER_IP for the URL, where MANAGER_IP is the real IP address of the Wazuh qserver. Enter “55000” for the Port.

    ../../_images/fields_api.png
  3. Click on Save.

    ../../_images/app_running.png

Configure prometheus with kubernetes

Quick start

To quickly start all the things just do this:

kubectl apply \
  --filename https://raw.githubusercontent.com/giantswarm/kubernetes-prometheus/master/manifests-all.yaml

This will create the namespace monitoring and bring up all components in there.

To shut down all components again you can just delete that namespace:

kubectl delete namespace monitoring

Default Dashboards

If you want to re-import the default dashboards from this setup run this job:

kubectl apply --filename ./manifests/grafana/grafana-import-dashboards-job.yaml

In case the job already exists from an earlier run, delete it before:

kubectl --namespace monitoring delete job grafana-import-dashboards

More Dashboards

See grafana.net for some example dashboards and plugins.

  • Configure Prometheus data source for Grafana.
    Grafana UI / Data Sources / Add data source

    • Nameprometheus
    • TypePrometheus
    • Urlhttp://prometheus:9090
    • Add
  • Import Prometheus Stats:
    Grafana UI / Dashboards / Import

    • Grafana.net Dashboardhttps://grafana.net/dashboards/2
    • Load
    • Prometheusprometheus
    • Save & Open
  • Import Kubernetes cluster monitoring:
    Grafana UI / Dashboards / Import

    • Grafana.net Dashboardhttps://grafana.net/dashboards/162
    • Load
    • Prometheusprometheus
    • Save & Open

To add a new graph Grafana UI -> Dashboards -> New

Select Graph -> Click on “Panel title” -> Edit. Then In the metrics section in the query box add the following: sum by (status) (irate(kubelet_docker_operations[5m]))

Select source as Prometheus.

You will see graph lines appearing. Thus this way you can add graphs in grafana.

What is Graphite?

What Graphite is and is not

Graphite does two things:

  1. Store numeric time-series data
  2. Render graphs of this data on demand

What Graphite does not do is collect data for you, however there are some tools out there that know how to send data to graphite. Even though it often requires a little code, sending data to Graphite is very simple.

About the project

Graphite is an enterprise-scale monitoring tool that runs well on cheap hardware. It was originally designed and written by Chris Davis at Orbitz in 2006 as side project that ultimately grew to be a foundational monitoring tool. In 2008, Orbitz allowed Graphite to be released under the open source Apache 2.0 license. Since then Chris has continued to work on Graphite and has deployed it at other companies including Sears, where it serves as a pillar of the e-commerce monitoring system. Today many large companies use it.

The architecture in a nutshell

Graphite consists of 3 software components:

  1. carbon – a Twisted daemon that listens for time-series data
  2. whisper – a simple database library for storing time-series data (similar in design to RRD)
  3. graphite webapp – A Django webapp that renders graphs on-demand using Cairo

Feeding in your data is pretty easy, typically most of the effort is in collecting the data to begin with. As you send datapoints to Carbon, they become immediately available for graphing in the webapp. The webapp offers several ways to create and display graphs including a simple URL APIfor rendering that makes it easy to embed graphs in other webpages.

_images/overview.png

What is Graphite?

Graphite is a highly scalable real-time graphing system. As a user, you write an application that collects numeric time-series data that you are interested in graphing, and send it to Graphite’s processing backend, carbon, which stores the data in Graphite’s specialized database. The data can then be visualized through graphite’s web interfaces.

The Carbon Daemons

When we talk about “Carbon” we mean one or more of various daemons that make up the storage backend of a Graphite installation. In simple installations, there is typically only one daemon, carbon-cache.py. As an installation grows, the carbon-relay.py and carbon-aggregator.py daemons can be introduced to distribute metrics load and perform custom aggregations, respectively.

All of the carbon daemons listen for time-series data and can accept it over a common set of protocols. However, they differ in what they do with the data once they receive it. This document gives a brief overview of what each daemon does and how you can use them to build a more sophisticated storage backend.

carbon-cache.py

carbon-cache.py accepts metrics over various protocols and writes them to disk as efficiently as possible. This requires caching metric values in RAM as they are received, and flushing them to disk on an interval using the underlying whisper library. It also provides a query service for in-memory metric datapoints, used by the Graphite webapp to retrieve “hot data”.

carbon-cache.py requires some basic configuration files to run:

carbon.conf
The [cache] section tells carbon-cache.py what ports (2003/2004/7002), protocols (newline delimited, pickle) and transports (TCP/UDP) to listen on.
storage-schemas.conf
Defines a retention policy for incoming metrics based on regex patterns. This policy is passed to whisper when the .wsp file is pre-allocated, and dictates how long data is stored for.

As the number of incoming metrics increases, one carbon-cache.py instance may not be enough to handle the I/O load. To scale out, simply run multiple carbon-cache.py instances (on one or more machines) behind a carbon-aggregator.py or carbon-relay.py.

carbon-relay.py

carbon-relay.py serves two distinct purposes: replication and sharding.

When running with RELAY_METHOD = rules, a carbon-relay.py instance can run in place of a carbon-cache.py server and relay all incoming metrics to multiple backend carbon-cache.py‘s running on different ports or hosts.

In RELAY_METHOD = consistent-hashing mode, a DESTINATIONS setting defines a sharding strategy across multiple carbon-cache.py backends. The same consistent hashing list can be provided to the graphite webapp via CARBONLINK_HOSTS to spread reads across the multiple backends.

carbon-relay.py is configured via:

carbon.conf
The [relay] section defines listener host/ports and a RELAY_METHOD
relay-rules.conf
With RELAY_METHOD = rules set, pattern/servers tuples in this file define which metrics matching certain regex rules are forwarded to which hosts.

carbon-aggregator.py

carbon-aggregator.py can be run in front of carbon-cache.py to buffer metrics over time before reporting them into whisper. This is useful when granular reporting is not required, and can help reduce I/O load and whisper file sizes due to lower retention policies.

carbon-aggregator.py is configured via:

carbon.conf
The [aggregator] section defines listener and destination host/ports.
aggregation-rules.conf
Defines a time interval (in seconds) and aggregation function (sum or average) for incoming metrics matching a certain pattern. At the end of each interval, the values received are aggregated and published to carbon-cache.py as a single metric.

carbon-aggregator-cache.py

carbon-aggregator-cache.py combines both carbon-aggregator.py and carbon-cache.py. This is useful to reduce the resource and administration overhead of running both daemons.

carbon-aggregator-cache.py is configured via:

carbon.conf
The [aggregator-cache] section defines listener and destination host/ports.
relay-rules.conf
See carbon-relay.py section.
aggregation-rules.conf
See carbon-aggregator.py section.

The Whisper Database

Whisper is a fixed-size database, similar in design and purpose to RRD (round-robin-database). It provides fast, reliable storage of numeric data over time. Whisper allows for higher resolution (seconds per point) of recent data to degrade into lower resolutions for long-term retention of historical data.

Data Points

Data points in Whisper are stored on-disk as big-endian double-precision floats. Each value is paired with a timestamp in seconds since the UNIX Epoch (01-01-1970). The data value is parsed by the Python float() function and as such behaves in the same way for special strings such as 'inf'. Maximum and minimum values are determined by the Python interpreter’s allowable range for float values which can be found by executing:

python -c 'import sys; print sys.float_info'

Archives: Retention and Precision

Whisper databases contain one or more archives, each with a specific data resolution and retention (defined in number of points or max timestamp age). Archives are ordered from the highest-resolution and shortest retention archive to the lowest-resolution and longest retention period archive.

To support accurate aggregation from higher to lower resolution archives, the precision of a longer retention archive must be divisible by precision of next lower retention archive. For example, an archive with 1 data point every 60 seconds can have a lower-resolution archive following it with a resolution of 1 data point every 300 seconds because 60 cleanly divides 300. In contrast, a 180 second precision (3 minutes) could not be followed by a 600 second precision (10 minutes) because the ratio of points to be propagated from the first archive to the next would be 3 1/3 and Whisper will not do partial point interpolation.

The total retention time of the database is determined by the archive with the highest retention as the time period covered by each archive is overlapping (see Multi-Archive Storage and Retrieval Behavior). That is, a pair of archives with retentions of 1 month and 1 year will not provide 13 months of data storage as may be guessed. Instead, it will provide 1 year of storage – the length of its longest archive.

Rollup Aggregation

Whisper databases with more than a single archive need a strategy to collapse multiple data points for when the data rolls up a lower precision archive. By default, an average function is used. Available aggregation methods are:

  • average
  • sum
  • last
  • max
  • min

Multi-Archive Storage and Retrieval Behavior

When Whisper writes to a database with multiple archives, the incoming data point is written to all archives at once. The data point will be written to the highest resolution archive as-is, and will be aggregated by the configured aggregation method (see Rollup Aggregation) and placed into each of the higher-retention archives. If you are in need for aggregation of the highest resolution points, please consider using carbon-aggregator for that purpose.

When data is retrieved (scoped by a time range), the first archive which can satisfy the entire time period is used. If the time period overlaps an archive boundary, the lower-resolution archive will be used. This allows for a simpler behavior while retrieving data as the data’s resolution is consistent through an entire returned series.

Configuring Graphite on Centos 7

Clone the source code:
git clone https://github.com/graphite-project/graphite-web.git
cd graphite-web
git checkout 0.9.x
cd ..
git clone https://github.com/graphite-project/carbon.git
cd carbon
git checkout 0.9.x
cd ..
git clone https://github.com/graphite-project/whisper.git
cd whisper
git checkout 0.9.x
cd ..

Configure whisper:
pushd whisper
sudo python setup.py install
popd

Configure carbon:
pushd carbon
sudo python setup.py install
popd
pushd /opt/graphite/conf/
sudo cp carbon.conf.example carbon.conf
sudo cp storage-schemas.conf.example storage-schemas.conf
popd

storage-schemas.conf has information about schema definitions for Whisper files. We can define the data retention time under this file. By default it retains everything for one day. Once graphite is configured changing this file wont change whisper’s internal metrics. You can use whisper-resize.py for that.

Configure Graphite
pushd graphite-web
python check-dependencies.py
Install the unmet dependencies:
sudo apt-get install python-cairo python-django python-django-tagging python-memcache python-ldap
python-txamqp
popd

Configure the webapp:
pushd graphite-web
sudo python setup.py install
popd

Configure Graphite webapp using Apache:
Install apache and mod_wsgi:
sudo apt-get install apache2 libapache2-mod-wsgi

Configure graphite virtual host:
sudo cp graphite-web/examples/example-graphite-vhost.conf
/etc/apache2/sites-available/graphite-vhost.conf
sudo ln -s /etc/apache2/sites-available/graphite-vhost.conf /etc/apache2/sites-enabled/graphite-vhost.conf
sudo unlink /etc/apache2/sites-enabled/000-default.conf

Edit /etc/apache2/sites-available/graphite-vhost.conf and add
WSGISocketPrefix /var/run/apache2/wsgi
Edit /etc/apache2/sites-available/graphite-vhost.conf and add
<Directory /opt/graphite/conf/>
Options FollowSymlinks
AllowOverride none
Require all granted
</Directory>
Reload apache configurations:
sudo service apache2 reload

Sync sqlite database for graphite-web:
cp /opt/graphite/webapp/graphite/local_settings.py.example
/opt/graphite/webapp/graphite/local_settings.py
cd /opt/graphite/webapp/graphite
You can add turn on debugging for graphite-web by adding following to local_settings.py:
DEBUG=True
sudo python manage.py syncdb
while doing db-sync you will be asked to create superuser for graphite. Create a superuser and password for it.
Change owner of graphite storage directory to a user through which apache is being run:
sudo chown -R www-data:www-data /opt/graphite/storage/

Configure nginx instead of apache:

Install necessary packages:
sudo apt-get install nginx php5-fpm uwsgi-plugin-python uwsgi
Configure nginx and uwsgi:
cd /opt/graphite/conf/
sudo cp graphite.wsgi.example wsgi.py
Create a file /etc/nginx/sites-available/graphite-vhost.conf and add following to it:
server {
listen 8080;
server_name graphite;
root /opt/graphite/webapp;
error_log /opt/graphite/storage/log/webapp/error.log error;
access_log /opt/graphite/storage/log/webapp/access.log;

location / {
include uwsgi_params;
uwsgi_pass 127.0.0.1:3031;
}

}
Enable the nginx server
sudo ln -s /etc/nginx/sites-available/graphite-vhost.conf /etc/nginx/sites-enabled/graphite-vhost.conf
Create a file /etc/uwsgi/apps-available/graphite.ini and add following to it:
[uwsgi]
processes = 2
socket = 127.0.0.1:3031
gid = www-data
uid = www-data
chdir = /opt/graphite/conf
module = wsgi:application
sudo ln -s /etc/uwsgi/apps-available/graphite.ini /etc/uwsgi/apps-enabled/graphite.ini

Restart services:
sudo /etc/init.d/uwsgi restart
sudo /etc/init.d/nginx restart

Start Carbon (the data aggregator):
cd /opt/graphite/
./bin/carbon-cache.py start

 

Access the graphite home page:
Graphite homepage is available at http://<ip-of-graphite-host&gt;

Connect graphite to DSP-Core:

For connecting your DSP-Core with graphite you need to install carbon agnet over your
DSP-Core machine so that carbon agent will send the data to your graphite host which can be
displayed over web-UI. Follow these steps to connect your DSP-Core with graphite.
Install carbon over DSP-Core machine:
git clone https://github.com/graphite-project/carbon.git
cd carbon
git checkout 0.9.x

cd ..
pushd carbon
sudo python setup.py install
popd
pushd /opt/graphite/conf/
sudo cp carbon.conf.example carbon.conf
sudo cp storage-schemas.conf.example storage-schemas.conf

popd
Configure DSP-Core to send data to graphite:
You need to add following to /data1/deploy/dsp/current/dsp-core/conf/bootstrap.json
“carbon-uri”: [“<IP-of-graphite-host>:2003”]

pyDash – A Web Based Linux Performance Monitoring Tool

pydash is a lightweight web-based monitoring tool for Linux written in Python and Django plus Chart.js. It has been tested and can run on the following mainstream Linux distributions: CentOS, Fedora, Ubuntu, Debian, Arch Linux, Raspbian as well as Pidora.

You can use it to keep an eye on your Linux PC/server resources such as CPUs, RAM, network stats, processes including online users and more. The dashboard is developed entirely using Python libraries provided in the main Python distribution, therefore it has a few dependencies; you don’t need to install many packages or libraries to run it.

In this article, we will show you how to install pydash to monitor Linux server performance.

How to Install pyDash in Linux System

1. First install required packages: git and Python pip as follows:

-------------- On Debian/Ubuntu -------------- 
$ sudo apt-get install git python-pip
-------------- On CentOS/RHEL -------------- 
# yum install epel-release
# yum install git python-pip
-------------- On Fedora 22+ --------------
# dnf install git python-pip

2. If you have git and Python pip installed, next, install virtualenv which helps to deal with dependency issues for Python projects, as below:

# pip install virtualenv
OR
$ sudo pip install virtualenv

3. Now using git command, clone the pydash directory into your home directory like so:

# git clone https://github.com/k3oni/pydash.git
# cd pydash

4. Next, create a virtual environment for your project called pydashtest using the virtualenv command below.

$ virtualenv pydashtest #give a name for your virtual environment like pydashtest

Create Virtual Environment

Important: Take note the virtual environment’s bin directory path highlighted in the screenshot above, yours could be different depending on where you cloned the pydash folder.

5. Once you have created the virtual environment (pydashtest), you must activate it before using it as follows.

$ source /home/aaronkilik/pydash/pydashtest/bin/activate

Active Virtual Environment

From the screenshot above, you’ll note that the PS1 prompt changes indicating that your virtual environment has been activated and is ready for use.

6. Now install the pydash project requirements; if you are curious enough, view the contents of requirements.txt using the cat command and the install them using as shown below.

$ cat requirements.txt
$ pip install -r requirements.txt

7. Now move into the pydash directory containing settings.py or simple run the command below to open this file to change the SECRET_KEY to a custom value.

$ vi pydash/settings.py

Set Secret Key

Save the file and exit.

8. Afterward, run the django command below to create the project database and install Django’s auth system and create a project super user.

$ python manage.py syncdb

Answer the questions below according to your scenario:

Would you like to create one now? (yes/no): yes
Username (leave blank to use 'root'): admin
Email address: aaronkilik@gmail.com
Password: ###########
Password (again): ############

Create Project Database

9. At this point, all should be set, now run the following command to start the Django development server.

$ python manage.py runserver

10. Next, open your web browser and type the URL: http://127.0.0.1:8000/ to get the web dashboard login interface. Enter the super user name and password you created while creating the database and installing Django’s auth system in step 8 and click Sign In.

pyDash Login Interface

11. Once you login into pydash main interface, you will get a section for monitoring general system info, CPU, memory and disk usage together with system load average.

Simply scroll down to view more sections.

pyDash Server Performance Overview

12. Next, screenshot of the pydash showing a section for keeping track of interfaces, IP addresses, Internet traffic, disk read/writes, online users and netstats.

pyDash Network Overview

13. Next is a screenshot of the pydash main interface showing a section to keep an eye on active processes on the system.

pyDash Active Linux Processes