wazuh – A Passionate Techie

Wazuh: Issues encountered and solutions

logstash service does not find config files in /etc/logstash/conf.d
I installed logstash via centos rpm and placed a valid logstash configuration file into /etc/logstash/conf.d. Starting the service it fails with following error message:

{:timestamp=>"2015-10-21T08:11:06.939000+0000", :message=>"Error: No config files found: /etc/logstash/conf.d/pipe.conf\nCan you make sure this path is a logstash config file?", :file=>"logstash/agent.rb", :line=>"159", :method=>"execute"}
{:timestamp=>"2015-10-21T08:11:06.948000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system.", :file=>"logstash/agent.rb", :line=>"161", :method=>"execute"}

there is definitely a file in this location read/write to anyone

[root@logserv logstash]# ls -al /etc/logstash/conf.d/
total 12
drwxr-xr-x. 2 root root 4096 Oct 21 08:00 .
drwx------. 4 root root 4096 Oct 21 07:54 ..
-rwxrwxrwx. 1 root root  234 Oct 21 07:56 pipe.conf

and its content is valid

[root@logserv logstash]# bin/logstash -f /etc/logstash/conf.d/pipe.conf --configtest
Configuration OK

Solution:

chown -R logstash:root /etc/logstash/conf.d
chmod 0750 /etc/logstash/conf.d
chmod 0640 /etc/logstash/conf.d/*

If in the Wazuh UI you see data in wazuh-alerts but not in any of the wazuh dashboards, check if the data is getting pushed to Elasticsearch first:

curl localhost:9200/_cat/indices

The output should look like below:

green open wazuh-alerts-3.x-2019.07.19 GIPOTyJuSxSZgVtsdkouxg 3 0 131 0 424.7kb 424.7kb
green open .kibana_task_manager cCFAzTqIQ6GuhVtJsfuUrQ 1 0 2 0 29.5kb 29.5kb
yellow open .wazuh tgqyhP1rQHqRk4bnfvjivg 1 1 1 0 11kb 11kb
green open wazuh-alerts-3.x-2019.07.20 vbSs-0TRRRKihI3vo67C0w 3 0 10 0 79.7kb 79.7kb
green open wazuh-alerts-3.x-2019.07.21 GYbynBOLTsedyuxIVfSmig 3 0 9 0 80.7kb 80.7kb
green open .kibana_1 24p2awqCTFafufPXuTkM_A 1 0 6 2 110.6kb 110.6kb
green open wazuh-monitoring-3.x-2019.07.18 GtPTclhVS6CveIoTB9s88w 2 0 192 0 174.7kb 174.7kb
green open wazuh-monitoring-3.x-2019.07.20 skX7aKIMTNa20VKvZdG-gg 2 0 192 0 210.9kb 210.9kb
green open wazuh-monitoring-3.x-2019.07.17 fERZ9LMeQheBUDo4CFZgbw 2 0 98 0 215.1kb 215.1kb
green open wazuh-monitoring-3.x-2019.07.21 fDT71M7bSNawPplieIEXRg 2 0 46 0 208.6kb 208.6kb
yellow open .wazuh-version 2TPSH17YQ4e_n6NiWDhQqQ 1 1 1 0 5.2kb 5.2kb
green open wazuh-monitoring-3.x-2019.07.19 8RDIhk0EQIOxNIYOOh6VXA 2 0 198 0 140.1kb 140.1kb

Check if wazuh-alerts-3.x-* index is present with current date. If yes, then check if data is present in the index:

curl localhost:9200/<INDEX_NAME>/_search?pretty=true&size=1000
Example:
localhost:9200/wazuh-alerts-3.x-2019.07.19/_search?pretty=true&size=1000

This should return first 1000 entries.
Check if latest entries are present. Also check if manager.name matches the hostname of the manager. If not, then change the hostname of the manager by executing command:

hostname <HOSTNAME>
Example:
hostname abc.example.com

The alerts in the Elasticsearch index will start coming in with the manager.hostname as abc.example.com

Get details of a Wazuh agent:

curl -u foo:bar localhost:55000/agents/<AGENT_ID>

foo:bar is the credentials for wazuh api.

Get details of nodes in Wazuh cluster:

curl -u foo:bar localhost:55000/cluster/nodes?pretty

Rules not getting reflected aftere change in /var/ossec/ruleset/rules
Restarting the wazuh-manger should reload the rules:

systemctl restart wazuh-manager

No events in wazuh-monitoring index:
Check if index wazuh-monitoring-3.x-* is present with today’s date:

curl elastic:9200/_cat/indices/wazuh-monitoring*

Check if there is any error in wazuhapp for wazuh-monitoring:

cat /usr/share/kibana/optimize/wazuh-logs/wazuhapp-plain.log | grep monitoring

Execute:

curl -XGET "http://elastic:9200/_cat/templates/wazuh-agent"

If you get something like:
wazuh-agent [wazuh-monitoring*, wazuh-monitoring-3.x-*] 0
You probably have a template issue. Execute the following to resolve it:

Stop Kibana: systemctl stop kibana

Delete the monitoring template, curl -XDELETE elastic:9200/_template/wazuh-agent

Restart Kibana: systemctl restart kibana, it should insert the monitoring template and the Kibana UI should start working shortly

Change wazuh app to debug mode:
Edit /usr/share/kibana/plugins/wazuh/config.yml
Replace #logs.level: info with logs.level: debug, then restart Kibana service (systemctl restart kibana)
Wazuh UI Error: “Saved field parameter is now invalid” OR “Error in Visualisation: field is a required parameter”
You will require a cleanup. Execute the following commands:
Delete wazuh-alerts-3.x-* index with today’s date:
```
curl -XDELETE localhost:9200/wazuh-alerts-3.x-2019.07.18
systemctl restart wazuh-manager
curl -XDELETE localhost:9200/.kiban*
systemctl restart kibana
rm -f /var/ossec/queue/db/0*
systemctl restart wazuh-manager
```

Generate SCA alerts:

rm -f /var/ossec/queue/db/0*
systemctl restart wazuh-manager

Error in visualisation: Expected numeric type on field on data.sca.score got numeric:
You most probably have wrong template. Just install the template according to your wazuh version from their github repo. To install latest (3.9.3) execute the following:
```
curl https://raw.githubusercontent.com/wazuh/wazuh/v3.9.3/extensions/elasticsearch/6.x/wazuh-template.json | curl -X PUT "http://localhost:9200/_template/wazuh" -H 'Content-Type: application/json' -d @-
```

java.lang.IllegalArgumentException: Rejecting mapping update to [wazuh-alerts-3.x] as the final mapping would have more than 1 type:
This would most possibly mean you have wrong template. Install the latest one with above step.
If already done you might have an issue with your logstash configuration:

curl -so /etc/logstash/conf.d/01-wazuh.conf https://raw.githubusercontent.com/wazuh/wazuh/v3.9.3/extensions/logstash/7.x/01-wazuh-remote.conf

Delete wazuh-alerts-3.x-* index with today’s date:

curl -XDELETE localhost:9200/wazuh-alerts-3.x-2019.07.18systemctl restart wazuh-manager
curl -XDELETE localhost:9200/.kiban*
systemctl restart kibana
rm -f /var/ossec/queue/db/0*
systemctl restart wazuh-manager

Version mismatch:
If you get the above error on Wazuh UI, execute the following commands:

service kibana stop
curl -XDELETE localhost:9200/.wazuh
curl -XDELETE localhost:9200/.wazuh_version
service kibana start

Other useful commands:

Check documents in an index:
curl elastic:9200/_cat/indices/wazuh-monitoring*

Check wazuhapp logs:
cat /usr/share/kibana/optimize/wazuh-logs/wazuhapp-plain.log | grep monitoring

Check wazuh config:
cat /usr/share/kibana/plugins/wazuh/config.yml

Get Wazuh id:
curl elastic:9200/.wazuh/_search?pretty -s | grep "_id"

Templates:
curl elastic:9200/_cat/templates/wazuh

Version:
cat /usr/share/kibana/plugins/wazuh/package.json | grep version
cat /etc/ossec-init.conf | grep -i version
curl -u foo:bar localhost:55000/version

Monitoring settings:
cat /usr/share/kibana/plugins/wazuh/config.yml | grep monitoring

Search monitoring index:
curl elastic:9200/wazuh-monitoring*/_search

List of agents:
curl -u foo:bar "localhost:55000/agents?q=id!=000"

Index settings:
curl  "elastic:9200/wazuh-monitoring-3.x-2019.07.17/_settings"

Get details of template:
curl -XGET "http://elastic:9200/_cat/templates/wazuh-agent"

Check if filebeat is configured correctly:
filebeat test output

Install and configure Wazuh with ELK 6.x

Wazuh helps you to gain deeper security visibility into your infrastructure by monitoring hosts at an operating system and application level. This solution, based on lightweight multi-platform agents, provides the following capabilities:

File integrity monitoring

Wazuh monitors the file system, identifying changes in content, permissions, ownership, and attributes of files that you need to keep an eye on.

Intrusion and anomaly detection

Agents scan the system looking for malware, rootkits or suspicious anomalies. They can detect hidden files, cloaked processes or unregistered network listeners, as well as inconsistencies in system call responses.

Automated log analysis

Wazuh agents read operating system and application logs, and securely forward them to a central manager for rule-based analysis and storage. The Wazuh rules help make you aware of application or system errors, misconfigurations, attempted and/or successful malicious activities, policy violations and a variety of other security and operational issues.

Policy and compliance monitoring

Wazuh monitors configuration files to ensure they are compliant with your security policies, standards and/or hardening guides. Agents perform periodic scans to detect applications that are known to be vulnerable, unpatched, or insecurely configured.

This diverse set of capabilities is provided by integrating OSSEC, OpenSCAP and Elastic Stack into a unified solution and simplifying their configuration and management.

Execute the following commands to install and configure Wazuh:

apt-get update
apt-get install curl apt-transport-https lsb-release gnupg2
curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add –
echo “deb https://packages.wazuh.com/3.x/apt/ stable main” | tee -a /etc/apt/sources.list.d/wazuh.list
apt-get update
apt-get install wazuh-manager
systemctl status wazuh-manager
curl -sL https://deb.nodesource.com/setup_8.x | bash –
apt-get install gcc g++ make
apt-get install -y nodejs
curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add –
echo “deb https://dl.yarnpkg.com/debian/ stable main” | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install yarn
apt-get install nodejs
apt-get install wazuh-api
systemctl status wazuh-api
sed -i “s/^deb/#deb/” /etc/apt/sources.list.d/wazuh.list
apt-get update
curl -s https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add –
echo “deb https://artifacts.elastic.co/packages/6.x/apt stable main” | tee /etc/apt/sources.list.d/elastic-6.x.list
apt-get update
apt-get install filebeat
curl -so /etc/filebeat/filebeat.yml https://raw.githubusercontent.com/wazuh/wazuh/v3.9.3/extensions/filebeat/6.x/filebeat.yml
Edit the file /etc/filebeat/filebeat.yml and replace YOUR_ELASTIC_SERVER_IP with the IP address or the hostname of the Logstash server.
apt search elasticsearch
apt-get install elasticsearch
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
curl https://raw.githubusercontent.com/wazuh/wazuh/v3.9.3/extensions/elasticsearch/6.x/wazuh-template.json | curl -X PUT “http://localhost:9200/_template/wazuh” -H ‘Content-Type: application/json’ -d @-
curl -X PUT “http://localhost:9200/*/_settings?pretty” -H ‘Content-Type: application/json’ -d’
“settings”: {
“number_of_replicas” : 0
}
‘
sed -i ‘s/#bootstrap.memory_lock: true/bootstrap.memory_lock: true/’ /etc/elasticsearch/elasticsearch.yml
sed -i ‘s/^-Xms.*/-Xms12g/;s/^-Xmx.*/-Xmx12g/’ /etc/elasticsearch/jvm.options
mkdir -p /etc/systemd/system/elasticsearch.service.d/
echo -e “[Service]\nLimitMEMLOCK=infinity” > /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf
systemctl daemon-reload
systemctl restart elasticsearch
apt-get install logstash
curl -so /etc/logstash/conf.d/01-wazuh.conf https://raw.githubusercontent.com/wazuh/wazuh/master/extensions/logstash/6.x/01-wazuh-local.conf
systemctl daemon-reload
systemctl enable logstash.service
systemctl start logstash.service
systemctl status filebeat
systemctl start filebeat
apt-get install kibana
export NODE_OPTIONS=”–max-old-space-size=3072″
sudo -u kibana /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.9.3_6.8.1.zip
Kibana will only listen on the loopback interface (localhost) by default. To set up Kibana to listen on all interfaces, edit the file /etc/kibana/kibana.yml uncommenting the setting server.host. Change the value to:
server.host: “0.0.0.0”
systemctl enable kibana.service
systemctl start kibana.service
cd /var/ossec/api/configuration/auth
Create a username and password for Wazuh API. When prompted, enter the password:
node htpasswd -c user admin
systemctl restart wazuh-api

Then in the agent machine execute the following commands:

apt-get install curl apt-transport-https lsb-release gnupg2

curl -s https://packages.wazuh.com/key/GPG-KEY-WAZUH | apt-key add –

echo “deb https://packages.wazuh.com/3.x/apt/ stable main” | tee /etc/apt/sources.list.d/wazuh.list

apt-get update

You can automate the agent registration and configuration using variables. It is necessary to define at least the variable WAZUH_MANAGER_IP. The agent will use this value to register and it will be the assigned manager for forwarding events.
WAZUH_MANAGER_IP=“10.0.0.2” apt-get install wazuh-agent

sed -i “s/^deb/#deb/” /etc/apt/sources.list.d/wazuh.list

apt-get update

In this section, we’ll register the Wazuh API (installed on the Wazuh server) into the Wazuh App in Kibana:

Open a web browser and go to the Elastic Stack server’s IP address on port 5601 (default Kibana port). Then, from the left menu, go to the Wazuh App.

Click on Add new API.
Fill Username and Password with appropriate credentials you created in previous step. Enter http://MANAGER_IP for the URL, where MANAGER_IP is the real IP address of the Wazuh qserver. Enter “55000” for the Port.
Click on Save.

Track down vulnerable applications

Of the many software packages installed on your Red Hat, CentOS, and/or Ubuntu systems, which ones have known vulnerabilities that might impact your security posture? Wazuh helps you answer this question with the syscollector and vulnerability-detector modules. On each agent, syscollector can scan the system for the presence and version of all software packages. This information is submitted to the Wazuh manager where it is stored in an agent-specific database for later assessment. On the Wazuh manager, vulnerability-detector maintains a fresh copy of the desired CVE sources of vulnerability data, and periodically compares agent packages with the relevant CVE database and generates alerts on matches.

In this lab, we will configure syscollector to run on wazuh-server and on both of the Linux agents. We will also configurevulnerability-detector on wazuh-server to periodically scan the collected inventory data for known vulnerable packages. We will observe relevant log messages and vulnerability alerts in Kibana including a dashboard dedicated to this. We will also interact with the Wazuh API to more deeply mine the inventory data, and even take a look at the databases where it is stored.

Configure `syscollector` for the Linux agents

In /var/ossec/etc/shared/linux/agent.conf on wazuh-server, just before the open-scap wodle configuration section, insert the following so each Linux agent will scan itself.

<wodle name="syscollector">
  <disabled>no</disabled>
  <interval>1d</interval>
  <scan_on_start>yes</scan_on_start>
  <hardware>yes</hardware>
  <os>yes</os>
  <packages>yes</packages>
</wodle>

Run verify-agent-conf to confirm no errors were introduced into agent.conf.

Configure `vulnerability-detector` and `syscollector` on wazuh-server

In ossec.conf on wazuh-server, just before the open-scap wodle configuration section, insert the following so that it will inventory its own software plus scan all collected software inventories against published CVEs, alerting where there are matches:

<wodle name="vulnerability-detector">
  <disabled>no</disabled>
  <interval>5m</interval>
  <run_on_start>yes</run_on_start>
  <feed name="ubuntu-18">
    <disabled>no</disabled>
    <update_interval>1h</update_interval>
  </feed>
</wodle>

Restart the Wazuh manager. This will also cause the agents to restart as they pick up their new configuration:

For Systemd:

systemctl restart wazuh-manager

Look at the logs

The vulnerability-detector module generates logs on the manager, and syscollector does as well on the manager and agents.

Try grep syscollector: /var/ossec/logs/ossec.log on the manager and on an agent:

2018/02/23 00:55:33 wazuh-modulesd:syscollector: INFO: Module started.
2018/02/23 00:55:34 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2018/02/23 00:55:35 wazuh-modulesd:syscollector: INFO: Evaluation finished.

and try grep vulnerability-detector: /var/ossec/logs/ossec.log on the manager

2018/02/23 00:55:33 wazuh-modulesd:vulnerability-detector: INFO: (5461): Starting Red Hat Enterprise Linux 7 DB update...
2018/02/23 00:55:33 wazuh-modulesd:vulnerability-detector: INFO: (5452): Starting vulnerability scanning.
2018/02/23 00:55:33 wazuh-modulesd:vulnerability-detector: INFO: (5453): Vulnerability scanning finished.

See the alerts in Kibana

Search Kibana for location:"vulnerability-detector" AND data.vulnerability.severity:"High", selecting some of the more helpful fields for viewing like below:

Expand one of the records to see all the information available:

Look deeper with the Wazuh API:

Up to now we have only seen the Wazuh API enable the Wazuh Kibana App to interface directly with the Wazuh manager. However, you can also access the API directly from your own scripts or from the command line with curl. This is especially helpful here as full software inventory data is not stored in Elasticsearch or visible in Kibana – only the CVE match alerts are. The actual inventory data is kept in agent-specific databases on the Wazuh manager. To see that, plus other information collected by syscollector, you can mine the Wazuh API. Not only are software packages inventoried, but basic hardware and operating system data is also tracked.

Run agent_control -l on wazuh-server to list your agents as you will need to query the API by agent id number:

Wazuh agent_control. List of available agents:
  ID: 000, Name: wazuh-server (server), IP: localhost, Active/Local
  ID: 001, Name: linux-agent, IP: any, Active
  ID: 002, Name: elastic-server, IP: any, Active
  ID: 003, Name: windows-agent, IP: any, Active

On wazuh-server, query the Wazuh API for scanned hardware data about agent 002.

# curl -u wazuhapiuser:wazuhlab -k -X GET "https://localhost:55000/syscollector/002/hardware?pretty"

The results should look like this:

{
  "error": 0,
  "data": {
      "board_serial": "unknown",
      "ram": {
        "total": 8009024,
        "free": 156764
      },
      "cpu": {
        "cores": 2,
        "mhz": 2400.188,
        "name": "Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz"
      },
      "scan": {
        "id": 1794797325,
        "time": "2018/02/18 02:05:31"
      }
  }
}

Next, query the Wazuh API for scanned OS data about agent 002.

# curl -u wazuhapiuser:wazuhlab -k -X GET "https://localhost:55000/syscollector/002/os?pretty"

The results should look like this:

{
  "error": 0,
  "data": {
      "sysname": "Linux",
      "version": "#1 SMP Thu Jan 25 20:13:58 UTC 2018",
      "architecture": "x86_64",
      "scan": {
        "id": 1524588903,
        "time": "2018/02/23 01:12:21"
      },
      "release": "3.10.0-693.17.1.el7.x86_64",
      "hostname": "elastic-server",
      "os": {
        "version": "7 (Core)",
        "name": "CentOS Linux"
      }
  }
}

You can also query the software inventory data in many ways. Let’s list the versions of wget on all of our Linux systems:

# curl -u wazuhapiuser:wazuhlab -k -X GET "https://localhost:55000/syscollector/packages?pretty&search=wget"

mapper_parsing_exception from logstash

I am receiving the following logstash errors after installing X-Pack in logstash:

[2018-06-14T13:24:40,458][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wazuh-alerts-3.x-2018.06.14", :_type=>"wazuh", :_routing=>nil}, #<LogStash::Event:0x31b18a15>], :response=>{"index"=>{"_index"=>"wazuh-alerts-3.x-2018.06.14", "_type"=>"wazuh", "_id"=>"kZ54_mMB86eT4RWzM1CD", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [host]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:114"}}}}}

The problem is Filebeat 6.3.0 is adding a new field by itself named “host” but we have the field “hostname” so “host” is not needed.

Regarding to Elasticsearch we don’t have that field in our template but we don’t want that field so the possible solution is to modify

the Logstash configuration (mutate section):

mutate { remove_field => [ "timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type","@src_ip", "host" ]}

	gamejudilebaran.word… on Chef: Roles and Environme…
	WARN: Waiting for se… on OSSEC start problem due to…
	aratik711 on Ansible issues
	aratik711 on Chef: Roles and Environme…
	situs judi on Chef: Roles and Environme…

Rate this:

Share this:

Rate this:

Share this:

Configure syscollector for the Linux agents

Configure vulnerability-detector and syscollector on wazuh-server

Look at the logs

See the alerts in Kibana

Look deeper with the Wazuh API:

Rate this:

Share this:

Rate this:

Share this:

Configure `syscollector` for the Linux agents

Configure `vulnerability-detector` and `syscollector` on wazuh-server