Tag: java.lang.IllegalArgumentException

Wazuh: Issues encountered and solutions

 

  1. logstash service does not find config files in /etc/logstash/conf.d
    I installed logstash via centos rpm and placed a valid logstash configuration file into /etc/logstash/conf.d. Starting the service it fails with following error message:

    {:timestamp=>"2015-10-21T08:11:06.939000+0000", :message=>"Error: No config files found: /etc/logstash/conf.d/pipe.conf\nCan you make sure this path is a logstash config file?", :file=>"logstash/agent.rb", :line=>"159", :method=>"execute"}
{:timestamp=>"2015-10-21T08:11:06.948000+0000", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system.", :file=>"logstash/agent.rb", :line=>"161", :method=>"execute"}

    there is definitely a file in this location read/write to anyone

    [root@logserv logstash]# ls -al /etc/logstash/conf.d/
total 12
drwxr-xr-x. 2 root root 4096 Oct 21 08:00 .
drwx------. 4 root root 4096 Oct 21 07:54 ..
-rwxrwxrwx. 1 root root  234 Oct 21 07:56 pipe.conf

    and its content is valid

    [root@logserv logstash]# bin/logstash -f /etc/logstash/conf.d/pipe.conf --configtest
Configuration OK

    Solution:

    chown -R logstash:root /etc/logstash/conf.d
chmod 0750 /etc/logstash/conf.d
chmod 0640 /etc/logstash/conf.d/*
  2.  If in the Wazuh UI you see data in wazuh-alerts but not in any of the wazuh dashboards, check if the data is getting pushed to Elasticsearch first:

    curl localhost:9200/_cat/indices

    The output should look like below:

    green open wazuh-alerts-3.x-2019.07.19 GIPOTyJuSxSZgVtsdkouxg 3 0 131 0 424.7kb 424.7kb
green open .kibana_task_manager cCFAzTqIQ6GuhVtJsfuUrQ 1 0 2 0 29.5kb 29.5kb
yellow open .wazuh tgqyhP1rQHqRk4bnfvjivg 1 1 1 0 11kb 11kb
green open wazuh-alerts-3.x-2019.07.20 vbSs-0TRRRKihI3vo67C0w 3 0 10 0 79.7kb 79.7kb
green open wazuh-alerts-3.x-2019.07.21 GYbynBOLTsedyuxIVfSmig 3 0 9 0 80.7kb 80.7kb
green open .kibana_1 24p2awqCTFafufPXuTkM_A 1 0 6 2 110.6kb 110.6kb
green open wazuh-monitoring-3.x-2019.07.18 GtPTclhVS6CveIoTB9s88w 2 0 192 0 174.7kb 174.7kb
green open wazuh-monitoring-3.x-2019.07.20 skX7aKIMTNa20VKvZdG-gg 2 0 192 0 210.9kb 210.9kb
green open wazuh-monitoring-3.x-2019.07.17 fERZ9LMeQheBUDo4CFZgbw 2 0 98 0 215.1kb 215.1kb
green open wazuh-monitoring-3.x-2019.07.21 fDT71M7bSNawPplieIEXRg 2 0 46 0 208.6kb 208.6kb
yellow open .wazuh-version 2TPSH17YQ4e_n6NiWDhQqQ 1 1 1 0 5.2kb 5.2kb
green open wazuh-monitoring-3.x-2019.07.19 8RDIhk0EQIOxNIYOOh6VXA 2 0 198 0 140.1kb 140.1kb

    Check if wazuh-alerts-3.x-* index is present with current date. If yes, then check if data is present in the index:

    curl localhost:9200/<INDEX_NAME>/_search?pretty=true&size=1000
    Example:
    localhost:9200/wazuh-alerts-3.x-2019.07.19/_search?pretty=true&size=1000

    This should return first 1000 entries.
    Check if latest entries are present. Also check if manager.name matches the hostname of the manager. If not, then change the hostname of the manager by executing command:

    hostname <HOSTNAME>
    Example:
    hostname abc.example.com

    The alerts in the Elasticsearch index will start coming in with the manager.hostname as abc.example.com

  3. Get details of a Wazuh agent:

    curl -u foo:bar localhost:55000/agents/<AGENT_ID>

    foo:bar is the credentials for wazuh api.

  4. Get details of nodes in Wazuh cluster: 
    curl -u foo:bar localhost:55000/cluster/nodes?pretty
  5. Rules not getting reflected aftere change in /var/ossec/ruleset/rules
    Restarting the wazuh-manger should reload the rules:

    systemctl restart wazuh-manager

  6. No events in wazuh-monitoring index:
    Check if index wazuh-monitoring-3.x-* is present with today’s date:

    curl elastic:9200/_cat/indices/wazuh-monitoring*

    Check if there is any error in wazuhapp for wazuh-monitoring:

    cat /usr/share/kibana/optimize/wazuh-logs/wazuhapp-plain.log | grep monitoring

    Execute:

    curl -XGET "http://elastic:9200/_cat/templates/wazuh-agent"

    If you get something like:
    wazuh-agent [wazuh-monitoring*, wazuh-monitoring-3.x-*] 0
    You probably have a template issue. Execute the following to resolve it:

    Stop Kibana: systemctl stop kibana

Delete the monitoring template, curl -XDELETE elastic:9200/_template/wazuh-agent

Restart Kibana: systemctl restart kibana, it should insert the monitoring template and the Kibana UI should start working shortly
  7. Change wazuh app to debug mode:
    Edit /usr/share/kibana/plugins/wazuh/config.yml
    Replace #logs.level: info with logs.level: debug, then restart Kibana service (systemctl restart kibana)
  8. Wazuh UI Error: “Saved field parameter is now invalid” OR “Error in Visualisation: field is a required parameter”
    You will require a cleanup. Execute the following commands:
    Delete wazuh-alerts-3.x-* index with today’s date:

    curl -XDELETE localhost:9200/wazuh-alerts-3.x-2019.07.18
systemctl restart wazuh-manager
curl -XDELETE localhost:9200/.kiban*
systemctl restart kibana
rm -f /var/ossec/queue/db/0*
systemctl restart wazuh-manager
  9. Generate SCA alerts: 
    rm -f /var/ossec/queue/db/0*
systemctl restart wazuh-manager
  10. Error in visualisation: Expected numeric type on field on data.sca.score got numeric:
    You most probably have wrong template. Just install the template according to your wazuh version from their github repo. To install latest (3.9.3) execute the following:

    curl https://raw.githubusercontent.com/wazuh/wazuh/v3.9.3/extensions/elasticsearch/6.x/wazuh-template.json | curl -X PUT "http://localhost:9200/_template/wazuh" -H 'Content-Type: application/json' -d @-
  11. java.lang.IllegalArgumentException: Rejecting mapping update to [wazuh-alerts-3.x] as the final mapping would have more than 1 type:
    This would most possibly mean you have wrong template. Install the latest one with above step.
    If already done you might have an issue with your logstash configuration:

    curl -so /etc/logstash/conf.d/01-wazuh.conf https://raw.githubusercontent.com/wazuh/wazuh/v3.9.3/extensions/logstash/7.x/01-wazuh-remote.conf

    Delete wazuh-alerts-3.x-* index with today’s date:

    curl -XDELETE localhost:9200/wazuh-alerts-3.x-2019.07.18systemctl restart wazuh-manager
curl -XDELETE localhost:9200/.kiban*
systemctl restart kibana
rm -f /var/ossec/queue/db/0*
systemctl restart wazuh-manager
  12. Version mismatch:
    If you get the above error on Wazuh UI, execute the following commands:

    service kibana stop
curl -XDELETE localhost:9200/.wazuh
curl -XDELETE localhost:9200/.wazuh_version
service kibana start
  13. Other useful commands: 
    Check documents in an index:
curl elastic:9200/_cat/indices/wazuh-monitoring*

Check wazuhapp logs:
cat /usr/share/kibana/optimize/wazuh-logs/wazuhapp-plain.log | grep monitoring

Check wazuh config:
cat /usr/share/kibana/plugins/wazuh/config.yml

Get Wazuh id:
curl elastic:9200/.wazuh/_search?pretty -s | grep "_id"

Templates:
curl elastic:9200/_cat/templates/wazuh

Version:
cat /usr/share/kibana/plugins/wazuh/package.json | grep version
cat /etc/ossec-init.conf | grep -i version
curl -u foo:bar localhost:55000/version

Monitoring settings:
cat /usr/share/kibana/plugins/wazuh/config.yml | grep monitoring

Search monitoring index:
curl elastic:9200/wazuh-monitoring*/_search

List of agents:
curl -u foo:bar "localhost:55000/agents?q=id!=000"

Index settings:
curl  "elastic:9200/wazuh-monitoring-3.x-2019.07.17/_settings"

Get details of template:
curl -XGET "http://elastic:9200/_cat/templates/wazuh-agent"

Check if filebeat is configured correctly:
filebeat test output