WARN: Waiting for server reply (not started) OSSEC agent

The first thing to understand is how to check the status of your agents and easiest way to do that is running the following on the server install (my mothership):

# /var/ossec/bin/agent_control -lc

This will list out all your agents and if they are active it’ll read Active. If they are inactive, they don’t read inactive unfortunately, they just don’t show up.

The next thing is to check your logs and in the default installations this is where it’ll be:

# tail -F /var/ossec/logs/ossec.log

If you have a connection issue you’re likely to see something like the following in the client log:

2012/10/09 03:39:33 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: '[mothership IP]'.
2012/10/09 03:39:35 ossec-agentd: INFO: Trying to connect to server ([mothership IP]:1514).
2012/10/09 03:39:35 ossec-agentd: INFO: Using IPv4 for: [mothership IP] .
2012/10/09 03:39:56 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: '[mothership IP]'.
2012/10/09 03:40:16 ossec-agentd: INFO: Trying to connect to server ([mothership IP]:1514).
2012/10/09 03:40:16 ossec-agentd: INFO: Using IPv4 for: [mothership IP].

As you are probably thinking this isn’t exactly the most helpful of warnings, it’s not telling you anything about the issue. But you do know you can’t connect. A couple of things I can say that will help troubleshoot on the client box is to do the following:

First check your IPTABLES rules:

# iptables -nL

If you have a number of rules and policies you might want to try disabling everything to see if you can establish a connection. To verify that its reaching the mothership server though you’ll want to run tcpdump on the mothership and see if any packets are reaching the box. Easiest way is to do the following:

# tcpdump -i eth0 port 1514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

Note that eth0 is your network interface card. If on a NIX box you can run ifconfig and you’re looking for the card that has your internet protocol address next to the inet addr:. So it’d look like this:

# ifconfig
eth0      Link encap:Ethernet  HWaddr G3:4P:91:CD:5A:6B  
          inet addr:100.1.5.68  Bcast:100.1.5.255  Mask:255.255.255.0

Once you identify the interface that is what you define in the syntax. And port is the UDP port that is used to communicate, if you didn’t change it on setup then it’ll be 1514. If it’s running you’ll start seeing traffic coming into the box as the servers kick it into gear. The easiest way to get it talking is to restart the agent boxes and you can do so here:

# /var/ossec/bin/ossec-control restart

If you have cleared your firewall and you don’t see traffic take a look at the ossec.log file on the mothership to see what might be going on. If you see the following you’re in luck:

# tail -F /var/ossec/logs/ossec.log
2012/10/09 03:47:17 ossec-remoted: WARN: Duplicate error:  global: 0, local: 51, saved global: 5, saved local:7563
2012/10/09 03:47:17 ossec-remoted(1407): ERROR: Duplicated counter for 'Agent001'.
2012/10/09 03:47:23 ossec-remoted: WARN: Duplicate error:  global: 0, local: 52, saved global: 5, saved local:7563
2012/10/09 03:47:23 ossec-remoted(1407): ERROR: Duplicated counter for 'Agent001'.
2012/10/09 03:47:27 ossec-remoted: WARN: Duplicate error:  global: 0, local: 53, saved global: 5, saved local:7563
2012/10/09 03:47:27 ossec-remoted(1407): ERROR: Duplicated counter for 'Agent001'.
2012/10/09 03:47:32 ossec-remoted: WARN: Duplicate error:  global: 0, local: 54, saved global: 5, saved local:7563
2012/10/09 03:47:32 ossec-remoted(1407): ERROR: Duplicated counter for 'Agent001'.
2012/10/09 03:47:38 ossec-remoted: WARN: Duplicate error:  global: 0, local: 55, saved global: 5, saved local:7563
2012/10/09 03:47:38 ossec-remoted(1407): ERROR: Duplicated counter for 'Agent001'.

If it still doesn’t work try removing the agent from master and re-adding it:

OSSEC start problem due to keys

UNIX / Linux: Explains setuid File Permission

setuid means set user ID upon execution. If setuid bit turned on a file, user executing that executable file gets the permissions of the individual or group that owns the file. You need to use the ls -l or find command to see setuid programs. All setuid programs displays S or s in the permission bit (owner-execute) of the ls command. Type the following command:

ls -l /usr/bin/passwd

Sample outputs:

-rwsr-xr-x 1 root root 42856 2009-07-31 19:29 /usr/bin/passwd

How Do I List All setuid Enabled Files?

The following command discovers and prints any setuid files on local system:
# find / -xdev \( -perm -4000 \) -type f -print0 | xargs -0 ls -l

The s bit can be removed with the following command:
# chmod -s /path/to/file

Setuid Programs Risk

A attacker can exploit setuid binaries using a shell script or by providing false data. Users normally should not have setuid programs installed, especially setuid to users other than themselves. For example, you should not find setuid enabled binary for root under /home/vivek/crack. These are usually Trojan Horses kind of programs.

Example

In this example, user vivek run the command called “/usr/bin/vi /shared/financialdata.txt”, and the permission on the vi command and the file /shared/financialdata.txt are as follows:

-rwxr-xr-x 1 root root 1871960 2009-09-21 16:57 /usr/bin/vi
-rw------- 1 root root    3960 2009-09-21 16:57 /shared/financialdata.txt

Vivek has permission to run /usr/bin/vi, but not permission to read /shared/financialdata.txt. So when vi attempts to read the file a “permission denied” error message will be displayed to vivek. However, if you set the SUID bit on the vi:

chmod u+s /usr/bin/vi
ls -l /usr/bin/vi

Now, when vivek runs this SUID program, the access to /shared/financialdata.txt is granted. How does it work? The UNIX system doesn’t think vivek is reading file via vi, it thinks “root” is the user and hence the access is granted.

How Do I Audit And Log setuid System Call Under Linux For Each setuid Binary?

auditd can be used for system auditing under Linux. It can log and audit setuid system call. Edit /etc/audit/audit.rules:
# vi /etc/audit/audit.rules

-a always,exit -F path=/bin/ping -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/mount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/su -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged
-a always,exit -F path=/bin/umount -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged

Run the following command to get setuid enabled binary from /bin and add them as above:
# find /bin -type f -perm -04000
Save and close the file. Restart auditd:
# service auditd restart
Use aureport command to view audit reports:
# aureport --key --summary
# ausearch --key access --raw | aureport --file --summary
# ausearch --key access --raw | aureport -x --summary
# ausearch --key access --file /bin/mount --raw | aureport --user --summary -i