Tag: order

Writing OSSEC Custom Rules, Decoders and Alerts

OSSEC (http://www.ossec.net) is an open-source host-based intrusion detection system (HIDS). OSSEC can be used to monitor your local files and logs to check for intrusions, alert you of rootkit installation and do file integrity checking. OSSEC is a wonderful tool because it is highly customizable. By default, OSSEC monitors many of the programs commonly installed on a machine, but its real power comes from the ability of system administrators to customize OSSEC. By writing custom rules and decoders, you can allow OSSEC to parse through non-standard log files and generate alerts based on custom criteria. This allows OSSEC to monitor custom applications and provide intrusion detection services that might otherwise not be available, or would have to be developed on a per-application basis.

OSSEC rules are based on log file parsing. The log files that OSSEC monitors are specified in the /var/log/ossec/etc/ossec.conf file in the following format:

  <!-- Files to monitor (localfiles) -->

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/messages</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/secure</location>
  </localfile>

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/log/maillog</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/httpd/error_log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/httpd/access_log</location>
  </localfile>

Each file that is monitored depends on a “decoder” which is a regular expression used to parse up the pieces of the log file to extract fields such as the source IP, the time, and the actual log message. The decoders are stored in /var/ossec/etc/decoder.xml. The following is an extract of the SSH decoder portion of the decoder.xml logfile:

<decoder name="sshd">
  <program_name>^sshd</program_name>
</decoder>

<decoder name="sshd-success">
  <parent>sshd</parent>
  <prematch>^Accepted</prematch>
  <regex offset="after_prematch">^ \S+ for (\S+) from (\S+) port </regex>
  <order>user, srcip</order>
  <fts>name, user, location</fts>
</decoder>

<decoder name="ssh-denied">
  <parent>sshd</parent>
  <prematch>^User \S+ from </prematch>
  <regex offset="after_parent">^User (\S+) from (\S+) </regex>
  <order>user, srcip</order>
</decoder>

You can see that the decoder.xml file is used to parse through the log file using regular expression pattern matching. This means that you can add additional files to the list of those which OSSEC is checking if you would like. You’ll also note that the XML rules in decoder.xml are nested so that you can use the <parent> tag to nest rules. A rule with a “parent” will only attempt matching if the parent rule matched successfully. Using the order and its statements you can populate OSSEC’s predefined variables with portions of the log file. The following variables are supported:

  • location
  • hostname
  • log_tag
  • srcip
  • dstip
  • srcport
  • dstport
  • protocol
  • action
  • user
  • dstuser
  • id
  • command
  • url
  • data

Supposing you have a log file produced by an application that isn’t covered by the default decoders you could write your own decoder and parsing rules. Unfortunately OSSEC only supports logs in the formats syslog, snort-full, snort-fast, squid, iis, eventlog, mysql_log, postgresql_log, nmapg or apache. Therefore any custom logging you write must conform to one of these formats. Syslog is probably the easiest to use as it is designed to handle any one line log entry.

Let us suppose we have a custom PHP based application that resides in /var/www/html/custom. Our application will write Apache format logs to a file called ‘alert.log’ in the ‘logs/’ application subdirectory. This program has the following lines in example.php:

<?php

$id = $_GET['id'];
$logfile = 'logs/alert.log';
if (! is_numeric($_GET['id'])) {
	$timestamp = date("Y-m-d H:m:s ");
	$ip = $_SERVER['REMOTE_ADDR'];
	$log = fopen($logfile, 'a');
	$message = $timestamp . $ip . ' PHP app Attempt at non-numeric input (possible attack) detected!' . "\n";
	fwrite($log, $message);
}

?>

This would write a log file to /var/www/html/custom/logs/alert.log in the format:

2009-10-13 11:10:36 192.168.97.1 PHP app Attempt at non-numeric input (possible attack) detected!

Once we have this application log set up we need to adjust our OSSEC configuration so that it reads the new log file. The following change needs to be done in both agent and server’s ossec.conf file. We can add the following lines to our /var/ossec/etc/ossec.conf file to enable OSSEC to read this new log file:

  <localfile>
    <log_format>syslog</log_format>
    <location>/var/www/html/custom/logs/alert.log</location>
  </localfile>

Once OSSEC is monitoring this file (this will require us to restart OSSEC) we’ll need an appropriate decoder. Make this change on ossec server. Add the following in /var/ossec/etc/local_decoder.xml. By default ossec reads only 2 decoder files: decoder.xml and local_decoder.xml. decoder.xml can be overwritten during upgrades, so add all the custom decoders in local_decoder.xml. Writing a decoder for this format would be quite simple. It would appear as:

<!-- Custom decoder for example -->
<decoder name="php-app">
  <prematch>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d</prematch>
</decoder>

<decoder name="php-app-alert">
  <parent>php-app</parent>
  <regex offset="after_parent">^ (\d+.\d+.\d+.\d+) PHP app</regex>
  <order>srcip</order>
</decoder>

What we’re doing here is telling OSSEC how to extract IP information from the log. All the strings in the regex portion of the new decoder can be assigned, in order, to options listed in the order tag. You can define each of OSSEC’s possible variables and tell OSSEC how to identify them in the logs using the decoder.

Once we have our decoder we can write custom rules based on the log file. This is to be done on ossec server. There are two ways to create custom rules for OSSEC. The first is to alter the ossec.conf configuration file and add a new rule file to the list. The second is to simply append your rules to the local-rules.xml rules file. Either one works, but the second makes upgrading to newer versions of OSSEC a little cleaner.

We’ll add the following group to our local-rules.xml file, found in the rules directory under the OSSEC installation root:

<group name="syslog,php-app,">
  <rule id="110000" level="0">
    <decoded_as>php-app</decoded_as>
    <description>PHP custom app group.</description>
  </rule>

  <rule id="110001" level="10">
    <if_sid>110000</if_sid>
    <srcip>127.0.0.1</srcip>
    <match>attack</match>
    <description>Possible attack from localhost?!?</description>
  </rule>
</group>

You’ll notice that we have two rules. Because rules can be nested it is usually helpful to subdivide them into small, hierarchical pieces. In this case, we have one rule that serves as a catch-all for our custom application alerts. After that, we can write rules for any number of circumstances and have these rules only checked if the parent rule is matched. This rule will fire if an entry is written into the custom alert.log that contains the source IP of 127.0.0.1. The rule id is extremely important in this definition. OSSEC reserves rule id’s above 100,000 for custom rules. It is useful to develop a schema for your new rules, for instance allocating each 1000 above 100,000 for a generic, catch-all rule and writing child rules in that space. This helps to avoid the hassle of having intermingled rule numbers and aids in long term maintenance.

To clarify the case above, there are two rules. The first rule will only fire if a log entry is “decoded_as” or matches the decoder for “php-app.” If this decoder is used then rule 110,000 will be triggered. The second rule is only checked if rule 110,000 is triggered as specified in the if_sid tag. This rule will only be triggered if the source ip, specified in the srcip tag, is equal to ‘127.0.0.1.’ If this is the case then the rule will do a string match for the word “attack” in the log entry. If this match is successful the rule will trigger at level 10 as specified in the rule tag. This will cause an OSSEC alert to be logged with the associated description. OSSEC by default also attempts to e-mail alerts with level 7 or higher to recipients specified in the ossec.conf file. As you can see, with the addition of the decoder and these rules we’ve allowed OSSEC to read our custom format logfile.

While this example may seem straightforward writing your own decoders and rules can be maddening. Because OSSEC will not dynamically load the XML files defining your decoders, rules, or files to watch, you must restart the program to propagate changes. This can be a real hassle when you’re debugging new XML rules or decoders. To alleviate the problem of constantly restarting the server you can use the program ossec-logtest found in the bin directory of the OSSEC installation root. This is present on the ossec server. This program allows you to paste, or type, one line of a log file into the input then traces the decoders and rules that the line matches like so:

# bin/ossec-logtest 
2009/10/13 13:30:25 ossec-testrule: INFO: Started (pid: 14330).
ossec-testrule: Type one log per line.

2009-10-13 12:10:09 127.0.0.1 PHP app Attempt to attack the host!


**Phase 1: Completed pre-decoding.
       full event: '2009-10-13 12:10:09 127.0.0.1 PHP app Attempt to attack the host!'
       hostname: 'webdev'
       program_name: '(null)'
       log: '2009-10-13 12:10:09 127.0.0.1 PHP app Attempt to attack the host!'

**Phase 2: Completed decoding.
       decoder: 'php-app'
       srcip: '127.0.0.1'

**Phase 3: Completed filtering (rules).
       Rule id: '110001'
       Level: '10'
       Description: 'Possible attack from localhost?!?'
**Alert to be generated.

Note that this program will not reload changes, but you can quit ossec-logtest, make changes to any of the XML files then restart it to test your changes. Using ossec-logtest is invaluable when trying to create new rules as it saves you the hassle of restarting the server and the hassle of actually triggering events for which you want to generate alerts.

In case you get an issue like “No decoders match” in ossec-logtest, please check your decoder file. It might be most probably due to some syntax error in your decoder xml.

After your testing in logtest is successful, restart your ossec agent and master with

/var/ossec/bin/ossec-control restart

After this your alerting system will be in place for your custom logs just like other alerts.

OSSEC: Writing your own rules

Before we start writing rules, we should be aware of some rules to be followed:

  • The first rule of writing custom rules is to never modify the existing rule files in the /var/ossec/rules directory except local_rules.xml. Changes to those rules may modify the behavior of entire chains of rules and complicate troubleshooting.
  • The second rule of writing custom rules is to use IDs above 100000 as IDs below it are reserved. Interfering with those IDs is the same as tampering with the distributed rules files themselves. You risk an update of OSSEC clobbering all your hard work.
  • The third rule is to maintain order in your rules. As the rules parser is loading the rules at startup, it validates the existence of referenced rules and groups. If you reference a rule that hasn’t been loaded, the parser will fail.

Minding these three rules helps to ensure that our installations won’t break with upgrades and that we can always get back to a “stock” OSSEC by removing the local_rules.xml file from our configuration.

Every rule must have an ID, a level, a description, and a match condition. The IDs must be unique, and our rules must have an ID over 100000. It’s important to note that re-using or reordering rule IDs can cause confusion or inaccuracy in historic data.

Rules in OSSEC have a level from 0 to 15. The higher the level, more certain the analyzer is of an attack. Level 0 is a special level to tell OSSEC to ignore the alerts where no log will be generated and OSSEC will discard the alert and data silently. By default, OSSEC considers anything at or exceeding level 7 to be e-mail worthy, but it is also configurable.

Rules also require a description field to explain what the rule does. This description will be used as the event identifier in the e-mails and log messages that OSSEC generates. As it will be a part of the reporting, it’s best to explain the rule professionally and format it consistently. Descriptions like “alerting for this thing” won’t be helpful to your colleagues, whereas “Ignore failed login attempts from vulnerability scanners between 4:00 and 7:00” will be clear and informative.

Now that we’re well versed with the protocols of the rules, let’s process some data from our custom application logging via syslog as follows:

May 4 19:12:03 server custom-app: Startup initiated.
May 4 19:12:07 server custom-app: No error detected during startup!
May 4 19:12:08 server custom-app: Startup completed, processing data.
May 4 19:12:08 server custom-app: Failed login from '4.5.6.7' as testuser

We’re receiving an alert about unknown errors and authentication failures from our custom application. We would prefer to silence these unknown error messages and ensure that we don’t provide alerts for failed logins from 4.5.6.7, our vulnerability scanner.

How to do it…

In order to figure out the first step, we need to understand what’s happening to generate the alerts:

  1. Use the ossec-logtest tool provided by OSSEC. It works by accepting log messages on STDIN (your terminal input) and explaining the path through the rules. Here’s how we can run it: 
    $ sudo /var/ossec/bin/ossec-logtest
  2. Then we can paste in log lines to see which ones are generating alerts: 
    ossec-testrule: Type one log per line.

May  4 19:12:03 server custom-app: Startup initiated.

**Phase 1: Completed pre-decoding.
     full event: 'May  4 19:12:03 server custom-app: Startup initiated.'
     hostname: 'server'
     program_name: 'custom-app'
     log: 'Startup initiated.'

**Phase 2: Completed decoding.
     No decoder matched.
  3. The first log message completed the parsing of the line and no alert was generated. So we try the next log message: 
    **Phase 1: Completed pre-decoding.
     full event: 'May  4 19:12:07 server custom-app: No error detected during startup!'
     hostname: 'server'
     program_name: 'custom-app'
     log: 'No error detected during startup!'

**Phase 2: Completed decoding.
     No decoder matched.

**Phase 3: Completed filtering (rules).
     Rule id: '1002'
     Level: '2'
     Description: 'Unknown problem somewhere in the system.'
**Alert to be generated.
  4. We can see from this output that our unknown problem is being generated by this log line. We get the rule ID and the level being generated and using this information, we can write a rule to ignore it using OSSEC’s level="0". 
    <!-- Local Rules for Example.com -->
<group name="local,syslog,">
   <rule id="100000" level="0">
     <if_sid>1002</if_sid>
     <program_name>custom-app</program_name>
     <description>Ignore errors for custom-app</description>
   </rule>
</group>
  5. Once we’ve saved the local_rules.xml file, we can restart ossec-logtest and try the event again: 
    **Phase 3: Completed filtering (rules).
     Rule id: '100000'
     Level: '0'
     Description: 'Ignore unknown errors for custom-app'
  6. Now that we’ve moved this event to level 0, we can look at the failed login events: 
    May  4 19:12:08 server custom-app: Failed login from '4.5.6.7' as testuser'
**Phase 3: Completed filtering (rules).
     Rule id: '2501'
     Level: '5'
     Description: 'User authentication failure.'
**Alert to be generated.
  7. We’ll use a simple match with this data to silence this alert from 4.5.6.7: 
    <rule id="100001" level="0">
  <if_sid>2501</if_sid>
  <program_name>custom-app</program_name>
  <match>4.5.6.7</match>
  <description>Ignore failed logins from scanner</description>
</rule>
  8. And now re-running ossec-logtest, we see: 
    May  4 19:12:08 server custom-app: Failed login from '4.5.6.7' as testuser'

**Phase 3: Completed filtering (rules).
     Rule id: '100001'
     Level: '1'
     Description: 'Ignore failed logins from our security scanner'
**Alert to be generated.

Using these two rules, we’ve been able to silence the noisiest log entries in our sample environment.

How it works…

OSSEC rules are processed sequentially. Each rule has a number of conditions and a logical AND is applied to the conditions. The more specific we make the rule, the more accurate it will be. In our example, we filtered events using the program_name attribute for the string, custom-app.

Each rule also specifies an if_sid element, which requires the log message to be flagged with the rule ID we specify. The if_sid parameter can take a comma-separated list of rule IDs, where the rule will match if any of those IDs are matched. Consider that multiple instances of the same element appear in a rule; refer to the following example:

<match>illegal user</match>
<match>unknown user</match>
<match>invalid user</match>

That grouping is surrounded in logical OR. It’s important to note that after our rules match, the ID is changed and other rules looking for those lines with the same if_sid parameter, which was originally set, will fail to match. When a new rule matches, it replaces the attributes of the alert with its own values, replacing the ID and level.

While our first rule stopped at eliminating the match based on the rule ID and program name only, the second rule used the match attribute to find a string in the log message itself. In addition to match, there is also a regex attribute to allow more flexible matching of strings.