Translator for converting Zeek logs to ILFs. Translator's compatibility with Zeek v5.2.2 and v4.2.2 are verified. Zeek v5.2.2 has breaking changes with zeek_to_ilf v4.0.4. Please do not use zeek_to_ilf v5.2.2 with the uncompatible versions of zeek mentioned

1.0.1

If a bad interpreter error is encountered,

1. Download dos2unix
2. cd /path/to/zeek_to_ilf
3. find . -type f -print0 | xargs -0 dos2unix

Before running zeek_to_ilf, you must first install Zeek. Depending on your use case, you may either install Zeek as a binary package or build it from source. If you are running Zeek in a performance-sensitive environment, then it is recommended to install and configure Zeek using the instructions in docs/perf.md. Otherwise, you can continue with this README.

1. Follow the instructions here to install the binary release package of Zeek for the appropriate system. Note that it is recommended to add the repository and install manually so that any dependencies will be resolved by the package manager. This zeek_to_ilf translator was written for Zeek v5.2.2. Please use this version.

2. After installing Zeek, you will have to modify sudo's $PATH environment variable to point to Zeek. You may do so by running sudo visudo and adding the path to Zeek to secure_path, which, by default, will be /usr/local/zeek/bin:

sudo visudo
    Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/zeek/bin

3. Zeek will also have to be added to the user's $PATH environment variable, which can be done as follows:

echo PATH=/path/to/zeek:\$PATH | tee -a ~/.bashrc
source ~/.bashrc
zeek -v

4. After installing Zeek, you must clone this repository (note that it is not recommended to download this repo as a .zip/.tar.gz since any submodules will not be included in the download):

git clone https://github.com/mitre/zeek_to_ilf.git
cd zeek_to_ilf
(optional) git config --global http.sslVerify "False"
git submodule update --init --recursive

5. Install Hiredis after cloning this repo:

cd hiredis
make
sudo make install
cd ..

6. Install the Redis plugin in zeek_to_ilf, which will allow the translator to publish ILFs to Redis:

cd plugins/redis-plugin
./configure
make
sudo make install
zeek -NN HBL::Redis
cd ../..

7. OPTIONAL: This step is not required to run the translator, but if you would like to install the translator alongside Zeek in a location available to other users, you may run the INSTALL.sh script in this repo:

sudo ./INSTALL.sh

Zeek, along with the translator, can be run with the following command in the root of the zeek_to_ilf directory:

sudo zeek -Ci <interface> ./scripts

The <interface> option can be obtained with ip addr:

ip addr
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host 
           valid_lft forever preferred_lft forever
    2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether ac:1f:6b:0a:6b:de brd ff:ff:ff:ff:ff:ff
        inet 172.16.0.115/24 brd 172.16.0.255 scope global noprefixroute eno1
           valid_lft forever preferred_lft forever
    3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether ac:1f:6b:0a:6b:df brd ff:ff:ff:ff:ff:ff
        inet 169.254.114.65/16 brd 169.254.255.255 scope link noprefixroute eno2
           valid_lft forever preferred_lft forever

The interface name will be the leftmost word in the top row for each entry. For example, we see the interfaces lo, eno1, and eno2 above.

Alternatively, if you ran the INSTALL.sh script, you can replace the ./scripts argument with zeek_to_ilf as follows:

sudo zeek -Ci <interface> zeek_to_ilf

From the root of the project, run: docker build -f Dockerfile.production -t zeek_to_ilf:latest .

Then, to start the image, run one of the following:

docker run --rm --cap-add=NET_ADMIN zeek_to_ilf:latest

docker run --rm -it --entrypoint /bin/bash -v pcap:/Zeek/pcap zeek_to_ilf:latest Once inside the container, run: zeek -r pcap/<file> ./scripts

NOTE: To run zeek live, you will need to have redis running and accessible from the contianer with the hostname "redis". For docker, this means a container will need to be running with the name "redis".

zeek_to_ilf can be configured to change the names of events and attributes or omit them altogether. To do so, modify cfg/zeek_to_ilf.cfg. The configuration file is structured such that each line contains the name of a value internal to the translator in the left column and the name of the value as it will appear in generated ILFs in the right column. Additionally, an attribute or event can be omitted entirely by setting its value to \x00.

The parameters for connecting to Redis can also be configured in cfg/redis_params.zeek, which is implemented as a Zeek script. An interesting note to make is that Redis configuration can only be performed once when the translator starts up whereas parameters in cfg/zeek_to_ilf.cfg can be changed in real-time while the translator is running.

If you are running the translator as installed underneath the same directory as Zeek, then the configuration files you must modify change to the following path:

$(dirname $(dirname $(which zeek)))/share/zeek/site/zeek_to_ilf/cfg

Additionally, a list of options for configuring Zeek itself can be obtained with zeek -h.

A list of all events and their corresponding attributes generated by zeek_to_ilf can be found in analytic_defs/zeek_to_ilf.hbl. zeek_to_ilf generates ILFs for a substantial number of Zeek logs, a comprehensive list of which is below:

The mapping of log files to event names is simply the name of the log file (e.g., conn.log) prepended with Zeek all in PascalCase (e.g., conn.log -> ZeekConn). The only exception is ZeekARP events since there is no arp.log.

When in doubt of the significance of an attribute, please refer to the Zeek documentation here. The mapping of values in Zeek logs to ILF attributes is a (mostly) one-to-one mapping, and so a verbose explanation of every ILF attribute can be found in the Zeek documentation. For example, all values stored in conn.log can be found here.

It is worth mentioning that not all values in Zeek logs are included in generated ILFs. Some values require that certain Zeek scripts be loaded. When in doubt, refer to the analytic definitions in analytic_defs/zeek_to_ilf.hbl.

For most use cases, this README should contain sufficient documentation. However, this repo also has a docs directory containing further documentation on exceptional cases:

• docs/perf.md: explains how to install and configure Zeek for high-performance use cases.
• docs/ics.md: explains how to use the translator for writing ICS (Industrial Control Systems) analytics.
• docs/dev.md: explains how the translator is implemented to aid with development on it.

This software is licensed under the Apache 2.0 license.