At trivago we run an internal cloud using Nomad from Hashicorp. Our logging solution is based on ELK and we use Filebeat to ship the logs from our client nodes into Kafka where it is later on ingested into Elasticsearch using Logstash. Previously we used the and input looking for new jobs in a defined path, but the logs lacked a lot of context/metadata from the Job definition/allocation.

This PR adds a new discover module (architecture based on the Kubernetes module). With this new provider, it is possible to start new harvesters by looking at the jobs allocated on each node. We currently run filebeat as a system job on each node and each filebeat instance is responsible for enriching and shipping the local logs.

Example of the configuration for the new provider:

filebeat.autodiscover:
  providers:
    - type: nomad
      host: {{ env "node.unique.name" }}
      hints.enabled: true
      hints.default_config:
        type: log
        paths:
          - /appdata/nomad/alloc/${data.meta.uuid}/alloc/logs/*stderr.[0-9]*
          - /appdata/nomad/alloc/${data.meta.uuid}/alloc/logs/*stdout.[0-9]*

By using the autodiscover module it is possible to define custom processors using the meta stanza on the Nomad job (similar to how it is defined using labels on Kubernetes). For instance:

task "nginx-web" {
    driver = "docker"

    meta {
    task-key = "custom-meta"
    "co.elastic.logs/processors.dissect.tokenizer" = "%{ip} - %{user} [%{local_time}] \"%{request}\" %{status} %{bytes_sent} \"%{referer}\" \"%{user_agent}\""
    }
}

This example defines a custom dissect tokenizer for the logs of this specific task that adds the dissect field with a content similar to:

"dissect": {
    "bytes_sent": "7231",
    "referer": "http://nginx-web.prod.trivago.com/",
    "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36",
    "ip": "10.2.10.138",
    "user": "-",
    "local_time": "15/Nov/2019:09:04:04 +0000",
    "request": "GET / HTTP/1.1",
    "status": "200"
}

By default the following fields are added from the Nomad job/allocation:

• job
• namespace
• status
• type (job type: system/service/batch)
• task.* (information about the task and custom metadata defined in the job/group/task using the meta stanza)
• datacenters
• region

The PR also includes an add_nomad_metadata processor that matches events to specific allocations and adds the metadata.

We've been running this in our production clusters for a few weeks now.

TODO:

• Metricbeat support for extracting stats from the Nomad allocation
• Documentation (it is a WIP and I will add a new commit with the documentation)
• CHANGELOG
• Fields reference

How to test locally

• Start a local development agent (nomad agent -dev).
• Start filebeat with a configuration like this one:

  filebeat.autodiscover:
    providers:
    - type: nomad
      templates:
        - config:
            - type: log
              paths:
                - /tmp/NomadClient*/${data.nomad.allocation.id}/alloc/logs/*stderr.[0-9]*
                - /tmp/NomadClient*/${data.nomad.allocation.id}/alloc/logs/*stdout.[0-9]*
  

• Run some service in nomad, for example:

  job "consul" {
    datacenters = ["dc1"]
  
    group "server" {
      task "consul-dev" {
        driver = "raw_exec"
  
        config {
          command = "consul"
          args = [
            "agent", "-dev",
          ]
        }
        artifact {
          source = "https://releases.hashicorp.com/consul/1.9.0/consul_1.9.0_linux_amd64.zip"
        }
      }
    }
  }
  

• Check that logs are collected for this service and they include the nomad metadata.
• Repeat with a configuration like this one for hints-based autodiscover:

  filebeat.autodiscover:
    providers:
    - type: nomad
      hints.enabled: true
      hints.default_config:
        type: log
        paths:
          - /tmp/NomadClient*/${data.nomad.allocation.id}/alloc/logs/*stderr.[0-9]*
          - /tmp/NomadClient*/${data.nomad.allocation.id}/alloc/logs/*stdout.[0-9]*