This commit resolves most of the comments made to the PR

P1llus · P1llus · commit 5a0e1e49be82 · 2021-01-29T15:19:51.000+01:00
diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -20774,16 +20774,6 @@ Module for parsing Cisco AMP logs.
 The timestamp in Epoch nanoseconds.
 
 
-type: date
-
---
-
-*`cisco.amp.date`*::
-+
---
-The timestamp in ISO8601 format.
-
-
 type: date
 
 --
@@ -21244,7 +21234,7 @@ type: keyword
 When the threat hunt finalized or closed.
 
 
-type: keyword
+type: date
 
 --
 
@@ -21254,7 +21244,7 @@ type: keyword
 When the threat hunt was initiated.
 
 
-type: keyword
+type: date
 
 --
 
diff --git a/filebeat/docs/modules/cisco.asciidoc b/filebeat/docs/modules/cisco.asciidoc
@@ -14,7 +14,7 @@ This is a module for Cisco network device's logs and Cisco Umbrella. It includes
 filesets for receiving logs over syslog or read from a file:
 
 - `asa` fileset: supports Cisco ASA firewall logs.
-- `amp` fileset: supports Cisco Umbrella logs.
+- `amp` fileset: supports Cisco AMP API logs.
 - `ftd` fileset: supports Cisco Firepower Threat Defense logs.
 - `ios` fileset: supports Cisco IOS router and switch logs.
 - `nexus` fileset: supports Cisco Nexus switch logs.
@@ -448,9 +448,9 @@ Maximum duration before AWS API request will be interrupted. Default to be 120 s
 [float]
 ==== `amp` fileset settings
 
-The Cisco AMP fileset focuses on collecting events from your Cisco AMP/Cisco Securi Endpoint API.
+The Cisco AMP fileset focuses on collecting events from your Cisco AMP/Cisco Secure Endpoint API.
 
-To configure the Cisco AMP fileset you will need to retrieve your client_id and client_key from the AMP dashboard.
+To configure the Cisco AMP fileset you will need to retrieve your `client_id` and `api_key` from the AMP dashboard.
 For more information on how to retrieve these credentials, please reference the https://api-docs.amp.cisco.com/api_resources?api_host=api.amp.cisco.com&api_version=v1[Cisco AMP API documentation].
 
 The URL configured for the API depends on which region your AMP is located, currently there is 3 choices:
@@ -488,12 +488,13 @@ It is also possible to select how often Filebeat will check the Cisco AMP API. A
     var.first_interval: 200h
     var.interval: 60m
     var.request_timeout: 120s
+    var.limit: 100
 
 ----
 
 *`var.input`*::
 
-The input from which messages are read. Supports httpjson(default) and file.
+The input from which messages are read. Supports httpjson.
 
 *`var.url`*::
 
@@ -516,6 +517,10 @@ timeout value for each request sent by Filebeat.
 
 How far back you would want to collect events the first time the Filebeat module starts up. Supports amount in hours.
 
+*`var.limit`*::
+
+This value controls how many events are returned by the Cisco AMP API per page.
+
 :has-dashboards!:
 
 :fileset_ex!:
diff --git a/x-pack/filebeat/module/cisco/_meta/docs.asciidoc b/x-pack/filebeat/module/cisco/_meta/docs.asciidoc
@@ -9,7 +9,7 @@ This is a module for Cisco network device's logs and Cisco Umbrella. It includes
 filesets for receiving logs over syslog or read from a file:
 
 - `asa` fileset: supports Cisco ASA firewall logs.
-- `amp` fileset: supports Cisco Umbrella logs.
+- `amp` fileset: supports Cisco AMP API logs.
 - `ftd` fileset: supports Cisco Firepower Threat Defense logs.
 - `ios` fileset: supports Cisco IOS router and switch logs.
 - `nexus` fileset: supports Cisco Nexus switch logs.
@@ -443,9 +443,9 @@ Maximum duration before AWS API request will be interrupted. Default to be 120 s
 [float]
 ==== `amp` fileset settings
 
-The Cisco AMP fileset focuses on collecting events from your Cisco AMP/Cisco Securi Endpoint API.
+The Cisco AMP fileset focuses on collecting events from your Cisco AMP/Cisco Secure Endpoint API.
 
-To configure the Cisco AMP fileset you will need to retrieve your client_id and client_key from the AMP dashboard.
+To configure the Cisco AMP fileset you will need to retrieve your `client_id` and `api_key` from the AMP dashboard.
 For more information on how to retrieve these credentials, please reference the https://api-docs.amp.cisco.com/api_resources?api_host=api.amp.cisco.com&api_version=v1[Cisco AMP API documentation].
 
 The URL configured for the API depends on which region your AMP is located, currently there is 3 choices:
@@ -483,12 +483,13 @@ It is also possible to select how often Filebeat will check the Cisco AMP API. A
     var.first_interval: 200h
     var.interval: 60m
     var.request_timeout: 120s
+    var.limit: 100
 
 ----
 
 *`var.input`*::
 
-The input from which messages are read. Supports httpjson(default) and file.
+The input from which messages are read. Supports httpjson.
 
 *`var.url`*::
 
@@ -511,6 +512,10 @@ timeout value for each request sent by Filebeat.
 
 How far back you would want to collect events the first time the Filebeat module starts up. Supports amount in hours.
 
+*`var.limit`*::
+
+This value controls how many events are returned by the Cisco AMP API per page.
+
 :has-dashboards!:
 
 :fileset_ex!:
diff --git a/x-pack/filebeat/module/cisco/amp/_meta/fields.yml b/x-pack/filebeat/module/cisco/amp/_meta/fields.yml
@@ -10,11 +10,6 @@
       description: >
         The timestamp in Epoch nanoseconds.
 
-    - name: date
-      type: date
-      description: >
-        The timestamp in ISO8601 format.
-
     - name: event_type_id
       type: keyword
       description: >
@@ -241,12 +236,12 @@
         The id of the related incident for the threat hunting activity.
 
     - name: threat_hunting.incident_end_time
-      type: keyword
+      type: date
       description: >
         When the threat hunt finalized or closed.
 
     - name: threat_hunting.incident_start_time
-      type: keyword
+      type: date
       description: >
         When the threat hunt was initiated.
 
diff --git a/x-pack/filebeat/module/cisco/amp/config/config.yml b/x-pack/filebeat/module/cisco/amp/config/config.yml
@@ -18,7 +18,7 @@ request.transforms:
     default: '[[ formatDate (now (parseDuration "-{{ .first_interval }}")) "2006-01-02T15:04:05-07:00" ]]'
 - set:
     target: url.params.limit
-    value: 100
+    value: {{ .limit }}
 request.rate_limit.limit: '[[ .last_response.header.Get "X-RateLimit-Limit" ]]'
 request.rate_limit.reset: '[[ .last_response.header.Get "X-RateLimit-Reset" ]]'
 request.rate_limit.remaining: '[[ .last_response.header.Get "X-RateLimit-Remaining" ]]'
@@ -68,4 +68,4 @@ processors:
   - add_fields:
       target: ''
       fields:
-        ecs.version: 1.6.0
+        ecs.version: 1.7.0
diff --git a/x-pack/filebeat/module/cisco/amp/ingest/pipeline.yml b/x-pack/filebeat/module/cisco/amp/ingest/pipeline.yml
@@ -1,4 +1,4 @@
-description: Pipeline for parsing checkpoint firewall logs
+description: Pipeline for parsing Cisco AMP logs
 processors:
 
 - remove:
@@ -54,7 +54,7 @@ processors:
     if: ctx?.cisco?.amp?.severity == 'High'
 - set:
     field: event.severity
-    value: 3
+    value: 4
     if: ctx?.cisco?.amp?.severity == 'Critical'
 - set:
     field: event.severity
@@ -94,11 +94,11 @@ processors:
     ignore_missing: true
 - set:
     field: network.direction
-    value: outbound
+    value: egress
     if: "ctx?.cisco?.amp?.network_info?.nfm?.direction == 'Outgoing connection from'"
 - set:
     field: network.direction
-    value: outbound
+    value: ingress
     if: "ctx?.cisco?.amp?.network_info?.nfm?.direction != null && ctx?.cisco?.amp?.network_info?.nfm?.direction != 'Outgoing connection from'"
 
 #####################
@@ -114,32 +114,24 @@ processors:
 ########################
 - rename:
     field: cisco.amp.network_info.local_ip
-    target_field: source.address
+    target_field: source.ip
     ignore_missing: true
 - rename:
     field: cisco.amp.network_info.local_port
     target_field: source.port
     ignore_missing: true
-- set:
-    field: source.ip
-    value: "{{ source.address }}"
-    if: ctx?.source?.address != null
 
 #############################
 ## ECS Destination Mapping ##
 #############################
 - rename:
     field: cisco.amp.network_info.remote_ip
-    target_field: destination.address
+    target_field: destination.ip
     ignore_missing: true
 - rename:
     field: cisco.amp.network_info.remote_port
     target_field: destination.port
     ignore_missing: true
-- set:
-    field: destination.ip
-    value: "{{ destination.address }}"
-    if: ctx?.destination?.address != null
 
 ######################
 ## ECS File Mapping ##
@@ -263,6 +255,10 @@ processors:
     value: "{{ cisco.amp.network_info.parent.identity.sha1 }}"
     if: ctx?.cisco?.amp?.network_info?.parent?.identity?.sha1 != null
     allow_duplicates: false
+- append:
+    field: related.hosts
+    value: "{{ host.name }}"
+    if: ctx?.host?.name != null
 - append:
     field: related.ip
     value: "{{ source.ip }}"
@@ -271,6 +267,10 @@ processors:
     field: related.ip
     value: "{{ destination.ip }}"
     if: ctx?.destination?.ip != null
+- append:
+    field: related.ip
+    value: "{{ cisco.amp.computer.external_ip }}"
+    if: ctx?.cisco?.amp?.computer?.external_ip != null
 - foreach:
     field: cisco.amp.computer.network_addresses
     processor:
@@ -293,9 +293,95 @@ processors:
         value: "{{ _ingest._value.cve }}"
     if: ctx?.cisco?.amp?.vulnerabilities != null
 
+#############
+## GeoIP ##
+#############
+- geoip:
+    field: source.ip
+    target_field: source.geo
+    ignore_missing: true
+    if: "ctx.source?.geo == null"
+- geoip:
+    field: destination.ip
+    target_field: destination.geo
+    ignore_missing: true
+    if: "ctx.destination?.geo == null"
+- geoip:
+    database_file: GeoLite2-ASN.mmdb
+    field: source.ip
+    target_field: source.as
+    properties:
+    - asn
+    - organization_name
+    ignore_missing: true
+- geoip:
+    database_file: GeoLite2-ASN.mmdb
+    field: destination.ip
+    target_field: destination.as
+    properties:
+    - asn
+    - organization_name
+    ignore_missing: true
+- rename:
+    field: source.as.asn
+    target_field: source.as.number
+    ignore_missing: true
+- rename:
+    field: source.as.organization_name
+    target_field: source.as.organization.name
+    ignore_missing: true
+- rename:
+    field: destination.as.asn
+    target_field: destination.as.number
+    ignore_missing: true
+- rename:
+    field: destination.as.organization_name
+    target_field: destination.as.organization.name
+    ignore_missing: true
+
 #############
 ## Cleanup ##
 #############
+- date:
+    field: cisco.amp.threat_hunting.incident_start_time
+    target_field: cisco.amp.threat_hunting.incident_start_time
+    formats: 
+      - UNIX
+    ignore_failure: true
+    if: ctx?.cisco?.amp?.threat_hunting?.incident_start_time != null
+- date:
+    field: cisco.amp.threat_hunting.incident_end_time
+    target_field: cisco.amp.threat_hunting.incident_end_time
+    formats: 
+      - UNIX
+    ignore_failure: true
+    if: ctx?.cisco?.amp?.threat_hunting?.incident_end_time != null
+
+- script:
+    lang: painless
+    if: ctx?.json != null
+    source: |
+      void handleMap(Map map) {
+        for (def x : map.values()) {
+          if (x instanceof Map) {
+              handleMap(x);
+          } else if (x instanceof List) {
+              handleList(x);
+          }
+        }
+      map.values().removeIf(v -> v == null);
+      }
+      void handleList(List list) {
+        for (def x : list) {
+            if (x instanceof Map) {
+                handleMap(x);
+            } else if (x instanceof List) {
+                handleList(x);
+            }
+        }
+      }
+      handleMap(ctx);
+
 - remove:
     field:
     - cisco.amp.timestamp
diff --git a/x-pack/filebeat/module/cisco/amp/manifest.yml b/x-pack/filebeat/module/cisco/amp/manifest.yml
@@ -10,6 +10,8 @@ var:
   - name: ssl
   - name: request_timeout
     default: 60s
+  - name: limit
+    default: 100
   - name: client_id
   - name: api_key
   - name: first_interval
diff --git a/x-pack/filebeat/module/cisco/amp/test/cisco_amp.ndjson.log-expected.json b/x-pack/filebeat/module/cisco/amp/test/cisco_amp.ndjson.log-expected.json
@@ -63,7 +63,11 @@
             "9a8557b98ed1469272fa0ace91d63477",
             "d0c4192b65e36553fvfd2b83f3113f6ae8390baa"
         ],
+        "related.hosts": [
+            "testhost"
+        ],
         "related.ip": [
+            "8.8.8.8",
             "192.168.196.22",
             "192.168.120.1",
             "192.168.160.1"
diff --git a/x-pack/filebeat/module/cisco/amp/test/cisco_amp2.ndjson.log-expected.json b/x-pack/filebeat/module/cisco/amp/test/cisco_amp2.ndjson.log-expected.json
diff --git a/x-pack/filebeat/module/cisco/fields.go b/x-pack/filebeat/module/cisco/fields.go
