Summary
The Logstash GeoIP filter uses the a maxminddb-formatted database (that may or may not be provided by Maxmind) to populate a number of fields to enrich an event based on an IP address. As presently implemented in the Logstash filter, all fields need to be sub-fields of a single target (e.g, with target => client, we would have client.geo.*, client.as.*, etc.).
Motivation:
Clearly define destinations for all GeoIP Filter fields as a sub-field of a single target, so that users can enable ECS Compatibility Mode without losing metadata that they currently rely on.
Specifically, we have six fields that do not have direct analogues in ECS:
timezone: the IANA name of the timezone e.g., America/New_Yorkpostal_code: a string postal code, length varies by countrycontinent_code: "A two character continent code like "NA" (North America) or "OC" (Oceania)"organization: the name of the business or ISP associated with an address, reportedly available for ~40% of lookups.isp: the name of the ISPdma_code: us-only code representing Designated Market Area (~metro area)
Many of the existing ECS fields underneath geo.* are named aligning with Geo*2 like these here. I see some fields, such as timezone, being a good candidates as ECS additions, but I’m not so sure about others, such as the US-specific dma_code.
-- @ebeahan
Detailed Design:
- propose the addition of specs to ECS for
geo.timezone (IANA name, presently up to 30 characters, e.g., America/Argentina/Buenos_Aires),geo.postal_code (freeform but relatively space limited, regulated by each country; see wikipedia),geo.continent_code (docs define AF, AN, AS, EU, NA, OC, and SA)
- guidance for namespacing the other fields so that they are usable and unlikely to present future conflict, bearing in mind that they must be sub-fields of the singular
target that is the parent of the related geo and as fields.
- (a) under a
mmdb sub-key? e.g, ${target}.mmdb.organization
- since the
mmdb being used may or may not be provided by Maxmind, I'd like to avoid explicitly using "maxmind" in the key name
dma_code, while provided in mmdb, is a US-only Nielsen Ratings construct and may not be adequately described with a mmdb prefix.
- (b) some other way?
Related: logstash-plugins/logstash-filter-geoip#163
Summary
The Logstash GeoIP filter uses the a maxminddb-formatted database (that may or may not be provided by Maxmind) to populate a number of fields to enrich an event based on an IP address. As presently implemented in the Logstash filter, all fields need to be sub-fields of a single
target(e.g, withtarget => client, we would haveclient.geo.*,client.as.*, etc.).Motivation:
Clearly define destinations for all GeoIP Filter fields as a sub-field of a single target, so that users can enable ECS Compatibility Mode without losing metadata that they currently rely on.
Specifically, we have six fields that do not have direct analogues in ECS:
timezone: the IANA name of the timezone e.g., America/New_Yorkpostal_code: a string postal code, length varies by countrycontinent_code: "A two character continent code like "NA" (North America) or "OC" (Oceania)"organization: the name of the business or ISP associated with an address, reportedly available for ~40% of lookups.isp: the name of the ISPdma_code: us-only code representing Designated Market Area (~metro area)Detailed Design:
geo.timezone(IANA name, presently up to 30 characters, e.g.,America/Argentina/Buenos_Aires),geo.postal_code(freeform but relatively space limited, regulated by each country; see wikipedia),geo.continent_code(docs defineAF,AN,AS,EU,NA,OC, andSA)targetthat is the parent of the relatedgeoandasfields.mmdbsub-key? e.g,${target}.mmdb.organizationmmdbbeing used may or may not be provided by Maxmind, I'd like to avoid explicitly using "maxmind" in the key namedma_code, while provided in mmdb, is a US-only Nielsen Ratings construct and may not be adequately described with ammdbprefix.Related: logstash-plugins/logstash-filter-geoip#163