Skip to content

Add Facility for deploying ElasticSearch Transform #23

@nnamdifrankie

Description

@nnamdifrankie

Background

And the ability to deploy a defined ElasticSearch transform to be be deployed when a package is applied or upgraded.

Acceptance Criteria

  • As a user I should be able to define an ElasticSearch transform as part of a package.
  • The ElasticSearch transform should be started after it is added to the search database.
  • As a user I should be able to update the attributes of the transform including the name possibly. This should not result in two Transform running.
  • As a user we should be able to delete a transform through an update to the package.
  • As a user I should be able to view statistics and information about a transform after deployment using the ElasticSearch API or Kibana if available.

Sample Transform Creation Statements Captured From Kibana Devtools

PUT _transform/endpoint_host_metadata_transform
{
  "source": {
    "index": "metrics-endpoint.metadata-default"
  },
  "dest": {
    "index": "metrics-endpoint.metadata_current-default"
  },
  "pivot": {
    "group_by": {
      "agent.id": {
        "terms": {
          "field": "agent.id"
        }
      }
    },
    "aggregations": {
      "HostDetails": {
        "scripted_metric": {
          "init_script": "state.timestamp_latest = 0L; state.last_doc=''",
          "map_script": "def current_date = doc['@timestamp'].getValue().toInstant().toEpochMilli(); if (current_date > state.timestamp_latest) {state.timestamp_latest = current_date;state.last_doc = new HashMap(params['_source']);}",
          "combine_script": "return state",
          "reduce_script": "def last_doc = '';def timestamp_latest = 0L; for (s in states) {if (s.timestamp_latest > (timestamp_latest)) {timestamp_latest = s.timestamp_latest; last_doc = s.last_doc;}} return last_doc"
        }
      }
    }
  },
  "description": "collapse and update the latest document for each host",
  "frequency": "1m",
  "sync": {
    "time": {
      "field": "event.created",
      "delay": "60s"
    }
  }
}

POST _transform/endpoint_host_metadata_transform/_start
DELETE _transform/endpoint_host_metadata_transform

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions