Add third-party REST API by leehinman · Pull Request #545 · elastic/integrations

leehinman · 2021-01-20T16:26:54Z

What does this PR do?

Adds third-party REST API to several packages. This allows pulling data from Splunk REST API.

Apache
nginx
AWS Cloudtrail
Zeek

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all datasets collect metrics or logs.

Screenshots

Add Apache Integration Screen

Third-Party REST API Configuration

Example ingested document

{
  "_index": "filebeat-8.0.0-2021.01.11-000001",
  "_type": "_doc",
  "_id": "d9a88109c64ba572d8319e3325901415ff78f6b6fae8346d38c28c98010ee0b0",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "name": "Lees-MacBook-Pro.local",
      "id": "7b45f55c-a54b-4f81-a8e5-8b4033d14305",
      "type": "filebeat",
      "ephemeral_id": "3170de0e-eb99-4784-b17e-eb4292443378",
      "version": "8.0.0"
    },
    "source": {
      "address": "127.0.0.1",
      "ip": "127.0.0.1"
    },
    "fileset": {
      "name": "access"
    },
    "url": {
      "original": "/"
    },
    "tags": [
      "forwarded"
    ],
    "input": {
      "type": "httpjson"
    },
    "file": {
      "path": "/var/log/apache2/access.log"
    },
    "apache": {
      "access": {}
    },
    "@timestamp": "2021-01-12T17:45:52.000Z",
    "ecs": {
      "version": "1.7.0"
    },
    "service": {
      "type": "apache"
    },
    "host": {
      "name": "ubuntu-bionic"
    },
    "http": {
      "request": {
        "referrer": "-",
        "method": "GET"
      },
      "response": {
        "status_code": 200,
        "body": {
          "bytes": 11173
        }
      },
      "version": "1.1"
    },
    "event": {
      "ingested": "2021-01-12T17:46:01.271439318Z",
      "kind": "event",
      "created": "2021-01-12T17:46:00.241Z",
      "module": "apache",
      "category": "web",
      "dataset": "apache.access",
      "outcome": "success"
    },
    "user": {
      "name": "-"
    },
    "user_agent": {
      "original": "curl/7.58.0",
      "name": "curl",
      "device": {
        "name": "Other"
      },
      "version": "7.58.0"
    }
  },
  "fields": {
    "event.ingested": [
      "2021-01-12T17:46:01.271Z"
    ],
    "@timestamp": [
      "2021-01-12T17:45:52.000Z"
    ],
    "event.created": [
      "2021-01-12T17:46:00.241Z"
    ]
  },
  "sort": [
    1610473552000
  ]
}

elasticmachine · 2021-02-05T16:27:40Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #545 updated
Start Time: 2021-03-10T21:13:49.730+0000
Duration: 32 min 23 sec
Commit: ce66995

Test stats 🧪

Test	Results
Failed	0
Passed	445
Skipped	0
Total	445

Trends 🧪

elasticmachine · 2021-02-25T16:24:31Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh

I would really like to see some tests that exercise the httpjson part of the config. This is the idea I had when I was thinking about how to test some other packages that use httpjson (like google workspace and okta): https://github.com/andrewkroh/stream/issues/9

andrewkroh · 2021-03-02T20:13:09Z

packages/apache/data_stream/access/agent/stream/httpjson.yml.hbs

This looks misspelled.

andrewkroh · 2021-03-02T20:17:03Z

packages/apache/manifest.yml

There are quite a few possible SSL options available. I suggest hiding the SSL options under an advanced setting and letting users configure it as YAML. Passing PEM encoded CA certs too is a little easier IMO with the textbox it gives you.

integrations/packages/zoom/data_stream/webhook/manifest.yml

Lines 40 to 50 in 8770467

- name: ssl

type: yaml

title: TLS

description: Options for enabling TLS for the listening webhook endpoint. Zoom requires webhooks listen on HTTPS. You must either provide a valid TLS certificate or use a reverse proxy in front of the integration. See the [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html) for a list of all options.

multi: false

required: false

show_user: false

default: |

enabled: false

certificate: "/etc/pki/client/cert.pem"

key: "/etc/pki/client/cert.key"

Done, see if the screenshot is what you were thinking

andrewkroh · 2021-03-02T21:37:27Z

packages/apache/data_stream/access/manifest.yml

We should probably mark this with "(experimental)". Once we have a few more necessary package/fleet features we'll be able to implement this an independent package and remove these.

- apache - nginx - zeek - aws/cloudtrail

This reverts commit 399e8b0.

andresrc added the Team:Security-External Integrations label Jan 22, 2021

leehinman force-pushed the apache_httpjson_poc branch from 687f5b7 to a04e10d Compare February 5, 2021 16:18

leehinman changed the title ~~httpjson for apache access and error~~ Add third-party REST API Feb 5, 2021

leehinman force-pushed the apache_httpjson_poc branch from a04e10d to cc7d8c2 Compare February 9, 2021 20:33

leehinman force-pushed the apache_httpjson_poc branch 4 times, most recently from d32aa06 to faa2773 Compare February 25, 2021 16:15

leehinman marked this pull request as ready for review February 25, 2021 16:24

andrewkroh reviewed Mar 2, 2021

View reviewed changes

leehinman mentioned this pull request Mar 8, 2021

[Change Proposal] support specifiying alternate data_stream.dataset elastic/package-spec#145

Open

leehinman added 2 commits March 10, 2021 15:13

Add third-party REST API

e7ab467

- apache - nginx - zeek - aws/cloudtrail

Incorporate feedback

ce66995

leehinman force-pushed the apache_httpjson_poc branch from faa2773 to ce66995 Compare March 10, 2021 21:13

andrewkroh approved these changes Mar 11, 2021

View reviewed changes

leehinman merged commit 399e8b0 into elastic:master Mar 11, 2021

leehinman added a commit that referenced this pull request Mar 11, 2021

Revert "Add third-party REST API (#545)"

d1d716d

This reverts commit 399e8b0.

leehinman mentioned this pull request Mar 11, 2021

Revert "Add third-party REST API" #789

Closed

leehinman deleted the apache_httpjson_poc branch September 28, 2021 20:22

herrBez mentioned this pull request Nov 6, 2025

Consider Removing third-party pipeline invokation and definition from Nginx, Apache, AWS CloudTrail, Zeek integration #15881

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add third-party REST API#545

Add third-party REST API#545
leehinman merged 2 commits intoelastic:masterfrom
leehinman:apache_httpjson_poc

leehinman commented Jan 20, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Feb 5, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

Uh oh!

elasticmachine commented Feb 25, 2021

Uh oh!

andrewkroh left a comment

Uh oh!

andrewkroh Mar 2, 2021

Uh oh!

andrewkroh Mar 2, 2021

Uh oh!

leehinman Mar 10, 2021

Uh oh!

andrewkroh Mar 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	- name: ssl
	type: yaml
	title: TLS
	description: Options for enabling TLS for the listening webhook endpoint. Zoom requires webhooks listen on HTTPS. You must either provide a valid TLS certificate or use a reverse proxy in front of the integration. See the [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html) for a list of all options.
	multi: false
	required: false
	show_user: false
	default: \|
	enabled: false
	certificate: "/etc/pki/client/cert.pem"
	key: "/etc/pki/client/cert.key"

Conversation

leehinman commented Jan 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Screenshots

Add Apache Integration Screen

Third-Party REST API Configuration

Example ingested document

Uh oh!

elasticmachine commented Feb 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

Uh oh!

elasticmachine commented Feb 25, 2021

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

andrewkroh Mar 2, 2021

Choose a reason for hiding this comment

Uh oh!

andrewkroh Mar 2, 2021

Choose a reason for hiding this comment

Uh oh!

leehinman Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

andrewkroh Mar 2, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

leehinman commented Jan 20, 2021 •

edited

Loading

elasticmachine commented Feb 5, 2021 •

edited

Loading