[Filebeat] Add Base64 encoded HMAC & UUID template functions to httpjson input by legoguy1000 · Pull Request #27873 · elastic/beats

legoguy1000 · 2021-09-10T18:39:49Z

What does this PR do?

Add Base64 encoded HMAC & UUID template functions to httpjson input

Why is it important?

In order to create an Akamai Agent integration.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Relates Akamai integrations#1304

Use cases

Screenshots

Logs

elasticmachine · 2021-09-10T18:43:50Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-09-20T09:21:34.833+0000
Duration: 89 min 41 sec
Commit: c08213a

Test stats 🧪

Test	Results
Failed	0
Passed	8206
Skipped	1201
Total	9407

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	8206
Skipped	1201
Total	9407

elasticmachine · 2021-09-13T16:54:51Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

mergify · 2021-09-14T09:15:07Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b akamai-edgegrid upstream/akamai-edgegrid
git merge upstream/master
git push upstream akamai-edgegrid

P1llus · 2021-09-14T09:47:31Z

Hi @legoguy1000 , thanks for your awesome PR's as always! I will discuss this one, as we might not want to use any additional libraries for this, and rather add in the functionality it requires to httpjson itself, it might be that our recent changes we made would work with this actually

legoguy1000 · 2021-09-14T18:39:23Z

I think the alternative to adding the EdgeGrid Library is to add another HMAC function that returns Base64 encoded signatures instead of Hex. See f746b13

Then do below to implement EdgeGrid and possibly remove the EdgeGrid Module

- type: httpjson
  enabled: true
  config_version: 2
  interval: 1m
  request.url: XXXXXX
  request.transforms:
    - set:
        target: url.params.from
        value: '[[.cursor.timestamp]]'
        default: '[[ (now (parseDuration "-24h")).Unix ]]'
    - set:
        target: header.XTimestamp
        value: '[[ formatDate (now) "20060102T15:04:05-0700" ]]'
    - set:
        target: header.XSignatureBase
        value: '[[ sprintf "EG1-HMAC-SHA256 client_token=%s;access_token=%s;timestamp=%s;nonce=%d;" "a" "b" (.header.Get "XTimestamp") now.UnixNano  ]]'
    - set:
        target: header.XSignatureKey
        value: '[[ hmacBase64 "sha256" "c" (.header.Get "XTimestamp") ]]'
    - set:
        target: header.XSignature
        value: '[[ hmacBase64 "sha256" (.header.Get "XSignatureKey") "GET\t" .url.Host "\t" .url.Path "?" .url.RawQuery "\t\t\t" (.header.Get "XSignatureBase") ]]'
    - set:
        target: header.Authorization
        value: 'EG1-HMAC-SHA256 [[ sprintf "%ssignature=%s" (.header.Get "XSignatureBase") (.header.Get "XSignature") ]]'

It appears to work and generates the below headers

xtimestamp	20210914T18:33:08+0000
xsignaturekey	L/1kX9gfY7GIFly7G3z3vkj3YUzFWpCHqqDR7HimwpM=
xsignaturebase	EG1-HMAC-SHA256 client_token=a;access_token=b;timestamp=20210914T18:33:08+0000;nonce=1631644388219193225;
xsignature	i4Ln1LOulRMhwJ8v8vAa5s+XfqcQ2fCkirJYHuGzLSQ=
authorization	EG1-HMAC-SHA256 EG1-HMAC-SHA256 client_token=a;access_token=b;timestamp=20210914T18:33:08+0000;nonce=1631644388219193225;signature=i4Ln1LOulRMhwJ8v8vAa5s+XfqcQ2fCkirJYHuGzLSQ=

legoguy1000 · 2021-09-15T11:02:14Z

I think I'm going to also add a UUID template function for the nonce field as there is no way currently to generate randomness.

legoguy1000 · 2021-09-17T11:50:02Z

@marc-gr @P1llus I think we're good to go now.

marc-gr

LGTM 👍 thanks for taking the time to change it with our feedback 😄 !

marc-gr · 2021-09-20T07:50:06Z

/test

marc-gr · 2021-09-20T08:39:09Z

/test

…son input (#27873) * Init commit * Update docs and tests * Add Base64 HMAC * Remove EdgeGrid Library * Add UUID and Base64 HMAC template functions * don't upgrade google/uuid library * Format value_tpl.go Co-authored-by: Marc Guasch <marc.guasch@elastic.co> (cherry picked from commit 942d537)

…son input (#27873) (#28001) * Init commit * Update docs and tests * Add Base64 HMAC * Remove EdgeGrid Library * Add UUID and Base64 HMAC template functions * don't upgrade google/uuid library * Format value_tpl.go Co-authored-by: Marc Guasch <marc.guasch@elastic.co> (cherry picked from commit 942d537) Co-authored-by: Alex Resnick <adr8292@gmail.com>

* upstream/master: (658 commits) Add complete k8s metadata through composable provider (elastic#27691) Revert "Fix issue where --insecure didn't propogate to Fleet Server ES connection (elastic#27969)" (elastic#27997) Remove deprecated kafka fields (elastic#27938) [Filebeat] Add Base64 encoded HMAC & UUID template functions to httpjson input (elastic#27873) Improve httpjson template function join (elastic#27996) Remove kubernetes.container.image alias (elastic#27898) [Elastic Agent] Golden files for program tests (elastic#27862) [Elastic Agent] Disable modules.d in metricbeat (elastic#27860) libbeat/common/seccomp: provide default policy for linux arm64 (elastic#27955) Fix logger statement in aws-s3 input (elastic#27982) Fix wrong merge (elastic#27976) Fix issue where --insecure didn't propogate to Fleet Server ES connection (elastic#27969) Forward-port 7.14.2 changelog to master (elastic#27975) [Filebeat] Removing duplicate modules (aliases) Observability (elastic#27919) Fix path in vagrant windows script (elastic#27966) [Filebeat] Removing duplicate modules (aliases) and Cyberark (elastic#27915) No changelog for 8.0.0-alpha2 (elastic#27961) Add write access to 'url.value' from 'request.transforms'. (elastic#27937) Docker: remove deprecated fields (elastic#27933) Filebeat: Make all filesets disabled in default configuration (elastic#27762) ...

…son input (elastic#27873) * Init commit * Update docs and tests * Add Base64 HMAC * Remove EdgeGrid Library * Add UUID and Base64 HMAC template functions * don't upgrade google/uuid library * Format value_tpl.go Co-authored-by: Marc Guasch <marc.guasch@elastic.co>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 10, 2021

legoguy1000 marked this pull request as ready for review September 10, 2021 18:41

legoguy1000 changed the title ~~Init commit~~ [Filebeat] Add Akamai EdgeGrid auth to httpjson input Sep 11, 2021

This was referenced Sep 11, 2021

Akamai elastic/integrations#1304

Closed

[Akamai] Add Akamai package elastic/integrations#1643

Merged

jsoriano added the Team:Security-External Integrations label Sep 13, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 13, 2021

P1llus assigned marc-gr Sep 14, 2021

legoguy1000 force-pushed the akamai-edgegrid branch from 52eb2d6 to ab8af73 Compare September 15, 2021 13:06

legoguy1000 changed the title ~~[Filebeat] Add Akamai EdgeGrid auth to httpjson input~~ [Filebeat] Add Base64 encoded HMAC & UUID template functions to httpjson input Sep 17, 2021

legoguy1000 added 5 commits September 17, 2021 11:45

Init commit

95ba8f2

Update docs and tests

ac14e79

Add Base64 HMAC

7efbbf4

Remove EdgeGrid Library

061f019

Add UUID and Base64 HMAC template functions

8d1f1f2

legoguy1000 force-pushed the akamai-edgegrid branch 2 times, most recently from 3b36486 to 8d1f1f2 Compare September 17, 2021 11:47

don't upgrade google/uuid library

f1b0d87

marc-gr approved these changes Sep 20, 2021

View reviewed changes

Format value_tpl.go

c08213a

marc-gr merged commit 942d537 into elastic:master Sep 20, 2021

marc-gr added backport-v7.16.0 Automated backport with mergify enhancement labels Sep 20, 2021

mergify bot mentioned this pull request Sep 20, 2021

[7.x](backport #27873) [Filebeat] Add Base64 encoded HMAC & UUID template functions to httpjson input #28001

Merged

legoguy1000 deleted the akamai-edgegrid branch September 20, 2021 11:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Add Base64 encoded HMAC & UUID template functions to httpjson input#27873

[Filebeat] Add Base64 encoded HMAC & UUID template functions to httpjson input#27873
marc-gr merged 7 commits intoelastic:masterfrom
legoguy1000:akamai-edgegrid

legoguy1000 commented Sep 10, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Sep 10, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

Uh oh!

elasticmachine commented Sep 13, 2021

Uh oh!

mergify bot commented Sep 14, 2021

Uh oh!

P1llus commented Sep 14, 2021

Uh oh!

legoguy1000 commented Sep 14, 2021 •

edited

Loading

Uh oh!

legoguy1000 commented Sep 15, 2021

Uh oh!

legoguy1000 commented Sep 17, 2021

Uh oh!

marc-gr left a comment

Uh oh!

marc-gr commented Sep 20, 2021

Uh oh!

marc-gr commented Sep 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

legoguy1000 commented Sep 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

elasticmachine commented Sep 10, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

elasticmachine commented Sep 13, 2021

Uh oh!

mergify bot commented Sep 14, 2021

Uh oh!

P1llus commented Sep 14, 2021

Uh oh!

legoguy1000 commented Sep 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

legoguy1000 commented Sep 15, 2021

Uh oh!

legoguy1000 commented Sep 17, 2021

Uh oh!

marc-gr left a comment

Choose a reason for hiding this comment

Uh oh!

marc-gr commented Sep 20, 2021

Uh oh!

marc-gr commented Sep 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

legoguy1000 commented Sep 10, 2021 •

edited

Loading

elasticmachine commented Sep 10, 2021 •

edited by jenkins-beats-ci bot

Loading

legoguy1000 commented Sep 14, 2021 •

edited

Loading