Skip to content

Introduce apm-server.auth.* config#5457

Merged
axw merged 5 commits intoelastic:masterfrom
axw:beater-config-auth
Jun 17, 2021
Merged

Introduce apm-server.auth.* config#5457
axw merged 5 commits intoelastic:masterfrom
axw:beater-config-auth

Conversation

@axw
Copy link
Copy Markdown
Member

@axw axw commented Jun 15, 2021

Motivation/summary

Introduce the new AgentAuth config structure, which holds API Key and secret token auth. Later we will add "anonymous" auth here too, and deprecate/replace some RUM config (rate limiting and allowed service names).

We also introduce a new YAML naming scheme for the config, apm-server.auth.*. The old config is deprecated and copied across to the new config fields.

Checklist

How to test these changes

  1. Set apm-server.secret_token and apm-server.api_key.*, make sure they are honoured (e.g. query "GET /" with/out auth)
  2. Set apm-server.auth.secret_token and apm-server.auth.api_key.*, same again.
  3. Set both new and old with different values: check that a warning is logged that the old config is ignored, and check that it is possible to auth using the new but not old

Related issues

#5347

@axw axw force-pushed the beater-config-auth branch from 8647dda to bbf9049 Compare June 15, 2021 10:06
@ghost
Copy link
Copy Markdown

ghost commented Jun 15, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #5457 updated

  • Start Time: 2021-06-17T02:01:44.290+0000

  • Duration: 42 min 30 sec

  • Commit: e0cc895

Test stats 🧪

Test Results
Failed 0
Passed 6114
Skipped 120
Total 6234

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 2

Expand to view the steps failures

Test Sync
  • Took 3 min 21 sec . View more details on here
  • Description: ./.ci/scripts/sync.sh
Build packages
  • Took 29 min 6 sec . View more details on here
  • Description: ./.ci/scripts/package.sh

Log output

Expand to view the last 100 lines of log output

[2021-06-17T02:37:26.368Z] === RUN   TestUnstartedAPMServer
[2021-06-17T02:37:26.368Z] --- PASS: TestUnstartedAPMServer (0.00s)
[2021-06-17T02:37:26.368Z] === RUN   TestAPMServerStartTLS
[2021-06-17T02:37:26.368Z] --- PASS: TestAPMServerStartTLS (0.11s)
[2021-06-17T02:37:26.368Z] === RUN   TestExpvar
[2021-06-17T02:37:26.368Z] --- PASS: TestExpvar (0.11s)
[2021-06-17T02:37:26.368Z] PASS
[2021-06-17T02:37:26.368Z] ok  	github.com/elastic/apm-server/systemtest/apmservertest	2.646s
[2021-06-17T02:37:26.368Z] ?   	github.com/elastic/apm-server/systemtest/benchtest	[no test files]
[2021-06-17T02:37:26.368Z] ?   	github.com/elastic/apm-server/systemtest/cmd/apmbench	[no test files]
[2021-06-17T02:37:26.368Z] ?   	github.com/elastic/apm-server/systemtest/estest	[no test files]
[2021-06-17T02:37:26.368Z] ?   	github.com/elastic/apm-server/systemtest/fleettest	[no test files]
[2021-06-17T02:37:26.368Z] + cleanup
[2021-06-17T02:37:26.368Z] + rm -rf /tmp/tmp.MWXM1RZ1mR
[2021-06-17T02:37:26.368Z] + .ci/scripts/docker-get-logs.sh
[2021-06-17T02:37:27.432Z] Post stage
[2021-06-17T02:37:27.444Z] Running in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5457/src/github.com/elastic/apm-server/build
[2021-06-17T02:37:27.466Z] Archiving artifacts
[2021-06-17T02:37:27.782Z] Recording test results
[2021-06-17T02:37:28.655Z] [Checks API] No suitable checks publisher found.
[2021-06-17T02:37:28.970Z] + tar --version
[2021-06-17T02:37:29.309Z] + tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
[2021-06-17T02:37:29.587Z] Archiving artifacts
[2021-06-17T02:37:30.094Z] Terminated
[2021-06-17T02:37:30.080Z] Terminated
[2021-06-17T02:40:54.892Z] [INFO] For detailed information see: https://apm-ci.elastic.co/job/apm-integration-tests-selector-mbp/job/master/17744/display/redirect
[2021-06-17T02:40:55.141Z] Copied 18 artifacts from "APM Integration Test MBP Selector » master" build number 17744
[2021-06-17T02:40:56.292Z] Post stage
[2021-06-17T02:40:56.302Z] Recording test results
[2021-06-17T02:40:56.957Z] [Checks API] No suitable checks publisher found.
[2021-06-17T02:41:03.358Z] >> package: Building apm-server type=zip for platform=windows/amd64
[2021-06-17T02:41:03.359Z] >> package: Building apm-server type=tar.gz for platform=linux/386
[2021-06-17T02:41:03.359Z] >> package: Building apm-server type=deb for platform=linux/amd64
[2021-06-17T02:41:03.359Z] >> package: Building apm-server-oss type=deb for platform=linux/amd64
[2021-06-17T02:41:03.359Z] >> package: Building apm-server-oss type=tar.gz for platform=darwin/amd64
[2021-06-17T02:41:05.898Z] >> package: Building apm-server type=tar.gz for platform=darwin/amd64
[2021-06-17T02:41:08.441Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:08.441Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:08.704Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:08.704Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:08.982Z] >> package: Building apm-server-oss type=tar.gz for platform=linux/386
[2021-06-17T02:41:14.286Z] >> package: Building apm-server-oss type=deb for platform=linux/386
[2021-06-17T02:41:15.241Z] >> package: Building apm-server-oss type=rpm for platform=linux/386
[2021-06-17T02:41:19.484Z] >> package: Building apm-server-oss type=docker for platform=linux/amd64
[2021-06-17T02:41:20.869Z] >> package: Building apm-server-oss type=rpm for platform=linux/amd64
[2021-06-17T02:41:21.445Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:21.445Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:24.016Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:24.016Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:30.675Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:30.675Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:30.675Z] >> package: Building apm-server-oss type=rpm for platform=linux/arm64
[2021-06-17T02:41:40.695Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:40.695Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:44.908Z] >> package: Building apm-server type=rpm for platform=linux/amd64
[2021-06-17T02:41:57.180Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:41:57.180Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:41:59.721Z] >> package: Building apm-server type=docker for platform=linux/amd64
[2021-06-17T02:42:06.305Z] >> package: Building apm-server type=docker for platform=linux/amd64
[2021-06-17T02:42:32.906Z] >> package: Building apm-server-oss type=tar.gz for platform=linux/arm64
[2021-06-17T02:42:39.517Z] >> package: Building apm-server-oss type=deb for platform=linux/arm64
[2021-06-17T02:42:47.699Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:42:47.699Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:42:58.010Z] >> package: Building apm-server type=tar.gz for platform=linux/amd64
[2021-06-17T02:43:01.394Z] >> package: Building apm-server type=rpm for platform=linux/arm64
[2021-06-17T02:43:09.566Z] >> package: Building apm-server type=tar.gz for platform=linux/arm64
[2021-06-17T02:43:11.526Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:43:11.527Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:43:16.889Z] >> package: Building apm-server type=deb for platform=linux/arm64
[2021-06-17T02:43:26.948Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:43:26.948Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:43:37.024Z] >> package: Building apm-server type=rpm for platform=linux/386
[2021-06-17T02:43:39.588Z] >> package: Building apm-server type=deb for platform=linux/386
[2021-06-17T02:43:42.121Z] >> package: Building apm-server-oss type=tar.gz for platform=linux/amd64
[2021-06-17T02:43:44.651Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:43:44.651Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:43:49.972Z] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[2021-06-17T02:43:49.972Z] Require just the needed backports instead, or 'backports/latest'.
[2021-06-17T02:43:49.972Z] >> package: Building apm-server type=zip for platform=windows/386
[2021-06-17T02:43:53.288Z] >> package: Building apm-server-oss type=zip for platform=windows/386
[2021-06-17T02:43:56.618Z] >> package: Building apm-server-oss type=zip for platform=windows/amd64
[2021-06-17T02:44:11.584Z] >> Testing package contents
[2021-06-17T02:44:11.584Z] # command-line-arguments
[2021-06-17T02:44:11.584Z] ../../../../pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20210614232151-2871d29be93a/dev-tools/packaging/package_test.go:39:2: missing go.sum entry for module providing package github.com/blakesmith/ar; to add:
[2021-06-17T02:44:11.584Z] 	go mod download github.com/blakesmith/ar
[2021-06-17T02:44:11.584Z] FAIL	command-line-arguments [setup failed]
[2021-06-17T02:44:11.584Z] FAIL
[2021-06-17T02:44:11.584Z] package ran for 28m2.983237017s
[2021-06-17T02:44:11.584Z] Error: running "go test /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5457/pkg/mod/github.com/elastic/beats/v7@v7.0.0-alpha2.0.20210614232151-2871d29be93a/dev-tools/packaging/package_test.go -files /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5457/src/github.com/elastic/apm-server/build/distributions/*" failed with exit code 1
[2021-06-17T02:44:11.584Z] Makefile:323: recipe for target 'release' failed
[2021-06-17T02:44:11.584Z] make: *** [release] Error 1
[2021-06-17T02:44:12.900Z] Stage "Publish" skipped due to earlier failure(s)
[2021-06-17T02:44:13.005Z] Failed in branch Package
[2021-06-17T02:44:13.405Z] Running on Jenkins in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-5457
[2021-06-17T02:44:13.458Z] [INFO] getVaultSecret: Getting secrets
[2021-06-17T02:44:13.519Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-17T02:44:14.467Z] + chmod 755 generate-build-data.sh
[2021-06-17T02:44:14.467Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5457/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5457/runs/13 FAILURE 2549912
[2021-06-17T02:44:14.467Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5457/runs/13/steps/?limit=10000 -o steps-info.json
[2021-06-17T02:44:14.717Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-5457/runs/13/tests/?status=FAILED -o tests-errors.json

@axw axw force-pushed the beater-config-auth branch 3 times, most recently from f43315f to 410009c Compare June 15, 2021 13:37
Copy link
Copy Markdown
Contributor

@simitt simitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great.

@axw axw force-pushed the beater-config-auth branch from 410009c to c8d6827 Compare June 16, 2021 03:27
Introduce the new AgentAuth config structure, which
holds API Key and secret token auth. Later we will
add "anonymous" auth here too.

We also introduce a new YAML naming scheme for the
config, `apm-server.auth.*`. The old config is
deprecated and copied across to the new config fields.
@axw axw force-pushed the beater-config-auth branch from c8d6827 to f273b34 Compare June 16, 2021 03:42
@axw axw force-pushed the beater-config-auth branch from f273b34 to 3257f5c Compare June 16, 2021 03:51
@axw axw requested a review from bmorelli25 June 16, 2021 03:51
@axw
Copy link
Copy Markdown
Member Author

axw commented Jun 16, 2021

@bmorelli25 would you please take a look at the docs changes? I've renamed some config, and added new deprecation sections for the deprecated config names.

I've also moved "api_key.* configuration options" into the "API keys" section, not sure if it was intentional that it was separate? I can move back if you prefer, but then I'm not sure where the deprecation section should go.

@axw axw marked this pull request as ready for review June 16, 2021 04:31
@axw axw requested a review from a team June 16, 2021 04:32
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Jun 16, 2021

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b beater-config-auth upstream/beater-config-auth
git merge upstream/master
git push upstream beater-config-auth

Copy link
Copy Markdown
Contributor

@simitt simitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also change the apmpackage please. I suggest we remove the deprecated settings and only support the new ones.

@axw
Copy link
Copy Markdown
Member Author

axw commented Jun 16, 2021

@simitt will do. I was going to wait for your changes (#5444) to land, but I'll just update now for the purposes of review and update again when that lands.

@axw axw requested a review from simitt June 16, 2021 07:28
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Jun 16, 2021

This pull request is now in conflicts. Could you fix it @axw? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b beater-config-auth upstream/beater-config-auth
git merge upstream/master
git push upstream beater-config-auth

Copy link
Copy Markdown
Member

@bmorelli25 bmorelli25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs look great–thanks!

It looks like there's one additional change that needs to be made to the command reference file. I'll fix that in the Beats repo and copy it over to apm-server before 7.14.

@axw axw added the v7.14.0 label Jun 17, 2021
@axw
Copy link
Copy Markdown
Member Author

axw commented Jun 17, 2021

Failure is related to the beats update, which will be resolved by #5471

@axw axw merged commit fc60576 into elastic:master Jun 17, 2021
@axw axw deleted the beater-config-auth branch June 17, 2021 03:17
mergify bot pushed a commit that referenced this pull request Jun 17, 2021
* Introduce `apm-server.auth.*` config

Introduce the new AgentAuth config structure, which
holds API Key and secret token auth. Later we will
add "anonymous" auth here too.

We also introduce a new YAML naming scheme for the
config, `apm-server.auth.*`. The old config is
deprecated and copied across to the new config fields.

* docs: update config names

* apmpackage: update auth config keys

(cherry picked from commit fc60576)

# Conflicts:
#	changelogs/head.asciidoc
axw added a commit that referenced this pull request Jun 17, 2021
* Introduce `apm-server.auth.*` config (#5457)

* Introduce `apm-server.auth.*` config

Introduce the new AgentAuth config structure, which
holds API Key and secret token auth. Later we will
add "anonymous" auth here too.

We also introduce a new YAML naming scheme for the
config, `apm-server.auth.*`. The old config is
deprecated and copied across to the new config fields.

* docs: update config names

* apmpackage: update auth config keys

(cherry picked from commit fc60576)

Co-authored-by: Andrew Wilkins <axw@elastic.co>
@simitt simitt self-assigned this Jul 9, 2021
mergify bot pushed a commit that referenced this pull request Jul 9, 2021
* Introduce `apm-server.auth.*` config

Introduce the new AgentAuth config structure, which
holds API Key and secret token auth. Later we will
add "anonymous" auth here too.

We also introduce a new YAML naming scheme for the
config, `apm-server.auth.*`. The old config is
deprecated and copied across to the new config fields.

* docs: update config names

* apmpackage: update auth config keys

(cherry picked from commit fc60576)

# Conflicts:
#	apmpackage/apm/agent/input/template.yml.hbs
#	beater/config/config.go
#	beater/jaeger/server.go
#	beater/processors.go
#	beater/server.go
#	changelogs/head.asciidoc
@simitt
Copy link
Copy Markdown
Contributor

simitt commented Jul 9, 2021

Tested with BC2:

  • old settings work as expected
  • new settings work as expected
  • API Key auth disabled when:
apm-server.api_key.enabled: true
apm-server.auth.api_key.enabled: false
  • Secret token 'abcd' expected
apm-server.secret_token: 'xxx'
apm-server.auth.secret_token: 'abcd'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants