Skip to content

Fix prometheus cadvisor authentication#9497

Merged
helenosheaa merged 3 commits intoinfluxdata:masterfrom
imranismail:patch-1
Jul 20, 2021
Merged

Fix prometheus cadvisor authentication#9497
helenosheaa merged 3 commits intoinfluxdata:masterfrom
imranismail:patch-1

Conversation

@imranismail
Copy link
Copy Markdown
Contributor

@imranismail imranismail commented Jul 13, 2021 •
edited
Loading

Building upon #8762, the kubelet API requires auth if --anonymous-auth flag is set to false for kubelet (which is the case for EKS).

Enabling that flag is a huge security risk

Closes #9408 #9349

Required for all PRs:

  • Updated associated README.md.
  • Wrote appropriate unit tests.

Before

$ curl -v -k "https://10.38.177.113:10250/pods"
*   Trying 10.38.177.113:10250...
* Connected to 10.38.177.113 (10.38.177.113) port 10250 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):    
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): 
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=system:nodes; CN=system:node:ip-10-38-177-113.ap-southeast-1.compute.internal
*  start date: Jul  8 06:58:00 2021 GMT
*  expire date: Jul  8 06:58:00 2022 GMT
*  issuer: CN=kubernetes
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f65d1620e0)
> GET /pods HTTP/2
> Host: 10.38.177.113:10250
> user-agent: curl/7.77.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 401
< content-type: text/plain; charset=utf-8
< content-length: 12
< date: Tue, 13 Jul 2021 19:21:15 GMT
<
* Connection #0 to host 10.38.177.113 left intact

After

$ export TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
$ curl -v -k -H "Authorization: Bearer $TOKEN" "https://10.38.177.113:10250/pods"
*   Trying 10.38.177.113:10250...
* Connected to 10.38.177.113 (10.38.177.113) port 10250 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):    
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): 
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=system:nodes; CN=system:node:ip-10-38-177-113.ap-southeast-1.compute.internal
*  start date: Jul  8 06:58:00 2021 GMT
*  expire date: Jul  8 06:58:00 2022 GMT
*  issuer: CN=kubernetes
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f84ff27500)
> GET /pods HTTP/2
> Host: 10.38.177.113:10250
> user-agent: curl/7.77.0
> accept: */*
> authorization: Bearer <redacted>
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< content-type: application/json
< date: Tue, 13 Jul 2021 19:19:37 GMT
<
<redacted>
* Connection #0 to host 10.38.177.113 left intact

@telegraf-tiger
Copy link
Copy Markdown
Contributor

telegraf-tiger bot commented Jul 13, 2021

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

@telegraf-tiger telegraf-tiger bot added the fix pr to fix corresponding bug label Jul 13, 2021
@imranismail
Copy link
Copy Markdown
Contributor Author

imranismail commented Jul 13, 2021

!signed-cla

@telegraf-tiger
Copy link
Copy Markdown
Contributor

telegraf-tiger bot commented Jul 13, 2021

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

This was referenced Jul 14, 2021
Closed
Closed
@srebhan
Copy link
Copy Markdown
Member

srebhan commented Jul 14, 2021

Seems to also resolve #9408.

@srebhan srebhan self-assigned this Jul 14, 2021
@srebhan srebhan added the regression something that used to work, but is now broken label Jul 14, 2021
srebhan
srebhan approved these changes Jul 14, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks for fixing this @imranismail!

@srebhan srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Jul 14, 2021
@srebhan
Copy link
Copy Markdown
Member

srebhan commented Jul 14, 2021

@imranismail can you please change your PR description such that it includes a line

resolves #9408

so that the issue is automatically closed when this is merged.

@imranismail
Copy link
Copy Markdown
Contributor Author

imranismail commented Jul 14, 2021

@srebhan, added those keywords

@helenosheaa helenosheaa merged commit 8965291 into influxdata:master Jul 20, 2021
reimda pushed a commit that referenced this pull request Jul 28, 2021
@imranismail @reimda
Fix prometheus cadvisor authentication (#9497)
03ff37b 
(cherry picked from commit 8965291)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srebhan srebhan srebhan approved these changes

Assignees

@srebhan srebhan

Labels

fix pr to fix corresponding bug ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. regression something that used to work, but is now broken

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

prometheus input plugin: mistaken delete

3 participants

@imranismail @srebhan @helenosheaa