[Heartbeat] Set HTTP user-agent in Heartbeat by andrewvc · Pull Request #14297 · elastic/beats

andrewvc · 2019-10-29T00:42:45Z

By default heartbeat uses the golang user agent, which is blacklisted by a number of services, including elastic.co. URLs like https://www.elastic.co/products/beats/heartbeat do not work with the
default go user agent due to such blacklists.

This changes the default user agent to be 'Elastic Heartbeat/VERSION (PLATFORM INFO)'

Resolves #10170 (comment)

elasticmachine · 2019-10-29T00:42:47Z

Pinging @elastic/uptime (:uptime)

By default heartbeat uses the golang user agent, which is blacklisted by a number of services, including elastic.co. URLs like https://www.elastic.co/products/beats/heartbeat do not work with the default go user agent due to such blacklists. This changes the default user agent to be 'elastic_heartbeat'. While the HTTP spec allows UAs to add version numbers to the end this PR does not add the version number for two reasons: 1. Adding the version wouldn't practically be of use to anyone as one UA version numbers for an uptime check aren't practically useful. 2. It would needlessly add complexity to this commit. Resolves elastic#10170 (comment)

andrewkroh · 2019-10-29T00:59:09Z

We set the User-Agent in Filebeat for the client that connects to Google Pub/Sub and in the client that connects to MISP. It would be good to align all of these to use the same exact format.

I recommend creating a util in libbeat to get the User-Agent string and make each of these use that.

beats/x-pack/filebeat/input/googlepubsub/input.go

Lines 180 to 184 in 6b6408f

    
           func userAgent() string { 
        
           	return fmt.Sprintf("Elastic Filebeat/%s (%s; %s; %s; %s)", 
        
           		version.GetDefaultVersion(), runtime.GOOS, runtime.GOARCH, 
        
           		version.Commit(), version.BuildTime()) 
        
           }

andrewvc · 2019-10-29T02:11:12Z

Love the util idea! I'm glad to do that in this PR.

…

On Mon, Oct 28, 2019, 7:59 PM Andrew Kroh ***@***.***> wrote: We set the User-Agent in Filebeat for the client that connects to Google Pub/Sub and in the client that connects to MISP. It would be good to align all of these to use the same exact format. I recommend create a util in libbeat to get the User-Agent string and make each of these use that. https://github.com/elastic/beats/blob/6b6408f843f61e6169e1a7d9eec2acb9c92cd95c/x-pack/filebeat/input/googlepubsub/input.go#L180-L184 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#14297?email_source=notifications&email_token=AABACY6E76FCIA3QKB7GG5TQQ6DHXA5CNFSM4JGB4JW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECO4EXQ#issuecomment-547209822>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABACYYKXY4GXK2AE3KNDB3QQ6DHXANCNFSM4JGB4JWQ> .

ruflin

This is great, +1 on the util approach. Do you plan to handle it in this PR or a follow up?

Probably this info should also show up somewhere in the heartbeat docs ...

andrewvc · 2019-10-29T13:52:01Z

@ruflin I'll do the util thing in this PR. ++ on adding docs.

andrewvc · 2019-10-29T16:31:43Z

@ruflin @andrewkroh I've updated this PR to make a common helper function

andrewkroh

LGTM. But there's one more to update at https://github.com/elastic/beats/blob/master/x-pack/filebeat/input/httpjson/input.go#L325.

andrewkroh · 2019-10-29T16:33:55Z

x-pack/filebeat/input/googlepubsub/input.go

 	"sync"
 	"time"

+	"github.com/elastic/beats/libbeat/common/useragent"


Move this down to the grouping of github.com/elastic imports.

ph · 2019-10-29T16:37:33Z

heartbeat/monitors/active/http/task.go

+	}

 	if enc != nil {
 		enc.AddHeaders(&request.Header)


concerning the implementation, you might want to take a look at https://github.com/elastic/beats/blob/fleet/agent/kibana/round_trippers.go#L33-L52 and use http.Roundtripper interface.

@andrewvc ^

What's the advantage of adding it there vs. here? Seems like they're fairly equivalent in this case.

There are, but you make sure that all connection through the http client get the headers no matter what.

But, you are right I am not familliar enough with hearthbeat client. so it might be equivalent in that case.

I think in our case it's equivalent since this is the only entry point through to the client. I think I'll leave the code here so there's only one spot where header stuff happens, which at least to me, seems easier to maintain, but I agree with you, generally speaking if you share the HTTP client it should go in the round tripper.

andrewvc · 2019-10-29T18:19:24Z

@andrewkroh I've added it to the httpjson module as well.

@ruflin I think this is ready for a final review.

ruflin

PR LGTM. I still think we should have this in our docs somewhere.

andrewvc · 2019-10-30T20:11:01Z

@ruflin I just added the docs for this. If it's still good let me know with another LGTM.

ruflin · 2019-10-31T12:27:57Z

heartbeat/docs/heartbeat-options.asciidoc

 *`method`*:: The HTTP method to use. Valid values are `"HEAD"`, `"GET"` and
 `"POST"`.
-*`headers`*:: A dictionary of additional HTTP headers to send.
+*`headers`*:: A dictionary of additional HTTP headers to send. By default heartbeat


Nit: It would be nice to have a header example posted here. So if someone configures a tool where he needs to filter on the headers, he can just copy-paste it here (or at least have the pattern).

ruflin · 2019-10-31T12:29:23Z

jenkins, test this

ruflin

The code LGTM, but CI ...

ruflin · 2019-10-31T12:35:57Z

x-pack/filebeat/input/httpjson/input.go

 	"sync"
 	"time"

+	"github.com/elastic/beats/libbeat/common/useragent"


Nit: Guess what ;-)

Argh, fixed :)

andrewvc · 2019-10-31T17:39:32Z

Build failures unrelated

By default heartbeat uses the golang user agent, which is blacklisted by a number of services, including elastic.co. URLs like https://www.elastic.co/products/beats/heartbeat do not work with the default go user agent due to such blacklists. This changes the default user agent to be 'Elastic Heartbeat/VERSION (PLATFORM INFO)' Resolves elastic#10170 (comment) (cherry picked from commit 956f87c)

OrangeDog · 2019-11-25T12:45:58Z

This change is probably incorrect.
Elastic Heartbeat/VERSION is two separate products: Elastic and Heartbeat/VERSION.
If the space separator was intentional, the primary identifier (Heartbeat) should still go first.

A better User-Agent would be something like Heartbeat/VERSION libbeat/VERSION Go-http-client/1.1

andrewvc · 2019-11-25T15:39:10Z

Thanks @OrangeDog , I've opened an issue to track this here #14747

By default heartbeat uses the golang user agent, which is blacklisted by a number of services, including elastic.co. URLs like https://www.elastic.co/products/beats/heartbeat do not work with the default go user agent due to such blacklists. This changes the default user agent to be 'Elastic Heartbeat/VERSION (PLATFORM INFO)' Resolves elastic#10170 (comment)

andrewvc added enhancement Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team labels Oct 29, 2019

andrewvc requested a review from ruflin October 29, 2019 00:42

andrewvc requested a review from a team as a code owner October 29, 2019 00:42

andrewvc self-assigned this Oct 29, 2019

andrewvc force-pushed the default-ua branch from 11013f2 to a0ed054 Compare October 29, 2019 00:44

andrewvc added [zube]: In Review review labels Oct 29, 2019

andrewvc changed the title ~~[Heartbeat] Set HTTP user-agent to 'elastic_heatbeat'~~ [Heartbeat] Set HTTP user-agent to 'elastic_heartbeat' Oct 29, 2019

andrewvc force-pushed the default-ua branch from a0ed054 to 6caa091 Compare October 29, 2019 00:46

ruflin reviewed Oct 29, 2019

View reviewed changes

andrewvc added 3 commits October 29, 2019 11:30

Use common helper

bbb2056

Merge remote-tracking branch 'origin/master' into default-ua

cd17ae5

Update changelog

f20b2f7

andrewkroh approved these changes Oct 29, 2019

View reviewed changes

Fix imports

3773d4d

ph reviewed Oct 29, 2019

View reviewed changes

andrewvc changed the title ~~[Heartbeat] Set HTTP user-agent to 'elastic_heartbeat'~~ [Heartbeat] Set HTTP user-agent in Heartbeat Oct 29, 2019

Added filebeat user agent to httpjson

d196690

andrewvc added the Filebeat Filebeat label Oct 29, 2019

ruflin approved these changes Oct 30, 2019

View reviewed changes

Document User-Agent header

7295fb5

ruflin reviewed Oct 31, 2019

View reviewed changes

andrewvc added 2 commits October 31, 2019 08:41

Fix imports

5908d66

Fix formatting

50a378f

andrewvc merged commit 956f87c into elastic:master Oct 31, 2019

andrewvc deleted the default-ua branch October 31, 2019 17:40

andrewvc mentioned this pull request Nov 25, 2019

Beats HTTP user agent does not follow RFC #14747

Closed

andrewvc mentioned this pull request Nov 26, 2019

[Heartbeat] Document the user agent sent by heartbeat #14805

Merged

Conversation

andrewvc commented Oct 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Oct 29, 2019

Uh oh!

andrewkroh commented Oct 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewvc commented Oct 29, 2019 via email

Uh oh!

ruflin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewvc commented Oct 29, 2019

Uh oh!

andrewvc commented Oct 29, 2019

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewvc commented Oct 29, 2019

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

andrewvc commented Oct 30, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruflin commented Oct 31, 2019

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewvc commented Oct 31, 2019

Uh oh!

OrangeDog commented Nov 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewvc commented Nov 25, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

andrewvc commented Oct 29, 2019 •

edited

Loading

andrewkroh commented Oct 29, 2019 •

edited

Loading

ruflin left a comment •

edited

Loading

OrangeDog commented Nov 25, 2019 •

edited

Loading