[azure-eventhub] Support for AMQP-over-WebSocket transport in the processor v2#47956
[azure-eventhub] Support for AMQP-over-WebSocket transport in the processor v2#47956
Conversation
🤖 GitHub commentsJust comment with:
|
|
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
kaiyan-sheng
left a comment
There was a problem hiding this comment.
Overall it looks good!!
da7a45f to
359cd69
Compare
|
Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services) |
c6b56fe to
7e7f2f3
Compare
# Conflicts: # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go
aea539d to
92d4f4c
Compare
|
@Mergifyio backport 8.19 9.1 9.2 9.3 |
❌ No backport have been createdDetails
GitHub error: |
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # go.mod # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # go.mod # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go # Conflicts: # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy.
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go # Conflicts: # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go # Conflicts: # go.mod # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy.
…cessor v2 (#47956) (#48119) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>
…cessor v2 (#47956) (#48120) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go # Conflicts: # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go # Conflicts: # go.mod # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go
…cessor v2 (#47956) (#48121) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go # Conflicts: # go.mod # x-pack/filebeat/input/azureeventhub/auth.go # x-pack/filebeat/input/azureeventhub/config.go # x-pack/filebeat/input/azureeventhub/v2_input.go Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>
|
Hello @zmoog |
I tested proxy support by setting the
Yes, all new features are available on processor v2 only.
Yes, sorry, my bad. The missing Beats docs are my fault. The integration docs are not there yet, but we'll update them once I'll add support for the transport option in a new integration release (probably early next week). |
✅ Backports have been createdDetails
Cherry-pick of af5d23e has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally
Cherry-pick of af5d23e has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally
Cherry-pick of af5d23e has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # go.mod # go.sum # x-pack/filebeat/input/azureeventhub/auth.go
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e)
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # go.mod # x-pack/filebeat/input/azureeventhub/auth.go
…cessor v2 (#47956) Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2. Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option. In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy. (cherry picked from commit af5d23e) # Conflicts: # go.mod # go.sum # x-pack/filebeat/input/azureeventhub/auth.go
Proposed commit message
Support for AMQP-over-WebSocket transport in the azure-eventhub processor v2.
Enterprise users often need to comply with network restrictions, which means using AMQP may not be an option.
In addition to AMQP-over-WebSocket support, this change allows users to run the azure-eventhub input behind an HTTPS proxy.
Checklist
stresstest.shscript to run them under stress conditions and race detector to verify their stability../changelog/fragmentsusing the changelog tool.Disruptive User Impact
No disruptive user impact. The input still defaults to AMQP transport, and unless users switch to WS transport, the behavior is unchanged.
Author's Checklist
How to test this PR locally
I used the following Filebeat config:
Here are a couple of vscode configuration to test the WebSocket-only and WebSocket with HTTPS proxy scenarios (make sure to replace the placeholders
YOUR * GOES HEREwith actual values).WebSocket only
{ "version": "0.2.0", "configurations": [ { "name": "Launch Filebeat", "type": "go", "request": "launch", "mode": "auto", "program": "${workspaceFolder}/x-pack/filebeat/main.go", "args": [ "-e", "-v", "-d", "*", "--strict.perms=false", "--path.home", "${workspaceFolder}/x-pack/filebeat", "-E", "cloud.id=<YOUR CLOUD ID GOES HERE>", "-E", "cloud.auth=<YOUR USERNAME GOES HERE>:<YOUR PASSWORD GOES HERE>", "-E", "gc_percent=100", "-E", "setup.ilm.enabled=false", "-E", "setup.template.enabled=false", "-E", "output.elasticsearch.allow_older_versions=true" ], "env": { "EVENTHUB_NAME": "logs", "EVENTHUB_CONNECTION_STRING": "<YOUR EVENT HUB CONNECTION STRING GOES HERE>", "EVENTHUB_CONSUMER_GROUP": "$Default", "STORAGE_ACCOUNT": "<YOUR STORAGE ACCOUNT NAME GOES HERE>", "STORAGE_ACCOUNT_KEY": "<YOUR STORAGE ACCOUNT KEY GOES HERE>", "STORAGE_ACCOUNT_CONNECTION_STRING": "<YOUR STORAGE ACCOUNT CONNECTION STRING GOES HERE>", "PROCESSOR_VERSION": "v2", "MIGRATE_CHECKPOINT": "true", "TRANSPORT": "websocket" } } ] }WebSocket with HTTPS proxy
{ "version": "0.2.0", "configurations": [ { "name": "Launch Filebeat", "type": "go", "request": "launch", "mode": "auto", "program": "${workspaceFolder}/x-pack/filebeat/main.go", "args": [ "-e", "-v", "-d", "*", "--strict.perms=false", "--path.home", "${workspaceFolder}/x-pack/filebeat", "-E", "cloud.id=<YOUR CLOUD ID GOES HERE>", "-E", "cloud.auth=<YOUR USERNAME GOES HERE>:<YOUR PASSWORD GOES HERE>", "-E", "gc_percent=100", "-E", "setup.ilm.enabled=false", "-E", "setup.template.enabled=false", "-E", "output.elasticsearch.allow_older_versions=true" ], "env": { "EVENTHUB_NAME": "logs", "EVENTHUB_CONNECTION_STRING": "<YOUR EVENT HUB CONNECTION STRING GOES HERE>", "EVENTHUB_CONSUMER_GROUP": "$Default", "STORAGE_ACCOUNT": "<YOUR STORAGE ACCOUNT NAME GOES HERE>", "STORAGE_ACCOUNT_KEY": "<YOUR STORAGE ACCOUNT KEY GOES HERE>", "STORAGE_ACCOUNT_CONNECTION_STRING": "<YOUR STORAGE ACCOUNT CONNECTION STRING GOES HERE>", "PROCESSOR_VERSION": "v2", "MIGRATE_CHECKPOINT": "true", "TRANSPORT": "websocket", "HTTP_PROXY": "http://127.0.0.1:9090", "HTTPS_PROXY": "http://127.0.0.1:9090" } } ] }To test the proxy support, I used mitmproxy, an interactive HTTPS proxy with request inspection:
Since the Azure services require using an HTTPS proxy, you also need to set up a certificate for mitmproxy. See https://docs.mitmproxy.org/stable/concepts/certificates/ fore more details.
The mitmproxy main view:
Upgrade to AMQP-over-WebSocket:
WebSocket details:
Event hub messages received using AMQP-over-websocket through the proxy:
Related issues
Use cases