Skip to content

Add option to notify test service after agent policy is applied#229

Merged
andrewkroh merged 2 commits intoelastic:masterfrom
andrewkroh:service_notify_signal
Jan 21, 2021
Merged

Add option to notify test service after agent policy is applied#229
andrewkroh merged 2 commits intoelastic:masterfrom
andrewkroh:service_notify_signal

Conversation

@andrewkroh
Copy link
Copy Markdown
Member

@andrewkroh andrewkroh commented Jan 20, 2021

This adds a config option, service_notify_signal, to the system test runner to
allow the test runner to notify the service via a signal (e.g. SIGHUP) after the
policy has been applied by the Agent.

This is useful when the test service is connecting to the listening agent or streaming data
to the agent.

It helps address two issues that I encountered.

  1. While repeatedly testing the agent stays up with the old policy applied, but
    before the new policy is applied and the old ES data cleared the test service would
    connect and deliver data to the "old" agent/policy.
  2. For UDP streams there is no reliable way to know that the agent is ready and listening.

An alternative would be to start the test service container after the policy is applied, but
this could affect tests where the agent connects to the service (e.g. a prometheus exporter)
since the test might expect the service to be ready at startup.

Test Logs

2021/01/21 09:03:13 DEBUG Wait until the policy (ID: 5df0cc00-5bf1-11eb-b44a-31c22f285a18) is assigned to all agents...
2021/01/21 09:03:15 DEBUG running command: /usr/local/bin/docker-compose -f /Users/akroh/go/src/github.com/elastic/elastic-package/test/packages/multiinput/_dev/deploy/docker/docker-compose.yml -p elastic-package-service kill -s SIGHUP test-udp
Killing elastic-package-service_test-udp_1 ... done
2021/01/21 09:03:16 DEBUG checking for expected data in data stream...
2021/01/21 09:03:16 DEBUG found 30 hits in logs-multiinput.test-ep data stream
2021/01/21 09:03:16 DEBUG reassigning original policy back to agent...

@andrewkroh andrewkroh added the enhancement New feature or request label Jan 20, 2021
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Jan 20, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #229 updated

    • Start Time: 2021-01-21T14:04:42.594+0000
  • Duration: 13 min 34 sec

  • Commit: 07cf800

Test stats 🧪

Test Results
Failed 0
Passed 290
Skipped 0
Total 290

@andrewkroh
Copy link
Copy Markdown
Member Author

Failing build is fixed by #230.

Copy link
Copy Markdown
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative would be to start the test service container after the policy is applied, but
this could affect tests where the agent connects to the service (e.g. a prometheus exporter)
since the test might expect the service to be ready at startup.

It looks like a classic "chicken and egg" problem. Most likely we'd have to ignore potential connectivity errors at the beginning. Maybe it's a better idea, but let's defer it at the moment.

Usually in such cases we depended on infinite stream of events (e.g. healthcheck calling curl :80/). I assume that you wouldn't like to follow this scheme, right?

This adds a config option, `service_notify_signal`, to the system test runner to
allow the test runner to notify the service via a signal (e.g. SIGHUP) after the
policy has been applied by the Agent.

This is useful when the test service is connecting to the listening agent or streaming data
to the agent.

It helps address two issues that I encountered.
  1. While repeatedly testing the agent stays up with the old policy applied, but
      before the new policy is applied and the old ES data cleared the test service would
      connect and deliver data to the "old" agent/policy.
  2. For UDP streams there is no reliable way to know that the agent is ready and listening.

An alternative would be to start the test service container after the policy is applied, but
this could affect tests where the agent connects to the service (e.g. a prometheus exporter)
since the test might expect the service to be ready at startup.
@andrewkroh andrewkroh force-pushed the service_notify_signal branch from 4346efe to 20759cc Compare January 21, 2021 13:05
@andrewkroh
Copy link
Copy Markdown
Member Author

Usually in such cases we depended on infinite stream of events (e.g. healthcheck calling curl :80/). I assume that you wouldn't like to follow this scheme, right?

That sounds interesting, could you elaborate a little more what you were thinking. Like making the test service wait on some condition to become true? The only option I can think of is to access the API in kibana to monitor where the agent policy is applied.

@mtojek
Copy link
Copy Markdown
Contributor

mtojek commented Jan 21, 2021

The Apache integration is tested with the regular traffic generated by Docker healthcheck: https://github.com/elastic/integrations/blob/master/packages/apache/_dev/deploy/docker/Dockerfile#L5 . Maybe you can also adapt a similar logic (the healthcheck is a trigger).

@mtojek mtojek self-requested a review January 21, 2021 14:38
Copy link
Copy Markdown
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this feature/configuration option is useful and we may also use it in other places. If there isn't any simpler solution, let's not block it.

@andrewkroh andrewkroh merged commit d17b3a4 into elastic:master Jan 21, 2021
andrewkroh added a commit to elastic/integrations that referenced this pull request Jan 25, 2021
Add system tests for the UDP and TCP inputs in the CheckPoint
Firewall data stream.

Depends on: elastic/elastic-package#229
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants