[Synthetics] Add Browser data streams by dominiqueclarke · Pull Request #1064 · elastic/integrations

dominiqueclarke · 2021-06-01T20:56:27Z

Relates to elastic/uptime#219

Type of change

Enhancement

What does this PR do?

This PR adds the browser, browser_screenshot, and browser_network data streams to the synthetics package.

This PR is a draft state.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs or synthetics.
I have added an entry to my package's changelog.yml file.
If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0). (Note: Even though this PR introduces a new feature, I have opted out of increasing the stack release constraint, as this version is compatible with 7.14 as it is additive).

elasticmachine · 2021-06-01T21:09:02Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-23T13:30:30.575+0000
Duration: 16 min 21 sec
Commit: 53b9266

Test stats 🧪

Test	Results
Failed	0
Passed	6
Skipped	0
Total	6

Trends 🧪

@dominiqueclarke

#27052) Creates a new offline tagged docker image that includes the dependencies required for synthetics. Building elastic-agent docker images will now create additional -offline tagged images with these extras. We may need to do additional work to ensure that these additional images are incorporated into the release process and onto the public website. These changes also set the ELASTIC_SYNTHETICS_CAPABLE env flag, as done in heartbeat, to enable synthetics features in the docker environment. This dovetails with the work @dominiqueclarke is doing in elastic/integrations#1064 and elsewhere Fixes #22932

@dominiqueclarke

#27052) Creates a new offline tagged docker image that includes the dependencies required for synthetics. Building elastic-agent docker images will now create additional -offline tagged images with these extras. We may need to do additional work to ensure that these additional images are incorporated into the release process and onto the public website. These changes also set the ELASTIC_SYNTHETICS_CAPABLE env flag, as done in heartbeat, to enable synthetics features in the docker environment. This dovetails with the work @dominiqueclarke is doing in elastic/integrations#1064 and elsewhere Fixes #22932 (cherry picked from commit 4727470)

@dominiqueclarke

#27052) (#27324) Creates a new offline tagged docker image that includes the dependencies required for synthetics. Building elastic-agent docker images will now create additional -offline tagged images with these extras. We may need to do additional work to ensure that these additional images are incorporated into the release process and onto the public website. These changes also set the ELASTIC_SYNTHETICS_CAPABLE env flag, as done in heartbeat, to enable synthetics features in the docker environment. This dovetails with the work @dominiqueclarke is doing in elastic/integrations#1064 and elsewhere Fixes #22932 (cherry picked from commit 4727470) Co-authored-by: Andrew Cholakian <andrew@andrewvc.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

elasticmachine · 2021-08-17T14:00:19Z

Pinging @elastic/uptime (Team:Uptime)

packages/synthetics/data_stream/browser/agent/stream/browser.yml.hbs

andrewvc · 2021-08-17T19:51:03Z

packages/synthetics/data_stream/browser/agent/stream/browser.yml.hbs

+{{#if tags}}
+tags: {{tags}}
+{{/if}}
+{{#if source.zip_url.username}}


I notice that the various fields for zip_url are all specifically enumerated. Is it easier just to say zip_url takes a JSON blob and leave it to the Kibana UI code to put the right fields in?

This is a great idea. I love it. I think it'll provide a lot more flexibility to add new nested keys to the UI without having to do a corresponding change on the integrations package side.

However, the UI is not currently set up to support this. Each field receives a key corresponding to the correct parameter, resulting in the UI in a large object with with all the field names, using dot notation for the nested fields, and the field values. So we have an object that looks like

{ 'source.zip_url.username': 'username', 'source.zip_url.password': 'password' }

Fleet exposes an integration policy object that represents the policy values. It looks something like

{ 'source.zip_url.username': { value: '', type: 'text, }, 'source.zip_url.password': { value: '', type: 'text, } }

We iterate over the first object's keys and assign values to the exposed fleet policy.
https://github.com/elastic/kibana/blob/master/x-pack/plugins/uptime/public/components/fleet_package/use_update_policy.ts#L62
Which ends up looking something like

dataStream.vars['source.zip_url.username'].value = ourConfig.source.zip_url.username;

So the UI doesn't currently support nested keys. It is a flattened object.

To do this properly, we should convert all of the fields in http, tcp, icmp, and browser to only expose their top level params and allow the UI to handle nested params.

We can then transform our flatted object into a nested object, and use those values to populate nested fields.

I don't think that'll make it in for 7.15, but sounds like it could be a useful refactor for 7.16. Thoughts?

That makes sense. Let's keep it the way it is now and reevaluate going forward

packages/synthetics/data_stream/browser/fields/base-fields.yml

packages/synthetics/data_stream/browser/fields/synthetics.yml

packages/synthetics/data_stream/browser_screenshot/fields/base-fields.yml

andrewvc · 2021-08-17T20:01:17Z

packages/synthetics/data_stream/browser_screenshot/fields/tls.yml

@@ -0,0 +1,39 @@
+- name: tls


We can nix this file for screenshots

andrewvc · 2021-08-17T20:01:22Z

packages/synthetics/data_stream/browser_screenshot/fields/tcp.yml

@@ -0,0 +1,30 @@
+- name: tcp


We can nix this file for screenshots

andrewvc · 2021-08-17T20:01:31Z

packages/synthetics/data_stream/browser_screenshot/fields/resolve.yml

@@ -0,0 +1,18 @@
+- name: resolve


We can nix this file for screenshots

andrewvc · 2021-08-17T20:01:59Z

packages/synthetics/data_stream/browser_screenshot/fields/http.yml

@@ -0,0 +1,91 @@
+- name: http


We can nix this file for screenshots

@dominiqueclarke

#27052) Creates a new offline tagged docker image that includes the dependencies required for synthetics. Building elastic-agent docker images will now create additional -offline tagged images with these extras. We may need to do additional work to ensure that these additional images are incorporated into the release process and onto the public website. These changes also set the ELASTIC_SYNTHETICS_CAPABLE env flag, as done in heartbeat, to enable synthetics features in the docker environment. This dovetails with the work @dominiqueclarke is doing in elastic/integrations#1064 and elsewhere Fixes #22932 (cherry picked from commit 4727470) # Conflicts: # dev-tools/mage/dockerbuilder.go

@dominiqueclarke

#27052) (#27324) Creates a new offline tagged docker image that includes the dependencies required for synthetics. Building elastic-agent docker images will now create additional -offline tagged images with these extras. We may need to do additional work to ensure that these additional images are incorporated into the release process and onto the public website. These changes also set the ELASTIC_SYNTHETICS_CAPABLE env flag, as done in heartbeat, to enable synthetics features in the docker environment. This dovetails with the work @dominiqueclarke is doing in elastic/integrations#1064 and elsewhere Fixes #22932 (cherry picked from commit 4727470) Co-authored-by: Andrew Cholakian <andrew@andrewvc.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit f9b2610) # Conflicts: # dev-tools/mage/dockerbuilder.go

andrewvc

LGTM

dominiqueclarke added 2 commits June 1, 2021 16:53

add browser mappings

10bcf71

add browser manifest.yml and browser.yml.hbs files

3c254ef

dominiqueclarke marked this pull request as draft June 1, 2021 20:56

dominiqueclarke added the Team:obs-ux-infra_services Obs UX: Infra & Services team [elastic/obs-ux-infra_services-team] label Jun 1, 2021

dominiqueclarke changed the title ~~Feature/216 synthetics browser~~ [Synthetics] Add Browser data streams Jun 1, 2021

adjust synthetics mappings formatting

61613de

add package policy keys for browser data stream

99e6951

andrewvc mentioned this pull request Jul 27, 2021

[Heartbeat] Build elastic agent with synthetics support / offline mode elastic/beats#27052

Merged

4 tasks

dominiqueclarke added 8 commits July 27, 2021 11:36

update manifest.yml

fd845a3

synthetics - add browser params config option

abdf660

merge master

c448912

update browswer data streams

0a74b91

format package

38ef810

update manifest.yml for browser input

8f24354

update description

9cac84f

update changelog

d30856a

dominiqueclarke added v7.15.0 enhancement New feature or request labels Aug 17, 2021

dominiqueclarke requested a review from a team August 17, 2021 13:52

merge master

ed9f93c

dominiqueclarke marked this pull request as ready for review August 17, 2021 14:00

andrewvc requested changes Aug 17, 2021

View reviewed changes

add ignore_https_errors to browser data stream

c06520f

dominiqueclarke added 6 commits August 18, 2021 14:53

remove unnecessary mappings

8ebeb25

add values to constant keywords

5d48aae

add journey tags mapping

ec55675

remove tls mappings

c28cd05

remove resolve mappings for screenshot

dcd009a

adjust manifest

53b9266

andrewvc approved these changes Aug 23, 2021

View reviewed changes

dominiqueclarke merged commit 227e245 into elastic:master Aug 23, 2021

dominiqueclarke deleted the feature/216-synthetics-browser branch August 23, 2021 15:07

dominiqueclarke mentioned this pull request Aug 23, 2021

[Uptime] [Synthetics integration] Add synthetics integration browser type functional tests elastic/kibana#109730

Closed

Conversation

dominiqueclarke commented Jun 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Uh oh!

elasticmachine commented Jun 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

Uh oh!

elasticmachine commented Aug 17, 2021

Uh oh!

Uh oh!

andrewvc Aug 17, 2021

Choose a reason for hiding this comment

Uh oh!

dominiqueclarke Aug 17, 2021

Choose a reason for hiding this comment

Uh oh!

andrewvc Aug 17, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewvc Aug 17, 2021

Choose a reason for hiding this comment

Uh oh!

andrewvc Aug 17, 2021

Choose a reason for hiding this comment

Uh oh!

andrewvc Aug 17, 2021

Choose a reason for hiding this comment

Uh oh!

andrewvc Aug 17, 2021

Choose a reason for hiding this comment

Uh oh!

andrewvc left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dominiqueclarke commented Jun 1, 2021 •

edited

Loading

elasticmachine commented Jun 1, 2021 •

edited

Loading