[Fleet] Support input-level vars & templates by jen-huang · Pull Request #83878 · elastic/kibana

jen-huang · 2020-11-20T00:46:53Z

Summary

Resolves #82599. This PR enables input-level agent YAML templates, similar to stream-level templates. Input-level vars will be passed to the specified template in package info. The input can have streams or no streams. Here is an example showing the package info containing template_path field on the input, the contents of that template, and the final agent yaml generated by Kibana with this change:

(for cleaner illustration purposes, this input does not have streams)

Package info

  "policy_templates": [
    {
      "name": "input_level_templates",
      "title": "Input level templates",
      "description": "Input with input-level template to use input-level vars with",
      "inputs": [
        {
          "type": "logs",
          "vars": [
            {
              "name": "paths",
              "type": "text",
              "title": "Paths",
              "multi": true,
              "required": true,
              "show_user": false,
              "default": [
                "/foo/bar",
                "/test/path"
              ]
            },
            {
              "name": "host",
              "type": "text",
              "title": "Host",
              "multi": false,
              "required": true,
              "show_user": false
            }
          ],
          "title": "Collect logs",
          "template_path": "template.yml.hbs"
        }
      ],
      "multiple": true
    }
  ]

/agent/input/template.yml.hbs

fixedField: some value
paths:
{{#each paths}}
  - {{this}}
{{/each}}
host: {{host}}

Agent yaml

inputs:
  - id: b42d8fc8-6fb7-4c5a-96bb-c4fa3120b6b8
    name: input_level_templates-1
    revision: 1
    type: logs
    use_output: default
    meta:
      package:
        name: input_level_templates
        version: 1.0.0
    data_stream:
      namespace: default
    fixedField: some value
    paths:
      - /foo/bar
      - /test/path
    host: localhost

Here's what the integration policy form UI looks like for the same. Note that there is only a top-level Settings section for the input-level vars, and no stream toggles are displayed underneath it:

Testing

Run Kibana using a local package registry running latest main or this PR branch
Find and add the test integration called "Input level templates"
The UI and agent policy yaml output should reflect above screenshots

This reverts commit 34f7014.

…ompilation method names more generic (instead of only for streams). Add testts

elasticmachine · 2020-11-20T20:02:02Z

Pinging @elastic/ingest-management (Team:Ingest Management)

jen-huang · 2020-11-23T15:29:45Z

@elasticmachine merge upstream

ruflin · 2020-11-24T08:26:31Z

@jen-huang Can we skip the streams part?

nchaulet

Looks good to me 👍 and work as expected

…-templates

jen-huang · 2020-11-24T19:16:47Z

@nchaulet Thanks for testing!

@ruflin 493e54a Sure, removed empty streams: [] from agent yaml when there are no streams.

kibanamachine · 2020-11-25T02:45:46Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: ac46477

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.1MB	1.1MB	+558.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	374.7KB	374.8KB	+93.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
`ingest-package-policies`	30	31	+1

History

💔 Build #90071 failed 493e54a
💚 Build #89660 succeeded 30459b9
💚 Build #89140 succeeded 62effd3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jalvz

works great, thanks!

* Fix bug creating new policy on the fly * Adjust UI for input with vars but no streams * Revert "Fix bug creating new policy on the fly" This reverts commit 34f7014. * Add `compiled_input` field and compile input template, if any. Make compilation method names more generic (instead of only for streams). Add testts * Add compiled input to generated agent yaml * Don't return empty streams in agent yaml when there aren't any * Update missed assertion Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* master: (70 commits) [Uptime] Fix headers io-ts type (elastic#84089) [fleet] Add config options to accepted docker env vars (elastic#84338) [Fleet] Support URL query state in agent logs UI (elastic#84298) [basePathProxy] include query in redirect (elastic#84356) [Security Solution] Add Endpoint policy feature checks (elastic#83972) Fix issues with show_license_expiration (elastic#84361) [Security Solution][Resolver] Add support for predefined schemas for endpoint and winlogbeat (elastic#84103) [cli/dev] log a warning when --no-base-path is used with --dev (elastic#84354) [Fleet] Support input-level vars & templates (elastic#83878) [APM] Elastic chart issues (elastic#84238) [Time to Visualize] Fix Unlink Action via Rollback of ReplacePanel (elastic#83873) redirect to visualize listing page when by value visualization editor doesn't have a value input (elastic#84287) add live region for field search (elastic#84310) [ML] Persisted URL state for Anomalies table (elastic#84314) [dev/cli] detect worker type using env, not cluster module (elastic#83977) [Workplace Search] Migrate DisplaySettings tree (elastic#84283) Deprecate `xpack.task_manager.index` setting (elastic#84155) [Search] Search batching using bfetch (again) (elastic#84043) Use .kibana instead of .kibana_current to mark migration completion (elastic#83373) [Monitoring] Only look at ES for the missing data alert for now (elastic#83839) ...

* master: (119 commits) [Uptime] Fix headers io-ts type (elastic#84089) [fleet] Add config options to accepted docker env vars (elastic#84338) [Fleet] Support URL query state in agent logs UI (elastic#84298) [basePathProxy] include query in redirect (elastic#84356) [Security Solution] Add Endpoint policy feature checks (elastic#83972) Fix issues with show_license_expiration (elastic#84361) [Security Solution][Resolver] Add support for predefined schemas for endpoint and winlogbeat (elastic#84103) [cli/dev] log a warning when --no-base-path is used with --dev (elastic#84354) [Fleet] Support input-level vars & templates (elastic#83878) [APM] Elastic chart issues (elastic#84238) [Time to Visualize] Fix Unlink Action via Rollback of ReplacePanel (elastic#83873) redirect to visualize listing page when by value visualization editor doesn't have a value input (elastic#84287) add live region for field search (elastic#84310) [ML] Persisted URL state for Anomalies table (elastic#84314) [dev/cli] detect worker type using env, not cluster module (elastic#83977) [Workplace Search] Migrate DisplaySettings tree (elastic#84283) Deprecate `xpack.task_manager.index` setting (elastic#84155) [Search] Search batching using bfetch (again) (elastic#84043) Use .kibana instead of .kibana_current to mark migration completion (elastic#83373) [Monitoring] Only look at ES for the missing data alert for now (elastic#83839) ...

jen-huang added 5 commits November 19, 2020 14:42

Fix bug creating new policy on the fly

34f7014

Adjust UI for input with vars but no streams

b391fbf

Revert "Fix bug creating new policy on the fly"

d38d5ee

This reverts commit 34f7014.

Add compiled_input field and compile input template, if any. Make c…

6120f65

…ompilation method names more generic (instead of only for streams). Add testts

Add compiled input to generated agent yaml

62effd3

jen-huang added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.11.0 labels Nov 20, 2020

jen-huang self-assigned this Nov 20, 2020

jen-huang changed the title ~~[Fleet] Allow input-level vars & templates, without streams~~ [Fleet] Support input-level vars & templates Nov 20, 2020

jen-huang marked this pull request as ready for review November 20, 2020 20:01

jen-huang requested a review from a team November 20, 2020 20:02

jen-huang requested a review from jalvz November 20, 2020 20:02

jalvz mentioned this pull request Nov 23, 2020

Integrate with Elastic Agent elastic/apm-server#4004

Closed

15 tasks

Merge branch 'master' into ingest/input-vars-templates

30459b9

nchaulet approved these changes Nov 24, 2020

View reviewed changes

jen-huang added 2 commits November 24, 2020 11:11

Merge remote-tracking branch 'upstream/master' into ingest/input-vars…

f0a9c97

…-templates

Don't return empty streams in agent yaml when there aren't any

493e54a

Update missed assertion

ac46477

jen-huang requested review from a team as code owners November 25, 2020 00:34

jalvz approved these changes Nov 25, 2020

View reviewed changes

jen-huang merged commit 284b104 into elastic:master Nov 25, 2020

jen-huang deleted the ingest/input-vars-templates branch November 25, 2020 18:13

jen-huang mentioned this pull request Nov 25, 2020

[7.x] [Fleet] Support input-level vars & templates (#83878) #84359

Merged

nchaulet mentioned this pull request Dec 8, 2020

[Fleet] Allow to edit package policy with input level template #85277

Merged

ruflin mentioned this pull request Jan 20, 2021

Fleet Server package elastic/integrations#544

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Support input-level vars & templates#83878

[Fleet] Support input-level vars & templates#83878
jen-huang merged 9 commits intoelastic:masterfrom
jen-huang:ingest/input-vars-templates

jen-huang commented Nov 20, 2020 •

edited

Loading

Uh oh!

elasticmachine commented Nov 20, 2020

Uh oh!

jen-huang commented Nov 23, 2020

Uh oh!

ruflin commented Nov 24, 2020

Uh oh!

nchaulet left a comment

Uh oh!

jen-huang commented Nov 24, 2020

Uh oh!

kibanamachine commented Nov 25, 2020

Uh oh!

jalvz left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

jen-huang commented Nov 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

elasticmachine commented Nov 20, 2020

Uh oh!

jen-huang commented Nov 23, 2020

Uh oh!

ruflin commented Nov 24, 2020

Uh oh!

nchaulet left a comment

Choose a reason for hiding this comment

Uh oh!

jen-huang commented Nov 24, 2020

Uh oh!

kibanamachine commented Nov 25, 2020

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

Saved Objects .kibana field count

History

Uh oh!

jalvz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jen-huang commented Nov 20, 2020 •

edited

Loading