Pinging @elastic/ingest-management (Team:Ingest Management)

@paul-tavares this is changing a little the comportment of the create Datasource endpoint if you are using a package, maybe worth to check it

Thanks @nchaulet for pinging me. I think we're ok. We are only currently using the config property of the datasource (datasource.inputs[].config) and the new one you added seems to be optional, so I also don't see that having an impact on us.

@paul-tavares yes this change is only concerning streams so you should be good 👍

Will the agent config YAML tab be accurate?

@jen-huang Yes it will contains the same thing that is returned to the agent

@jen-huang good catch I am currently not using the input level variable to populate the stream template.

In my opinion the generated config should be the second one, but that means we remove config from the input too, and only rely on the template to send data.

@jen-huang I understand the confusion, but the way @ruflin and I were thinking about key inheritance was that keys defined in the input would be applied to all the streams node. The driving force behind that opinion was linked to the standalone agent mode to reduce the number of fields a user needs to configure when configuring a data source and reduce duplication of fields/source of errors.

But, I think I should describe what are the rules in the agent in that context.

• If a key is defined in the input level, the configuration will be copied in the stream. (with some exception id, enabled.)
• If the same key exists in both input or streams an error will be produced.
• If the key only exist at the stream level the agent will use that value.

But because of the above rules, this actually means that the agent would support both configurations you are describing. (@michalpristas this is correct right?).

I would prefer if we have a single way to define it, where values are defined at the input level, so its consistent with the user. But maybe the later is easier for code generation.

@jen-huang I understand the confusion here but I think what @ph describes is correct. Even though we define it on the input level in the end, the config we create is all on the stream level and very verbose as it is easiest to generate. Also not sure how otherwise we could have single stream template.

Updated my PR to use agent config variable when building the agent stream from template

it's generate this input for nginx (disabling log error as the template is currently broken)

  - id: nginx-1 
    package:
       name: nginx
       version: 1.2.0
    namespace: default
    enabled: true
    use_output: default
    inputs:
      - type: nginx/metrics
        enabled: true
        streams:
          - enabled: true
            dataset: nginx.stubstatus
            input: nginx/metrics
            metricsets:
              - stubstatus
            period: 10s
            hosts:
              - 'http://127.0.0.1'
            username: nicolas
            password: mypassword
            id: nginx/metrics-nginx.stubstatus
      - type: logs
        enabled: true
        streams:
          - enabled: true
            dataset: nginx.access
            input: log
            paths:
              - /var/log/nginx/access.log*
            exclude_files:
              - .gz$
            processors:
              - add_locale: null
            id: logs-nginx.access

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 35145be

History

• 💔 Build #40710 failed c4471a0
• 💚 Build #39720 succeeded 54f8e90
• 💚 Build #39404 succeeded 153063b
• 💚 Build #39245 succeeded 87b7a0c
• 💚 Build #39088 succeeded 271195f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

I pulled this down and looked at it in the UI. LGTM so far. One thing I stumbled over is:

processors:
              - add_locale: null

But in the stream.yml it is actually

processors:
  - add_locale: ~

@ph I wonder if null instead of ~ is going to work? Also wondering why it converts ~ to null.

Yes null and ~ should be the same, we are using js-yaml to serialize, deserialize with the default config is serializing all null values to null