How can I use keystore variables in autodiscover.providers.templates.config section of metricbeat.yml? It looks like broken.

• Version: Metricbeat 7.1.0
• Operating System: containers from https://www.docker.elastic.co/
• Discuss Forum URL: https://discuss.elastic.co/t/variables-substitution-doesnt-work-from-keystore-in-username-password-fields-of-metricbeat-yml-in-autodiscover-providers-templates-config-section/186125

Steps to Reproduce

1. Setup Docker for running next services
2. Setup Elasticsearch and configure Metricbeat to connect to it
3. Run a RabbitMQ service with an username and a password for authorization
4. Add QUEUE_USERNAME and QUEUE_PASSWORD to Metricbeat keystore via metricbeat keystore add with appropriate values
5. Add a block like this to metricbeat.yml:

  autodiscover:
    providers:
      - type: docker
        templates:
          - condition:
              # You can use any other way to match the service here
              equals:
                docker.container.labels.com.docker.swarm.service.name: my_rabbitmq_service
            config:
              - module: rabbitmq
                period: 10s
                hosts: ['${data.host}:${data.port}']
                username: ${QUEUE_USERNAME}
                password: ${QUEUE_PASSWORD}

6. Start the beat

Expected:
Metricbeat successfully connected to RabbitMQ service.

But it doesn't.

Interesting additional info:

1. For debugging purposes, you can use "${QUEUE_PASSWORD}" as a value for field output.elasticsearch.hosts of metricbeat.yml and you will see the finish value of QUEUE_PASSWORD variable from the keystore in logs of the beat (in error messages about the connection to Elasticsearch). Therefore the keystore really store the correct value.

2. If you replace ${QUEUE_USERNAME} and ${QUEUE_PASSWORD} by hard-coded values in metricbeat.yml and restart the beat it will work. Therefore Metricbeat can connect to RabbitMQ with required auth and username/password fields of the metricbeat.yml really works.

3. Even if you return the vars back to metricbeat.yml and add 2 Linux env vars QUEUE_USERNAME and QUEUE_PASSWORD and restart the beat it will work too. Therefore environ variables substitution work in the autodiscover.providers.templates.config section. But keystore variables substitution doesn't work.