This issue was created from a comment on another issue, the comment has the description of the problem.

@strawgate I'm using the Cloudflare Logpush integration, the Firewall Events and HTTP Requests dataset.

I have multiple Inputs with the same configuration only the queue change, I'm sharing one of them here:

  - id: aws-s3-cloudflare-f56bb89c-97bd-441e-ae4b-071d5296cd1d
    name: cloudflare-REDACT
    revision: 6
    type: aws-s3
    use_output: c20b778c-0220-40f4-a8ca-03b974ada63b
    meta:
      package:
        name: cloudflare_logpush
        version: 1.35.0
    data_stream:
      namespace: cloudflare
    package_policy_id: f56bb89c-97bd-441e-ae4b-071d5296cd1d
    streams:
      - id: >-
          aws-s3-cloudflare_logpush.firewall_event-f56bb89c-97bd-441e-ae4b-071d5296cd1d
        data_stream:
          dataset: cloudflare_logpush.firewall_event
          type: logs
        queue_url: >-
          https://sqs.us-east-1.amazonaws.com/REDACTED
        visibility_timeout: 300s
        api_timeout: 120s
        max_number_of_messages: 70
        file_selectors:
          - regex: fw/
        access_key_id: '${SECRET_3}'
        secret_access_key: '${SECRET_4}'
        tags:
          - collect_sqs_logs
          - cloudflare-firewall
          - cloudflare-fw-REDACT
        processors:
          - drop_event:
              when:
                or:
                  - contains:
                      message: REDACTED
      - id: >-
          aws-s3-cloudflare_logpush.http_request-f56bb89c-97bd-441e-ae4b-071d5296cd1d
        data_stream:
          dataset: cloudflare_logpush.http_request
          type: logs
        queue_url: >-
          https://sqs.us-east-1.amazonaws.com/REDACTED
        visibility_timeout: 300s
        api_timeout: 120s
        max_number_of_messages: 70
        file_selectors:
          - regex: http/
        access_key_id: '${SECRET_3}'
        secret_access_key: '${SECRET_4}'
        tags:
          - collect_sqs_logs
          - cloudflare-http
          - cloudflare-http-REDACT

As you can see, there is the max_number_of_messages with the value we set, but not the number of workers, which is set in the integration.

Inspecting the input of the integration here, it seems that the number_of_workers setting will only get the value defined by the user only if the s3 collection of logs is enabled. (polling direct from the buckets)

{{#if collect_s3_logs}}
{{! shared S3 bucket polling options }}
{{#if number_of_workers }}
number_of_workers: {{ number_of_workers }}
{{/if}}

Which I assume is populated by this setting in the UI:

But since I'm using SQS, I cannot enable this setting as it will complain that bucket arn and other settings are missing.

It seems an issue on the integration, so we may need to open another issue.

In the Elastic Agent logs I have this, with the default value of 5.

Originally posted by @leandrojmp in #12101