Skip to content

[New Integration] Envoyproxy#11215

Merged
tehbooom merged 36 commits intoelastic:mainfrom
tehbooom:envoy
Dec 18, 2024
Merged

[New Integration] Envoyproxy#11215
tehbooom merged 36 commits intoelastic:mainfrom
tehbooom:envoy

Conversation

@tehbooom
Copy link
Copy Markdown
Member

  • Enhancement

Proposed commit message

This PR adds the envoyproxy integration for logs and metrics.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@andrewkroh andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Sep 23, 2024
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@tehbooom tehbooom added the Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] label Sep 24, 2024
Copy link
Copy Markdown
Contributor

@ritalwar ritalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all newly developed integrations, we enable TSDB by identifying dimensions. You can read more about this process and follow the guidelines for implementation in this integration here.

@tehbooom
Copy link
Copy Markdown
Member Author

@agithomas , I removed a big chunk of dimensions fields and only kept those that seem like they wont change as much over time. I also added container.id as a dimension.

Should we still create a dashboard for the initial integration or can this be for another PR?

Finally, any thoughts on the following fields also being added as dimensions?

  • host.name
  • service.address
  • cloud.account.id
  • cloud.provider
  • cloud.region
  • cloud.availability_zone
  • agent.id
  • cloud.instance.id

@agithomas
Copy link
Copy Markdown
Contributor

agithomas commented Nov 27, 2024

Finally, any thoughts on the following fields also being added as dimensions?

In the absence of a globally unique identifier, the common dimension fields could help.

For example, if envoyproxy has to be installed in multiple regions, does there exist a unique identifier that separates the installation from the first region from the second region? Adding the above-mentioned dimension fields helps if there exist no Globally unique resource identifier.

@agithomas
Copy link
Copy Markdown
Contributor

agithomas commented Nov 27, 2024

Should we still create a dashboard for the initial integration or can this be for another PR?

Unless this is an urgent requirement, it is always good to package the dashboard as part of the initial release.

@tehbooom
Copy link
Copy Markdown
Member Author

Verified this integration with envoyproxy version 1.32.1.

Also tested with TSDB Migration toolkit with results below

Values being used:
        data_stream = metrics-envoyproxy.stats-76262
        docs_index = First index of the data stream
        settings_mappings_index = Last index of the data stream
        max_docs = All documents
        get_overlapping_files = True
        directory_overlapping_files = overwritten-docs-metrics-envoyproxy.stats-76262
        display_docs = 10
        copy_docs_per_dimension = 2

You're testing with version 8.16.0.

Testing data stream metrics-envoyproxy.stats-76262.
Index being used for the documents is .ds-metrics-envoyproxy.stats-76262-2024.11.27-000001.
Index being used for the settings and mappings is .ds-metrics-envoyproxy.stats-76262-2024.11.27-000001.

The time series fields for the TSDB index are:
        - dimension (4 fields):
                - container.id
                - envoyproxy.cluster_name
                - envoyproxy.http_conn_manager_prefix
                - envoyproxy.listener_address
        - counter (52 fields):
                - envoyproxy.envoy_cluster_default_total_match_count.count
                - envoyproxy.envoy_cluster_external_upstream_rq.count
                - envoyproxy.envoy_cluster_external_upstream_rq_completed.count
                - envoyproxy.envoy_cluster_external_upstream_rq_xx.count
                - envoyproxy.envoy_cluster_health_check_attempt.count
                - envoyproxy.envoy_cluster_health_check_failure.count
                - envoyproxy.envoy_cluster_health_check_network_failure.count
                - envoyproxy.envoy_cluster_membership_change.count
                - envoyproxy.envoy_cluster_ssl_ciphers.count
                - envoyproxy.envoy_cluster_ssl_curves.count
                - envoyproxy.envoy_cluster_ssl_handshake.count
                - envoyproxy.envoy_cluster_ssl_sigalgs.count
                - envoyproxy.envoy_cluster_ssl_versions.count
                - envoyproxy.envoy_cluster_upstream_cx_destroy.count
                - envoyproxy.envoy_cluster_upstream_cx_destroy_local.count
                - envoyproxy.envoy_cluster_upstream_cx_destroy_local_with_active_rq.count
                - envoyproxy.envoy_cluster_upstream_cx_destroy_with_active_rq.count
                - envoyproxy.envoy_cluster_upstream_cx_rx_bytes_total.count
                - envoyproxy.envoy_cluster_upstream_cx_tx_bytes_total.count
                - envoyproxy.envoy_cluster_upstream_rq.count
                - envoyproxy.envoy_cluster_upstream_rq_pending_total.count
                - envoyproxy.envoy_cluster_upstream_rq_total.count
                - envoyproxy.envoy_http_downstream_cx_destroy.count
                - envoyproxy.envoy_http_downstream_cx_destroy_active_rq.count
                - envoyproxy.envoy_http_downstream_cx_destroy_local.count
                - envoyproxy.envoy_http_downstream_cx_destroy_remote.count
                - envoyproxy.envoy_http_downstream_cx_destroy_remote_active_rq.count
                - envoyproxy.envoy_http_downstream_cx_http1_total.count
                - envoyproxy.envoy_http_downstream_cx_rx_bytes_total.count
                - envoyproxy.envoy_http_downstream_cx_ssl_total.count
                - envoyproxy.envoy_http_downstream_cx_total.count
                - envoyproxy.envoy_http_downstream_cx_tx_bytes_total.count
                - envoyproxy.envoy_http_downstream_rq_completed.count
                - envoyproxy.envoy_http_downstream_rq_http1_total.count
                - envoyproxy.envoy_http_downstream_rq_rx_reset.count
                - envoyproxy.envoy_http_downstream_rq_time.count
                - envoyproxy.envoy_http_downstream_rq_total.count
                - envoyproxy.envoy_http_downstream_rq_xx.count
                - envoyproxy.envoy_http_rq_direct_response.count
                - envoyproxy.envoy_http_rq_total.count
                - envoyproxy.envoy_listener_connections_accepted_per_socket_event.count
                - envoyproxy.envoy_listener_downstream_cx_destroy.count
                - envoyproxy.envoy_listener_downstream_cx_total.count
                - envoyproxy.envoy_listener_http_downstream_rq_completed.count
                - envoyproxy.envoy_listener_http_downstream_rq_xx.count
                - envoyproxy.envoy_listener_ssl_ciphers.count
                - envoyproxy.envoy_listener_ssl_curves.count
                - envoyproxy.envoy_listener_ssl_handshake.count
                - envoyproxy.envoy_listener_ssl_no_certificate.count
                - envoyproxy.envoy_listener_ssl_ocsp_staple_requests.count
                - envoyproxy.envoy_listener_ssl_versions.count
                - envoyproxy.envoy_listener_worker_downstream_cx_total.count
        - gauge (15 fields):
                - envoyproxy.envoy_cluster_circuit_breakers_default_cx_open.value
                - envoyproxy.envoy_cluster_membership_degraded.value
                - envoyproxy.envoy_cluster_membership_excluded.value
                - envoyproxy.envoy_cluster_membership_healthy.value
                - envoyproxy.envoy_cluster_membership_total.value
                - envoyproxy.envoy_cluster_upstream_cx_active.value
                - envoyproxy.envoy_cluster_warming_state.value
                - envoyproxy.envoy_http_downstream_cx_active.value
                - envoyproxy.envoy_http_downstream_cx_http1_active.value
                - envoyproxy.envoy_http_downstream_cx_rx_bytes_buffered.value
                - envoyproxy.envoy_http_downstream_cx_ssl_active.value
                - envoyproxy.envoy_http_downstream_rq_active.value
                - envoyproxy.envoy_listener_downstream_cx_active.value
                - envoyproxy.envoy_listener_downstream_pre_cx_active.value
                - envoyproxy.envoy_listener_worker_downstream_cx_active.value
        - routing_path (4 fields):
                - container.id
                - envoyproxy.cluster_name
                - envoyproxy.http_conn_manager_prefix
                - envoyproxy.listener_address

Index tsdb-index-enabled successfully created.

Copying documents from .ds-metrics-envoyproxy.stats-76262-2024.11.27-000001 to tsdb-index-enabled...
All 224 documents taken from index .ds-metrics-envoyproxy.stats-76262-2024.11.27-000001 were successfully placed to index tsdb-index-enabled.

I think we will also need to remove the beats from the package registry as seen below

Screenshot From 2024-11-27 15-53-04

@agithomas
Copy link
Copy Markdown
Contributor

It would be best to include the below-mentioned common dimension fields.

host.name
service.address
cloud.account.id
cloud.provider
cloud.region
cloud.availability_zone
agent.id
cloud.instance.id

Copy link
Copy Markdown
Contributor

@ritalwar ritalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please handle TSDB-related feedback. The rest looks good, and we can create dashboards in a separate PR.

- name: mongo_prefix
type: keyword
description: Stats prefix for the Mongo Proxy network filter
- name: mongo_cmd
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could there be multiple values for this field ? If so, kindly make this field as a dimension.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all the filters mentioned below, including GRPC, mongo, Router HTTP, Network , RBAC network filter, etc, kindly verify if multiple values are possible at a given time. If so, it may be added to the dimensions list. Please confirm.

- name: mongo_cmd
type: keyword
description: Request command for the Mongo Proxy network filter
- name: mongo_collection
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could there be multiple values for this field ? If so, kindly make this field as a dimension.

@tehbooom
Copy link
Copy Markdown
Member Author

tehbooom commented Dec 3, 2024

I did more testing using the envoy examples repo to send as much data I can get to the agent.

What I've found is every field with *_prefix should be a dimension and most of the keyword fields should also be a dimension besides ssl_*.

Furthermore, a lot of the fields can be broken up into envoy.http, envoy.cluster, and envoy.listener. This can be achieved with a simple loop of all fields in envoy splitting on _, removing the first envoy, and replacing the next _ with a .

For example envoy.envoy_udp_downstream_sess_rx_datagrams.count would be envoy.udp.downstream_sess_rx_datagrams.count

Now, the solution works for most of the fields but there are some that I have seen that will not follow this pattern like envoy.envoy_thread_local_cluster_manager_worker_clusters_inflated.value since it should be thread_local_cluster_manager.* instead of thread.*

Thoughts? I'm also going to try datadogd instead of graphite. Below is the fields that I gathered from my testing with the examples repo to help.

[
  {
    "name": "envoy.cluster.grpc.bridge_method",
    "type": "string"
  },
  {
    "name": "envoy.cluster.grpc.bridge_service",
    "type": "string"
  },
  {
    "name": "envoy.cluster.name",
    "type": "string"
  },
  {
    "name": "envoy.cluster.ratelimit.prefix",
    "type": "string"
  },
  {
    "name": "envoy.cluster.response_code",
    "type": "string"
  },
  {
    "name": "envoy.config.ext_authz.prefix",
    "type": "string"
  },
  {
    "name": "envoy.connection_limit.prefix",
    "type": "string"
  },
  {
    "name": "envoy.dns_filter.prefix",
    "type": "string"
  },
  {
    "name": "envoy.envoy_cluster_circuit_breakers_default_cx_open.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_circuit_breakers_default_cx_pool_open.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_circuit_breakers_default_rq_open.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_circuit_breakers_default_rq_pending_open.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_circuit_breakers_high_cx_pool_open.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_default_total_match_count.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_ext_authz_denied.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_ext_authz_ok.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_external_upstream_rq.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_external_upstream_rq_completed.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_external_upstream_rq_time.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_external_upstream_rq_time.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_external_upstream_rq_time.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_external_upstream_rq_time.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_external_upstream_rq_time.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_external_upstream_rq_time.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_external_upstream_rq_time.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_external_upstream_rq_xx.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_health_check_attempt.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_health_check_failure.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_health_check_network_failure.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_http2_outbound_control_frames_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_http2_outbound_frames_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_http2_pending_send_bytes.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_http2_streams_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_internal_upstream_rq.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_internal_upstream_rq_completed.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_internal_upstream_rq_time.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_internal_upstream_rq_time.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_internal_upstream_rq_time.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_internal_upstream_rq_time.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_internal_upstream_rq_time.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_internal_upstream_rq_time.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_internal_upstream_rq_time.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_internal_upstream_rq_xx.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_maglev_lb_max_entries_per_host.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_maglev_lb_min_entries_per_host.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_manager_active_clusters.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_manager_cluster_added.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_manager_warming_clusters.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_max_host_weight.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_membership_change.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_membership_degraded.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_membership_excluded.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_membership_healthy.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_membership_total.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_redis_cluster_upstream_commands_upstream_rq_time.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_redis_cluster_upstream_commands_upstream_rq_time.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_redis_cluster_upstream_commands_upstream_rq_time.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_redis_cluster_upstream_commands_upstream_rq_time.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_redis_cluster_upstream_commands_upstream_rq_time.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_redis_cluster_upstream_commands_upstream_rq_time.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_redis_cluster_upstream_commands_upstream_rq_time.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_ssl_ciphers.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_ssl_curves.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_ssl_handshake.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_ssl_session_reused.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_ssl_sigalgs.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_ssl_versions.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_udp_sess_tx_datagrams.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_update_attempt.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_update_no_rebuild.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_update_success.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_connect_ms.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_connect_ms.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_connect_ms.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_connect_ms.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_connect_ms.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_connect_ms.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_connect_ms.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_destroy.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_destroy_local.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_destroy_local_with_active_rq.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_destroy_remote.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_destroy_remote_with_active_rq.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_destroy_with_active_rq.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_http1_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_http2_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_length_ms.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_length_ms.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_length_ms.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_length_ms.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_length_ms.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_length_ms.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_length_ms.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_rx_bytes_buffered.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_rx_bytes_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_cx_tx_bytes_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_completed.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_pending_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_pending_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_time.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_time.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_time.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_time.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_time.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_time.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_time.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_upstream_rq_xx.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_cluster_warming_state.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_dns_cares_not_found.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_dns_cares_pending_resolutions.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_dns_cares_resolve_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_filesystem_flushed_by_timer.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_filesystem_write_buffered.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_filesystem_write_completed.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_filesystem_write_total_buffered.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_destroy.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_destroy_active_rq.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_destroy_local.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_destroy_remote.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_destroy_remote_active_rq.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_http1_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_http1_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_length_ms.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_length_ms.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_length_ms.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_length_ms.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_length_ms.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_length_ms.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_length_ms.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_rx_bytes_buffered.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_rx_bytes_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_ssl_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_ssl_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_cx_tx_bytes_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_completed.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_http1_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_rx_reset.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_time.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_time.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_time.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_time.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_time.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_time.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_time.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_downstream_rq_xx.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_ext_authz_denied.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_ext_authz_ok.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_local_rate_limit_enabled.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_local_rate_limit_enforced.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_local_rate_limit_ok.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_local_rate_limit_rate_limited.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_rbac_allowed.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_rbac_denied.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_http_rq_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_connections_accepted_per_socket_event.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_connections_accepted_per_socket_event.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_connections_accepted_per_socket_event.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_connections_accepted_per_socket_event.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_connections_accepted_per_socket_event.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_connections_accepted_per_socket_event.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_connections_accepted_per_socket_event.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_cx_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_cx_destroy.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_cx_length_ms.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_cx_length_ms.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_cx_length_ms.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_cx_length_ms.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_cx_length_ms.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_cx_length_ms.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_cx_length_ms.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_cx_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_downstream_pre_cx_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_http_downstream_rq_completed.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_http_downstream_rq_xx.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_main_thread_downstream_cx_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_admin_main_thread_downstream_cx_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_connections_accepted_per_socket_event.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_connections_accepted_per_socket_event.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_connections_accepted_per_socket_event.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_connections_accepted_per_socket_event.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_connections_accepted_per_socket_event.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_connections_accepted_per_socket_event.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_connections_accepted_per_socket_event.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_cx_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_cx_destroy.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_cx_length_ms.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_cx_length_ms.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_cx_length_ms.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_cx_length_ms.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_cx_length_ms.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_cx_length_ms.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_cx_length_ms.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_cx_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_downstream_pre_cx_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_http_downstream_rq_completed.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_http_downstream_rq_xx.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_manager_listener_added.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_manager_listener_create_success.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_manager_total_listeners_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_manager_total_listeners_warming.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_manager_workers_started.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_ssl_ciphers.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_ssl_curves.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_ssl_handshake.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_ssl_no_certificate.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_ssl_ocsp_staple_requests.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_ssl_versions.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_worker_downstream_cx_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_listener_worker_downstream_cx_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_get_latency.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_get_latency.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_get_latency.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_get_latency.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_get_latency.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_get_latency.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_get_latency.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_get_success.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_get_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_set_latency.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_set_latency.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_set_latency.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_set_latency.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_set_latency.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_set_latency.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_set_latency.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_set_success.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_command_set_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_downstream_cx_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_downstream_cx_rx_bytes_buffered.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_downstream_cx_rx_bytes_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_downstream_cx_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_downstream_cx_tx_bytes_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_downstream_rq_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_redis_downstream_rq_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_runtime_load_success.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_runtime_num_keys.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_runtime_num_layers.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_runtime_override_dir_not_exists.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_compilation_settings_fips_mode.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_concurrency.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_days_until_first_cert_expiring.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_dynamic_unknown_fields.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_hot_restart_epoch.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_hot_restart_generation.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_initialization_time_ms.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_initialization_time_ms.max",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_initialization_time_ms.mean",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_initialization_time_ms.mean_rate",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_initialization_time_ms.median",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_initialization_time_ms.min",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_initialization_time_ms.stddev",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_live.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_memory_allocated.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_memory_heap_size.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_memory_physical_size.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_parent_connections.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_state.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_static_unknown_fields.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_stats_recent_lookups.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_total_connections.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_uptime.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_version.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_server_wip_protos.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_tcp_downstream_cx_rx_bytes_buffered.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_tcp_downstream_cx_rx_bytes_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_tcp_downstream_cx_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_tcp_downstream_cx_tx_bytes_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_thread_local_cluster_manager_main_thread_clusters_inflated.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_thread_local_cluster_manager_worker_0_clusters_inflated.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_thread_local_cluster_manager_worker_1_clusters_inflated.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_thread_local_cluster_manager_worker_clusters_inflated.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_tls_inspector_alpn_not_found.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_tls_inspector_sni_found.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_tls_inspector_tls_found.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_udp_downstream_sess_active.value",
    "type": "number"
  },
  {
    "name": "envoy.envoy_udp_downstream_sess_rx_bytes.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_udp_downstream_sess_rx_datagrams.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_udp_downstream_sess_total.count",
    "type": "number"
  },
  {
    "name": "envoy.envoy_udp_idle_timeout.count",
    "type": "number"
  },
  {
    "name": "envoy.http.conn_manager_prefix",
    "type": "string"
  },
  {
    "name": "envoy.http.dynamodb.operation",
    "type": "string"
  },
  {
    "name": "envoy.http.dynamodb.partition_id",
    "type": "string"
  },
  {
    "name": "envoy.http.dynamodb.table",
    "type": "string"
  },
  {
    "name": "envoy.http.fault_downstream_cluster",
    "type": "string"
  },
  {
    "name": "envoy.http.rbac.http_prefix",
    "type": "string"
  },
  {
    "name": "envoy.http.rbac.policy_name",
    "type": "string"
  },
  {
    "name": "envoy.http.rbac.prefix",
    "type": "string"
  },
  {
    "name": "envoy.http.rds.route_config",
    "type": "string"
  },
  {
    "name": "envoy.http.rds.scoped_config",
    "type": "string"
  },
  {
    "name": "envoy.http.response_code_class",
    "type": "string"
  },
  {
    "name": "envoy.http.route",
    "type": "string"
  },
  {
    "name": "envoy.http.user_agent",
    "type": "string"
  },
  {
    "name": "envoy.listener.address",
    "type": "string"
  },
  {
    "name": "envoy.listener.ssl.cipher",
    "type": "string"
  },
  {
    "name": "envoy.listener.ssl.cipher_suite",
    "type": "string"
  },
  {
    "name": "envoy.listener.ssl.clientssl_prefix",
    "type": "string"
  },
  {
    "name": "envoy.listener.ssl.curve",
    "type": "string"
  },
  {
    "name": "envoy.listener.ssl.sigalg",
    "type": "string"
  },
  {
    "name": "envoy.listener.ssl.version",
    "type": "string"
  },
  {
    "name": "envoy.listener.worker_id",
    "type": "string"
  },
  {
    "name": "envoy.local_http_ratelimit.prefix",
    "type": "string"
  },
  {
    "name": "envoy.local_listener_ratelimit.prefix",
    "type": "string"
  },
  {
    "name": "envoy.local_network_ratelimit.prefix",
    "type": "string"
  },
  {
    "name": "envoy.mongo.callsite",
    "type": "string"
  },
  {
    "name": "envoy.mongo.cmd",
    "type": "string"
  },
  {
    "name": "envoy.mongo.collection",
    "type": "string"
  },
  {
    "name": "envoy.mongo.prefix",
    "type": "string"
  },
  {
    "name": "envoy.proxy_proto.prefix",
    "type": "string"
  },
  {
    "name": "envoy.proxy_proto.version",
    "type": "string"
  },
  {
    "name": "envoy.redis.prefix",
    "type": "string"
  },
  {
    "name": "envoy.tcp.prefix",
    "type": "string"
  },
  {
    "name": "envoy.thrift.prefix",
    "type": "string"
  },
  {
    "name": "envoy.udp.prefix",
    "type": "string"
  },
  {
    "name": "envoy.virtual.cluster",
    "type": "string"
  },
  {
    "name": "envoy.virtual.host",
    "type": "string"
  }
]

@tehbooom
Copy link
Copy Markdown
Member Author

tehbooom commented Dec 4, 2024

Using DogStatsdSink did not change anything still the same problem.

I was able to create the fields as I described before with the following

- script:
    lang: painless
    description: This script processor iterates over the whole document to remove few fields.
    params:
      special_cases:
        - thread_local_cluster_manager
        - cluster_manager
        - tls_inspector
    source: |-
      for (entry in ctx.envoy.entrySet()) {
          def k = entry.getKey();
          def v = entry.getValue();
          if (k.startsWith('envoy_')) {
              def parts = k.splitOnToken('_');
              def newKey = '';
 
              def matchFound = false;
              for (specialCase in params.special_cases) {
                  def prefix = 'envoy_' + specialCase;
                  if (k.startsWith(prefix)) {
                      def suffix = k.substring(prefix.length());
                      if (suffix.startsWith('_')) {
                          suffix = suffix.substring(1);
                      }
                      newKey = specialCase + (suffix.length() > 0 ? '.' + suffix : '');
                      matchFound = true;
                      break;                  }
              }
              if (!matchFound && parts.length >= 2) {
                  def remaining = '';
                  for (int i = 2; i < parts.length; i++) {
                      remaining += parts[i];
                      if (i < parts.length - 1) remaining += '_';
                  }
                  newKey = parts[1] + '.' + remaining;
              }
              if (newKey.length() > 0) {
                  ctx.envoy.remove(k);
                  ctx.envoy[newKey] = v;
              }
          }
          if (!(v instanceof String)) {
              v.keySet().stream().filter(s -> s.startsWith('1m_rate') || s.startsWith('5m_rate') ||s.startsWith('15m_rate') || s.startsWith('p99_9') || s.startsWith('p75') || s.startsWith('p99') || s.startsWith('p95'))
              .collect(Collectors.toList()).forEach(m -> v.remove(m))
          }
      }

Here are the fields generated using the envoy examples repo again

{
  "envoy": {
    "cluster": {
      "1_circuit_breakers_default_cx_open": {
        "value": {
          "type": "number"
        }
      },
      "1_circuit_breakers_default_rq_open": {
        "value": {
          "type": "number"
        }
      },
      "1_circuit_breakers_default_rq_pending_open": {
        "value": {
          "type": "number"
        }
      },
      "1_default_total_match_count": {
        "count": {
          "type": "number"
        }
      },
      "1_max_host_weight": {
        "value": {
          "type": "number"
        }
      },
      "1_membership_change": {
        "count": {
          "type": "number"
        }
      },
      "1_membership_degraded": {
        "value": {
          "type": "number"
        }
      },
      "1_membership_excluded": {
        "value": {
          "type": "number"
        }
      },
      "1_membership_healthy": {
        "value": {
          "type": "number"
        }
      },
      "1_membership_total": {
        "value": {
          "type": "number"
        }
      },
      "1_update_attempt": {
        "count": {
          "type": "number"
        }
      },
      "1_update_no_rebuild": {
        "count": {
          "type": "number"
        }
      },
      "1_update_success": {
        "count": {
          "type": "number"
        }
      },
      "1_upstream_cx_active": {
        "value": {
          "type": "number"
        }
      },
      "1_upstream_cx_connect_ms": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "1_upstream_cx_destroy": {
        "count": {
          "type": "number"
        }
      },
      "1_upstream_cx_destroy_remote": {
        "count": {
          "type": "number"
        }
      },
      "1_upstream_cx_destroy_remote_with_active_rq": {
        "count": {
          "type": "number"
        }
      },
      "1_upstream_cx_destroy_with_active_rq": {
        "count": {
          "type": "number"
        }
      },
      "1_upstream_cx_length_ms": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "1_upstream_cx_rx_bytes_buffered": {
        "value": {
          "type": "number"
        }
      },
      "1_upstream_cx_rx_bytes_total": {
        "count": {
          "type": "number"
        }
      },
      "1_upstream_cx_total": {
        "count": {
          "type": "number"
        }
      },
      "1_upstream_cx_tx_bytes_total": {
        "count": {
          "type": "number"
        }
      },
      "1_upstream_rq_active": {
        "value": {
          "type": "number"
        }
      },
      "1_upstream_rq_pending_active": {
        "value": {
          "type": "number"
        }
      },
      "1_upstream_rq_pending_total": {
        "count": {
          "type": "number"
        }
      },
      "1_upstream_rq_total": {
        "count": {
          "type": "number"
        }
      },
      "1_warming_state": {
        "value": {
          "type": "number"
        }
      },
      "circuit_breakers_default_cx_open": {
        "value": {
          "type": "number"
        }
      },
      "circuit_breakers_default_cx_pool_open": {
        "value": {
          "type": "number"
        }
      },
      "circuit_breakers_default_rq_open": {
        "value": {
          "type": "number"
        }
      },
      "circuit_breakers_default_rq_pending_open": {
        "value": {
          "type": "number"
        }
      },
      "circuit_breakers_high_cx_pool_open": {
        "value": {
          "type": "number"
        }
      },
      "default_total_match_count": {
        "count": {
          "type": "number"
        }
      },
      "ext_authz": {
        "prefix": {
          "type": "string"
        }
      },
      "external_upstream_rq": {
        "count": {
          "type": "number"
        }
      },
      "external_upstream_rq_completed": {
        "count": {
          "type": "number"
        }
      },
      "external_upstream_rq_time": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "external_upstream_rq_xx": {
        "count": {
          "type": "number"
        }
      },
      "grpc": {
        "bridge_method": {
          "type": "string"
        },
        "bridge_service": {
          "type": "string"
        }
      },
      "health_check_attempt": {
        "count": {
          "type": "number"
        }
      },
      "health_check_failure": {
        "count": {
          "type": "number"
        }
      },
      "health_check_network_failure": {
        "count": {
          "type": "number"
        }
      },
      "http2_outbound_control_frames_active": {
        "value": {
          "type": "number"
        }
      },
      "http2_outbound_frames_active": {
        "value": {
          "type": "number"
        }
      },
      "http2_pending_send_bytes": {
        "value": {
          "type": "number"
        }
      },
      "http2_streams_active": {
        "value": {
          "type": "number"
        }
      },
      "maglev_lb_max_entries_per_host": {
        "value": {
          "type": "number"
        }
      },
      "maglev_lb_min_entries_per_host": {
        "value": {
          "type": "number"
        }
      },
      "max_host_weight": {
        "value": {
          "type": "number"
        }
      },
      "membership_change": {
        "count": {
          "type": "number"
        }
      },
      "membership_degraded": {
        "value": {
          "type": "number"
        }
      },
      "membership_excluded": {
        "value": {
          "type": "number"
        }
      },
      "membership_healthy": {
        "value": {
          "type": "number"
        }
      },
      "membership_total": {
        "value": {
          "type": "number"
        }
      },
      "name": {
        "type": "string"
      },
      "ratelimit": {
        "prefix": {
          "type": "string"
        }
      },
      "redis_cluster_upstream_commands_upstream_rq_time": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "response_code": {
        "type": "string"
      },
      "ssl_ciphers": {
        "count": {
          "type": "number"
        }
      },
      "ssl_curves": {
        "count": {
          "type": "number"
        }
      },
      "ssl_handshake": {
        "count": {
          "type": "number"
        }
      },
      "ssl_sigalgs": {
        "count": {
          "type": "number"
        }
      },
      "ssl_versions": {
        "count": {
          "type": "number"
        }
      },
      "udp_sess_tx_datagrams": {
        "count": {
          "type": "number"
        }
      },
      "update_attempt": {
        "count": {
          "type": "number"
        }
      },
      "update_no_rebuild": {
        "count": {
          "type": "number"
        }
      },
      "update_success": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_active": {
        "value": {
          "type": "number"
        }
      },
      "upstream_cx_connect_ms": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "upstream_cx_destroy": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_destroy_local": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_destroy_local_with_active_rq": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_destroy_remote": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_destroy_remote_with_active_rq": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_destroy_with_active_rq": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_http1_total": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_http2_total": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_length_ms": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "upstream_cx_rx_bytes_buffered": {
        "value": {
          "type": "number"
        }
      },
      "upstream_cx_rx_bytes_total": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_total": {
        "count": {
          "type": "number"
        }
      },
      "upstream_cx_tx_bytes_total": {
        "count": {
          "type": "number"
        }
      },
      "upstream_rq": {
        "count": {
          "type": "number"
        }
      },
      "upstream_rq_active": {
        "value": {
          "type": "number"
        }
      },
      "upstream_rq_cancelled": {
        "count": {
          "type": "number"
        }
      },
      "upstream_rq_completed": {
        "count": {
          "type": "number"
        }
      },
      "upstream_rq_pending_active": {
        "value": {
          "type": "number"
        }
      },
      "upstream_rq_pending_total": {
        "count": {
          "type": "number"
        }
      },
      "upstream_rq_time": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "upstream_rq_total": {
        "count": {
          "type": "number"
        }
      },
      "upstream_rq_xx": {
        "count": {
          "type": "number"
        }
      },
      "warming_state": {
        "value": {
          "type": "number"
        }
      }
    },
    "cluster_manager": {
      "active_clusters": {
        "value": {
          "type": "number"
        }
      },
      "cluster_added": {
        "count": {
          "type": "number"
        }
      },
      "warming_clusters": {
        "value": {
          "type": "number"
        }
      }
    },
    "connection_limit": {
      "prefix": {
        "type": "string"
      }
    },
    "dns": {
      "cares_get_addr_failure": {
        "count": {
          "type": "number"
        }
      },
      "cares_not_found": {
        "count": {
          "type": "number"
        }
      },
      "cares_pending_resolutions": {
        "value": {
          "type": "number"
        }
      },
      "cares_resolve_total": {
        "count": {
          "type": "number"
        }
      }
    },
    "dns_filter": {
      "prefix": {
        "type": "string"
      }
    },
    "filesystem": {
      "flushed_by_timer": {
        "count": {
          "type": "number"
        }
      },
      "write_buffered": {
        "count": {
          "type": "number"
        }
      },
      "write_completed": {
        "count": {
          "type": "number"
        }
      },
      "write_total_buffered": {
        "value": {
          "type": "number"
        }
      }
    },
    "http": {
      "conn_manager_prefix": {
        "type": "string"
      },
      "downstream_cx_active": {
        "value": {
          "type": "number"
        }
      },
      "downstream_cx_destroy": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_destroy_active_rq": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_destroy_local": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_destroy_remote": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_destroy_remote_active_rq": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_http1_active": {
        "value": {
          "type": "number"
        }
      },
      "downstream_cx_http1_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_length_ms": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "downstream_cx_rx_bytes_buffered": {
        "value": {
          "type": "number"
        }
      },
      "downstream_cx_rx_bytes_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_ssl_active": {
        "value": {
          "type": "number"
        }
      },
      "downstream_cx_ssl_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_tx_bytes_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_rq_active": {
        "value": {
          "type": "number"
        }
      },
      "downstream_rq_completed": {
        "count": {
          "type": "number"
        }
      },
      "downstream_rq_http1_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_rq_rx_reset": {
        "count": {
          "type": "number"
        }
      },
      "downstream_rq_time": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "downstream_rq_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_rq_xx": {
        "count": {
          "type": "number"
        }
      },
      "dynamodb": {
        "operation": {
          "type": "string"
        },
        "partition_id": {
          "type": "string"
        },
        "table": {
          "type": "string"
        }
      },
      "fault_downstream_cluster": {
        "type": "string"
      },
      "rbac": {
        "http_prefix": {
          "type": "string"
        },
        "policy_name": {
          "type": "string"
        },
        "prefix": {
          "type": "string"
        }
      },
      "rbac_allowed": {
        "count": {
          "type": "number"
        }
      },
      "rbac_denied": {
        "count": {
          "type": "number"
        }
      },
      "rds": {
        "route_config": {
          "type": "string"
        },
        "scoped_config": {
          "type": "string"
        }
      },
      "response_code_class": {
        "type": "string"
      },
      "route": {
        "type": "string"
      },
      "rq_total": {
        "count": {
          "type": "number"
        }
      },
      "user_agent": {
        "type": "string"
      }
    },
    "listener": {
      "address": {
        "type": "string"
      },
      "admin_connections_accepted_per_socket_event": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "admin_downstream_cx_active": {
        "value": {
          "type": "number"
        }
      },
      "admin_downstream_cx_total": {
        "count": {
          "type": "number"
        }
      },
      "admin_downstream_pre_cx_active": {
        "value": {
          "type": "number"
        }
      },
      "admin_http_downstream_rq_completed": {
        "count": {
          "type": "number"
        }
      },
      "admin_http_downstream_rq_xx": {
        "count": {
          "type": "number"
        }
      },
      "admin_main_thread_downstream_cx_active": {
        "value": {
          "type": "number"
        }
      },
      "admin_main_thread_downstream_cx_total": {
        "count": {
          "type": "number"
        }
      },
      "connections_accepted_per_socket_event": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "downstream_cx_active": {
        "value": {
          "type": "number"
        }
      },
      "downstream_cx_destroy": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_length_ms": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "downstream_cx_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_pre_cx_active": {
        "value": {
          "type": "number"
        }
      },
      "http_downstream_rq_completed": {
        "count": {
          "type": "number"
        }
      },
      "http_downstream_rq_xx": {
        "count": {
          "type": "number"
        }
      },
      "manager_listener_added": {
        "count": {
          "type": "number"
        }
      },
      "manager_listener_create_success": {
        "count": {
          "type": "number"
        }
      },
      "manager_total_listeners_active": {
        "value": {
          "type": "number"
        }
      },
      "manager_total_listeners_warming": {
        "value": {
          "type": "number"
        }
      },
      "manager_workers_started": {
        "value": {
          "type": "number"
        }
      },
      "ssl": {
        "cipher": {
          "type": "string"
        },
        "cipher_suite": {
          "type": "string"
        },
        "clientssl_prefix": {
          "type": "string"
        },
        "curve": {
          "type": "string"
        },
        "sigalg": {
          "type": "string"
        },
        "version": {
          "type": "string"
        }
      },
      "ssl_ciphers": {
        "count": {
          "type": "number"
        }
      },
      "ssl_curves": {
        "count": {
          "type": "number"
        }
      },
      "ssl_handshake": {
        "count": {
          "type": "number"
        }
      },
      "ssl_no_certificate": {
        "count": {
          "type": "number"
        }
      },
      "ssl_ocsp_staple_requests": {
        "count": {
          "type": "number"
        }
      },
      "ssl_versions": {
        "count": {
          "type": "number"
        }
      },
      "worker_downstream_cx_active": {
        "value": {
          "type": "number"
        }
      },
      "worker_downstream_cx_total": {
        "count": {
          "type": "number"
        }
      },
      "worker_id": {
        "type": "string"
      }
    },
    "local_http_ratelimit": {
      "prefix": {
        "type": "string"
      }
    },
    "local_listener_ratelimit": {
      "prefix": {
        "type": "string"
      }
    },
    "local_network_ratelimit": {
      "prefix": {
        "type": "string"
      }
    },
    "mongo": {
      "callsite": {
        "type": "string"
      },
      "cmd": {
        "type": "string"
      },
      "collection": {
        "type": "string"
      },
      "prefix": {
        "type": "string"
      }
    },
    "proxy_proto": {
      "prefix": {
        "type": "string"
      },
      "version": {
        "type": "string"
      }
    },
    "redis": {
      "command_get_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_active": {
        "value": {
          "type": "number"
        }
      },
      "downstream_cx_protocol_error": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_rx_bytes_buffered": {
        "value": {
          "type": "number"
        }
      },
      "downstream_cx_rx_bytes_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_rq_active": {
        "value": {
          "type": "number"
        }
      },
      "downstream_rq_total": {
        "count": {
          "type": "number"
        }
      },
      "prefix": {
        "type": "string"
      },
      "splitter_unsupported_command": {
        "count": {
          "type": "number"
        }
      }
    },
    "runtime": {
      "load_success": {
        "count": {
          "type": "number"
        }
      },
      "num_keys": {
        "value": {
          "type": "number"
        }
      },
      "num_layers": {
        "value": {
          "type": "number"
        }
      },
      "override_dir_not_exists": {
        "count": {
          "type": "number"
        }
      }
    },
    "server": {
      "compilation_settings_fips_mode": {
        "value": {
          "type": "number"
        }
      },
      "concurrency": {
        "value": {
          "type": "number"
        }
      },
      "days_until_first_cert_expiring": {
        "value": {
          "type": "number"
        }
      },
      "dynamic_unknown_fields": {
        "count": {
          "type": "number"
        }
      },
      "hot_restart_epoch": {
        "value": {
          "type": "number"
        }
      },
      "hot_restart_generation": {
        "value": {
          "type": "number"
        }
      },
      "initialization_time_ms": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "live": {
        "value": {
          "type": "number"
        }
      },
      "memory_allocated": {
        "value": {
          "type": "number"
        }
      },
      "memory_heap_size": {
        "value": {
          "type": "number"
        }
      },
      "memory_physical_size": {
        "value": {
          "type": "number"
        }
      },
      "parent_connections": {
        "value": {
          "type": "number"
        }
      },
      "state": {
        "value": {
          "type": "number"
        }
      },
      "static_unknown_fields": {
        "count": {
          "type": "number"
        }
      },
      "stats_recent_lookups": {
        "value": {
          "type": "number"
        }
      },
      "total_connections": {
        "value": {
          "type": "number"
        }
      },
      "uptime": {
        "value": {
          "type": "number"
        }
      },
      "version": {
        "value": {
          "type": "number"
        }
      },
      "wip_protos": {
        "count": {
          "type": "number"
        }
      }
    },
    "tcp": {
      "downstream_cx_rx_bytes_buffered": {
        "value": {
          "type": "number"
        }
      },
      "downstream_cx_rx_bytes_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_total": {
        "count": {
          "type": "number"
        }
      },
      "downstream_cx_tx_bytes_total": {
        "count": {
          "type": "number"
        }
      },
      "prefix": {
        "type": "string"
      }
    },
    "thread_local_cluster_manager": {
      "main_thread_clusters_inflated": {
        "value": {
          "type": "number"
        }
      },
      "worker_0_clusters_inflated": {
        "value": {
          "type": "number"
        }
      },
      "worker_1_clusters_inflated": {
        "value": {
          "type": "number"
        }
      },
      "worker_clusters_inflated": {
        "value": {
          "type": "number"
        }
      }
    },
    "thrift": {
      "prefix": {
        "type": "string"
      }
    },
    "tls_inspector": {
      "alpn_found": {
        "count": {
          "type": "number"
        }
      },
      "alpn_not_found": {
        "count": {
          "type": "number"
        }
      },
      "bytes_processed": {
        "count": {
          "type": "number"
        },
        "max": {
          "type": "number"
        },
        "mean": {
          "type": "number"
        },
        "mean_rate": {
          "type": "number"
        },
        "median": {
          "type": "number"
        },
        "min": {
          "type": "number"
        },
        "stddev": {
          "type": "number"
        }
      },
      "sni_found": {
        "count": {
          "type": "number"
        }
      },
      "tls_found": {
        "count": {
          "type": "number"
        }
      },
      "tls_not_found": {
        "count": {
          "type": "number"
        }
      }
    },
    "udp": {
      "downstream_sess_active": {
        "value": {
          "type": "number"
        }
      },
      "downstream_sess_rx_bytes": {
        "count": {
          "type": "number"
        }
      },
      "downstream_sess_rx_datagrams": {
        "count": {
          "type": "number"
        }
      },
      "downstream_sess_total": {
        "count": {
          "type": "number"
        }
      },
      "prefix": {
        "type": "string"
      }
    },
    "vhost": {
      "cluster": {
        "type": "string"
      },
      "host": {
        "type": "string"
      }
    }
  }
}

@agithomas
Copy link
Copy Markdown
Contributor

@daniela-elastic , as discussed kindly inspect the metrics included as part of the integration. Please verify if are not having more metrics than actually recommended.

Secondly, please review the package ownership details.

@elastic-sonarqube
Copy link
Copy Markdown

@elasticmachine
Copy link
Copy Markdown

💚 Build Succeeded

History

@agithomas
Copy link
Copy Markdown
Contributor

Hi @lalit-satapathy and @daniela-elastic

Could you please look at this comment](#11215 (comment)) and advice?

Thanks

@lalit-satapathy
Copy link
Copy Markdown
Contributor

lalit-satapathy commented Dec 18, 2024

Secondly, please review the package ownership details.

LGTM.

@tehbooom tehbooom merged commit 201fb65 into elastic:main Dec 18, 2024
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package envoyproxy - 0.1.0 containing this change is available at https://epr.elastic.co/package/envoyproxy/0.1.0/

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

New Integration Issue or pull request for creating a new integration package. Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EnvoyProxy Integration

7 participants