PIP-320

Motivation

PIP-264 laid out the foundation for switching our entire metrics pipeline to OpenTelemetry. PIP-320 describes the first step in this process, adding an SDK wrapper to instantiate an OpenTelemetry client with a couple of sane defaults for Pulsar brokers, proxies and function workers. This PR provides the implementation for the new wrapper.

Modifications

• Add new artifact pulsar-otel-metrics-provider to encapsulate the new OpenTelemetryService. Our immediate goal is to make the service available to the broker, proxy, and function worker. Putting this in a new artifact allows us to cherry-pick this dependency as needed. Alternatively, this could be moved to pulsar-common, but it would then leak into the clients as well, which are out of scope for now. For the desired out-of-the-box experience, this artifact additionally pulls in the OTLP and Prometheus exporter dependencies.
• Add the OpenTelemetryService class as a wrapper for the safe instantiation of the OpenTelemetry SDK. It serves the following purposes:
  • Disable OpenTelemetry by default. Can be overridden by the customer using environment variables or system properties.
  • Adjust the default number of permissible attributes per Meter (called the cardinality limit) from 2000 to 10000.
  • Add pulsar.cluster as a resource attribute (aka label) to all metrics emitted by this SDK instance. Note that this parameter cannot be null or empty. Many of the existing Proxy tests were not setting this value in their configuration and had to be adapted. This value can also be overridden by environment variables or system properties.
  • With the SDK instantiated, provide access to OpenTelemetry Meter objects. These are used to emit the actual metrics.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• Added unit test for class OpenTelemetryService, including:
  • testClusterNameCannotBeEmpty
  • testClusterNameCannotBeNull
  • testResourceAttributesAreSet
  • testIsInstrumentationNameSetOnMeter
  • testMetricCardinality
  • testLongCounter: verifies basic integer counter metrics can be emitted
  • testServiceIsDisabledByDefault
• Added integration test class MetricsTest, validating the entire end-to-end metrics pipeline for brokers, proxies, and function-workers, using both the in-process Prometheus exporter and remote OTLP collector. Separated to run in its own CI integration test target, as it did not naturally fit anywhere else.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
  • OpenTelemetry SDK libraries
  • OpenTelemetry SDK Autoconfigure Extension
    • This references the OpenTelemetry logs and traces libraries, too, which we don't need. Removing them does not work, though, as the Autoconfigure Extension does not load properly without them.
  • OpenTelemetry OTLP exporter
  • OpenTelemetry Prometheus exporter
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
  • This PR adds support for further migration of metrics to OpenTelemetry. No metrics are being changed as part of this PR itself.
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: dragosvictor#6