PIP: #19074

Motivation

When multiple PulsarClient instances run within the same JVM, each client creates separate thread pools, timers, event loops, and DNS resolvers. This approach leads to significant resource waste, including:

• Excessive memory usage due to duplicated thread pools and executors
• High thread counts that can impact application performance
• Increased DNS lookup load and latency from duplicate DNS resolvers
• Inefficient resource utilization in applications requiring multiple client connections (e.g., Flink pipelines with multiple Pulsar Sinks/Sources)

This change addresses the resource sharing gap by introducing a public API for sharing core client infrastructure while maintaining isolated connection pools per client instance.

Modifications

This PR implements PIP-234 by adding the following components:

• PulsarClientSharedResources - Main interface for managing shared resources across multiple PulsarClient instances
• PulsarClientSharedResourcesBuilder - Fluent API for configuring and building shared resources with support for:
  • Thread pool configuration (listener, internal, scheduled, lookup executors)
  • Event loop group settings with busy-wait support
  • Timer configuration with customizable tick duration
  • DNS resolver settings with comprehensive options (TTL, timeouts, server addresses, etc.)
• ClientBuilder.sharedResources() - Method to attach shared resources to a client instance
• Resource sharing implementation - Internal changes to accept externally provided executors, timers, event loops, and DNS resolvers

Key Features

• Selective sharing: Choose specific resources to share (event loops, thread pools, DNS resolver, timer)
• Advanced DNS configuration: Full control over DNS resolver behavior including TTL settings, server addresses, and timeout configurations
• HTTP client integration: Shared DNS resolver and timer are also used by HTTP-based lookup services
• Service provider integration: AutoClusterFailover and ControlledClusterFailover also benefit from shared DNS resolution

Usage Example

// Create shared resources for 1000+ clients
PulsarClientSharedResources sharedResources = PulsarClientSharedResources.builder()
    .configureEventLoop(config -> config.numberOfThreads(10))
    .configureDnsResolver(config -> config
        .serverAddresses(List.of(new InetSocketAddress("8.8.8.8", 53)))
        .maxTtl(5))
    .build();

// Use shared resources across multiple clients
List<PulsarClient> clients = new ArrayList<>();
for (int i = 0; i < 1000; i++) {
    clients.add(PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .sharedResources(sharedResources)
        .build());
}

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete