Tracking

Current status (February 2025):

1. The original proposal is below in this description. This was accepted for proceeding on in Java SIG meeting July 20 2023, after example walkthroughs of instrumentation being moved are provided
2. Currently simple instrumentation modules are dynamically migrated at runtime (next step includes converting this to static migration)
3. The complex instrumentations are now migrated (Indy module migration #11457)
4. The remaining steps are listed here

• A couple of examples of how the proposed changes would affect existing instrumentation, if moved to the new framework were provided for the following SIG meeting July 27 2023, and a decision to proceed on the proposal was then agreed
• The proposed architecture outline and an implementation plan outline were added in this comment and PRs for the implementation began - a working POC is available for reference### Related PRs

Note not for inclusion in this issue are preferred post completion task to remove shading from the agent

[COMPLETED] Phase 1 - Enabling invoke dynamic capability for instrumentations

• Implemented InstrumentationModuleClassLoader for invokedynamic Advice dispatching #9177
• Add isIndyModule() default to InstrumentationModule #9180
• Bytecode version upgrade for invokedynamic dispatch #9239
• Implement invokedynamic advice bootstrapping #9382
• Replace InstrumentationModule.isIndyModule() with a new interface in the internal package, per example - this is no longer being done as it was decided to leave the method as is for now and change or deprecate as desired (as is not yet InstrumentationModule stable so deprecation and change is relatively quick)

[COMPLETED] Phase 2a - Migrate simple instrumentation modules

• Write migration guide (so we can link to it in reviews)
• Any instrumentation that can be migrated without "Phase 2b features", - this is now detailed in this PR which identifies all simple and non-simple instrumentations

[COMPLETED] Phase 2b - Additional support for muzzle and complex instrumentation modules

• muzzle runtime dual-parent verification that dependencies are present (currently only application classloader dependencies are checked, need to verify dependencies in both parents)
  -- Make more tests run with indy #9729
• muzzle compile-time reference/dependency gathering from static methods/fields (currently excluded)
• muzzle/runtime compile/runtime enforcing non-inlined instrumentations only - reject instrumentations with inlined advices
• muzzle add indy modules advices to helper classes
• opening java 9+ modules mechanism for instrumentations that need specific module access
  -- donate LDAP instrumentation
• member access support (to make package-private stuff accessible as advices are no longer inlined)
  -- Make more tests run with indy #9729
• support for SPI implementations
  -- Implemented factory for generating invokedynamic proxy classes #9502
  -- Add capability for invokedynamic InstrumentationModules to inject proxies #9565
  -- Allow injection of helper bytecode as resources #9752
• Allow InstrumentationModules to optionally share their InstrumentationModuleClassloader (see this discussion)

Phase 3 - Migrate remaining instrumentation modules

• migrate all remaining modules

Proposal

Contribute alternative invokedynamic based instrumentation mechanism to the OpenTelemetry Java agent, with no required changes to the existing OpenTelemetry instrumentation mechanism or instrumentations

Benefits to OpenTelemetry of this contribution and mechanism

• Allows instrumentations to have breakpoints set in them and be debugged using standard debugging techniques
• No shading required, allowing standard debugging in IDEs
• Works cleanly with the Java Module system
• Clean isolation of instrumentation advice from the application and other instrumentations
• Uses the approach recommended by Byte Buddy creator Rafael Winterhalter (see background section below)
• Makes available additional contributors/maintainers from Elastic for OpenTelemetry
• It's a parallel instrumentation approach (to the existing OpenTelemetry Java agent) that would live alongside the existing one and not impact the existing approach at all
• Implementors would have the option to use either instrumentation approach, whichever they prefer
• It will subsequently allow existing instrumentations using this technology to be contributed to the OpenTelemetry agent
• The contribution will not require any implementation effort from existing contributors (unless existing contributors want to become involved!) except for reviewing, as Elastic will make contributors available for this. (Elastic will also try to reduce reviewing effort by adding maintainers, but this will take time)
• It is mature technology and has been production tested for several years
• None of the advice class restrictions apply

Benefits to Elastic

• It will allow Elastic Java agent to converge to the OpenTelemetry agent so that Elastic can support the OpenTelemetry Java agent

Background

1. The Elastic APM Java agent is fully open source on the Apache 2.0 license
2. The agent instrumentation uses Byte Buddy delegated advice (allows breakpoint setting) - the recommended approach by the byte buddy creator Rafael Winterhalter, which reduces agent implementation complexity
3. To support delegated advice, the agent has implemented custom classloaders that enable access and reversion (the instrumentation has correct isolated access, and can be unloaded and the transformed code fully reverted)
4. To support the delegated advice being able to access the instrumented methods runtime state, the agent uses Byte Buddy to insert an invokedynamic bytecode instructions -

the invokedynamic instruction can be used to call an advice method that is loaded from a child class loader of the instrumented class' defining class loader ... this allows the agent to hide its classes from the application while providing a way to invoke the isolated methods from the application classes it instruments ... also avoids injecting the advice and helper classes into the target class loader directly

• https://www.elastic.co/de/blog/embracing-invokedynamic-to-tame-class-loaders-in-java-agents