Skip to content

Enable Safe Composition of Metadata for ShardingSphere Proxy Native#33179

Merged
strongduanmu merged 1 commit into
apache:masterfrom
linghengqian:scm-re
Oct 11, 2024
Merged

Enable Safe Composition of Metadata for ShardingSphere Proxy Native#33179
strongduanmu merged 1 commit into
apache:masterfrom
linghengqian:scm-re

Conversation

@linghengqian

@linghengqian linghengqian commented Oct 9, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

For #29052.

Changes proposed in this pull request:

To make usage of third-party reachability metadata safe we must guarantee that metadata addition can't break existing programs. This is currently not the case

  • Adds GraalVM Reachability Metadata for ch.qos.logback:logback-core:1.2.13

  • Adds GraalVM Reachability Metadata for com.alibaba:druid:1.2.7

  • Adds GraalVM Reachability Metadata for com.github.docker-java:docker-java:3.4.0

  • Adds GraalVM Reachability Metadata for com.mysql:mysql-connector-j:8.3.0

  • Adds GraalVM Reachability Metadata for io.grpc:grpc-netty:1.65.1

  • Adds GraalVM Reachability Metadata for io.vertx:vertx-core:4.5.1

  • Adds GraalVM Reachability Metadata for io.netty:netty-transport-classes-epoll:4.1.112.Final

  • Adds GraalVM Reachability Metadata for org.apache.groovy:groovy:4.0.22

  • Adds GraalVM Reachability Metadata for org.apache.hadoop:hadoop-common:3.3.6

  • Adds GraalVM Reachability Metadata for org.apiguardian:apiguardian-api:1.1.2

  • Adds GraalVM Reachability Metadata for org.jboss.logging:jboss-logging:3.2.1.Final

  • Adds GraalVM Reachability Metadata for org.jboss.narayana.jta:jta:5.12.7.Final

  • Adds GraalVM Reachability Metadata for org.junit.jupiter:junit-jupiter:5.11.1

  • Adds GraalVM Reachability Metadata for org.testcontainers:junit-jupiter:1.20.2

  • Adds GraalVM Reachability Metadata for com.typesafe:config:1.2.1

  • Adds GraalVM Reachability Metadata for com.google.protobuf:protobuf-java:3.21.12. This sounds like what I missed in Add support for com.google.protobuf:protobuf-java-util:3.21.12 oracle/graalvm-reachability-metadata#167 .

  • Adds GraalVM Reachability Metadata for org.springframework:spring-jcl:5.3.27

  • Adds GraalVM Reachability Metadata for com.clickhouse:clickhouse-jdbc:0.6.3

  • Updates documentation. According to Unable to apply parallel builds to thread-unsafe native-maven-plugin graalvm/native-build-tools#619 , GraalVM Native Build Tools 0.10.3 no longer prevents parallel builds of Maven.

  • As for [GR-49770] Safe Composition of Metadata  oracle/graal#5173, the SCM proposal involving buildArg entered GraalVM CE For JDK22 as an experimental option and became an official option on GraalVM CE For JDK23. The SCM proposal brings more readable Error Logs, which is definitely beneficial to contributors. The Error Logs brought by the SCM proposal are as follows.

-H:+UnlockExperimentalVMOptions -H:ThrowMissingRegistrationErrors= -H:MissingRegistrationReportingMode=Warn

com.oracle.svm.core.jdk.resources.MissingResourceRegistrationError: The program tried to access the resource at path 

   META-INF/services/javax.xml.parsers.SAXParserFactory

 without it being registered as reachable. Add it to the resource metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#resources-and-resource-bundles for help
  java.base@22.0.2/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1387)
  java.xml@22.0.2/javax.xml.parsers.FactoryFinder$1.run(FactoryFinder.java:255)
  java.base@22.0.2/java.security.AccessController.executePrivileged(AccessController.java:128)
  java.base@22.0.2/java.security.AccessController.doPrivileged(AccessController.java:319)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access method

   ch.qos.logback.core.encoder.Encoder#valueOf(java.lang.String)

 without it being registered for runtime reflection. Add ch.qos.logback.core.encoder.Encoder#valueOf(java.lang.String) to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@22.0.2/java.lang.Class.getMethod(DynamicHub.java:1070)
  ch.qos.logback.core.joran.util.StringToObjectConverter.getValueOfMethod(StringToObjectConverter.java:101)
  ch.qos.logback.core.joran.util.StringToObjectConverter.followsTheValueOfConvention(StringToObjectConverter.java:110)
  ch.qos.logback.core.joran.util.StringToObjectConverter.canBeBuiltFromSimpleString(StringToObjectConverter.java:39)
  • On ShardingSphere's own GRM side, remove {"excludeClasses": "java.**"}. Because this type of GRM begins to be required by Safe Composition of Metadata. There are obviously too many java.** GRM JSONs that need to be defined.
    {"excludeClasses": "java.**"},
    {"includeClasses": "java.io.BufferedWriter"},
    {"includeClasses": "java.io.Closeable"},
    {"includeClasses": "java.io.Flushable"},
    {"includeClasses": "java.io.InputStream"},
    {"includeClasses": "java.io.OutputStream"},
    {"includeClasses": "java.io.Reader"},
    {"includeClasses": "java.io.Writer"},
    {"includeClasses": "java.lang.Appendable"},
    {"includeClasses": "java.lang.AutoCloseable"},
    {"includeClasses": "java.lang.Boolean"},
    {"includeClasses": "java.lang.ClassLoader"},
    {"includeClasses": "java.lang.CharSequence"},
    {"includeClasses": "java.lang.Comparable"},
    {"includeClasses": "java.lang.Integer"},
    {"includeClasses": "java.lang.Iterable"},
    {"includeClasses": "java.lang.Long"},
    {"includeClasses": "java.lang.Number"},
    {"includeClasses": "java.lang.Process"},
    {"includeClasses": "java.lang.Readable"},
    {"includeClasses": "java.lang.constant.ConstantDesc"},
    {"includeClasses": "java.lang.reflect.AnnotatedElement"},
    {"includeClasses": "java.lang.management.RuntimeMXBean"},
    {"includeClasses": "java.util.AbstractCollection"},
    {"includeClasses": "java.util.AbstractMap"},
    {"includeClasses": "java.util.Collection"},
    {"includeClasses": "java.util.Deque"},
    {"includeClasses": "java.util.Enumeration"},
    {"includeClasses": "java.util.HashSet"},
    {"includeClasses": "java.util.Iterator"},
    {"includeClasses": "java.util.LinkedHashSet"},
    {"includeClasses": "java.util.List"},
    {"includeClasses": "java.util.Map"},
    {"includeClasses": "java.util.NavigableSet"},
    {"includeClasses": "java.util.Properties"},
    {"includeClasses": "java.util.Set"},
    {"includeClasses": "java.util.SortedMap"},
    {"includeClasses": "java.util.SortedSet"},
    {"includeClasses": "java.util.Timer"},
    {"includeClasses": "java.util.concurrent.BlockingQueue"},
    {"includeClasses": "java.util.concurrent.atomic.LongAdder"}
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.util.Collection.getMethods()

 without it being registered for runtime reflection. Add java.util.Collection.getMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@22.0.2/java.lang.Class.getMethods(DynamicHub.java:1022)
  org.codehaus.groovy.reflection.stdclasses.CachedSAMClass.getSAMMethod(CachedSAMClass.java:172)
  org.codehaus.groovy.reflection.ClassInfo.isSAM(ClassInfo.java:388)
  org.codehaus.groovy.reflection.ClassInfo.createCachedClass(ClassInfo.java:378)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.util.List.getMethods()

 without it being registered for runtime reflection. Add java.util.List.getMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@22.0.2/java.lang.Class.getMethods(DynamicHub.java:1022)
  org.codehaus.groovy.reflection.stdclasses.CachedSAMClass.getSAMMethod(CachedSAMClass.java:172)
  org.codehaus.groovy.reflection.ClassInfo.isSAM(ClassInfo.java:388)
  org.codehaus.groovy.reflection.ClassInfo.createCachedClass(ClassInfo.java:378)

Before committing this PR, I'm sure that I have checked the following options:

  • My code follows the code of conduct of this project.
  • I have self-reviewed the commit code.
  • I have (or in comment I request) added corresponding labels for the pull request.
  • I have passed maven check locally : ./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.
  • I have made corresponding changes to the documentation.
  • I have added corresponding unit tests for my changes.

@linghengqian linghengqian force-pushed the scm-re branch 8 times, most recently from 8018a74 to d4cd0ac Compare October 10, 2024 08:36
linghengqian
linghengqian commented Oct 10, 2024
View reviewed changes

@linghengqian linghengqian left a comment •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • This PR became quite complicated because the Warning Log even included GraalVM CE's internal missing GRM, which definitely involved multiple potential GraalVM bugs. Before supporting compiling shardingsphere project with GraalVM CE For JDK23, I needed to change -H:ThrowMissingRegistrationErrors= to -H:ThrowMissingRegistrationErrors=org.apache.shardingsphere to avoid a wider range of impacts.
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   com.oracle.svm.core.genscavenge.HeapImplMemoryMXBean.getConstructors()

 without it being registered for runtime reflection. Add com.oracle.svm.core.genscavenge.HeapImplMemoryMXBean.getConstructors() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@22.0.2/java.lang.Class.getConstructors(DynamicHub.java:1028)
  java.management@22.0.2/com.sun.jmx.mbeanserver.MBeanIntrospector.findConstructors(MBeanIntrospector.java:461)
  java.management@22.0.2/com.sun.jmx.mbeanserver.MBeanIntrospector.getClassMBeanInfo(MBeanIntrospector.java:432)
  java.management@22.0.2/com.sun.jmx.mbeanserver.MBeanIntrospector.getMBeanInfo(MBeanIntrospector.java:391)
  • The path scanned by seata client is also suspicious. I don't believe the seata client uses the registry file directly.
com.oracle.svm.core.jdk.resources.MissingResourceRegistrationError: The program tried to access the resource at path 

   registry

 without it being registered as reachable. Add it to the resource metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#resources-and-resource-bundles for help
  java.base@22.0.2/java.lang.ClassLoader.getResource(ClassLoader.java:1407)
  org.apache.seata.config.FileConfiguration.getFileFromClasspath(FileConfiguration.java:194)
  org.apache.seata.config.FileConfiguration.getConfigFile(FileConfiguration.java:144)
  org.apache.seata.config.FileConfiguration.<init>(FileConfiguration.java:108)

@linghengqian linghengqian marked this pull request as ready for review October 10, 2024 10:45
@linghengqian linghengqian mentioned this pull request Oct 10, 2024
Open
@linghengqian linghengqian force-pushed the scm-re branch 3 times, most recently from 4e0fc60 to b66fc2d Compare October 10, 2024 16:04
@linghengqian linghengqian mentioned this pull request Oct 11, 2024
Closed
@strongduanmu strongduanmu added this to the 5.5.1 milestone Oct 11, 2024
@strongduanmu strongduanmu merged commit 2b6fcca into apache:master Oct 11, 2024
@linghengqian linghengqian deleted the scm-re branch October 11, 2024 07:11
@linghengqian linghengqian mentioned this pull request Oct 11, 2024
Open
54 tasks
@linghengqian linghengqian mentioned this pull request Nov 1, 2024
Closed
@linghengqian linghengqian mentioned this pull request Nov 11, 2024
Closed
@linghengqian linghengqian mentioned this pull request Jan 20, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@strongduanmu strongduanmu strongduanmu approved these changes

Assignees

No one assigned

Labels

in: proxy native type: refactor

Projects

None yet

Milestone

5.5.1

Development

Successfully merging this pull request may close these issues.

2 participants

@linghengqian @strongduanmu