Provides built-in GraalVM Reachability Metadata for core features of ShardingSphere JDBC and adds nativeTest-related unit test subsets#29000
Merged
Conversation
1fe94c5 to
c0d6d1d
Compare
linghengqian
commented
Nov 10, 2023
Member
Author
There was a problem hiding this comment.
-
Update in 2023.11.12: maybe fixes in #29009 .
-
This PR is still pending. Some classes of ShardingSphere need to be reconstructed for file system-related operations, which I hope to handle in another PR. For example
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader. -
The current execution of nativeTest using
./mvnw -PnativeTestInShardingSphere -T1C -e clean testwill cause the following Error. This is caused by GraalVM's inherent design of the file system, which involves the background of oracle/graal#7682 .
Error Log
[2/8] Performing analysis... [********] (148.8s @ 2.93GB)
29,136 (81.15%) of 35,902 types reachable
49,445 (57.37%) of 86,186 fields reachable
149,418 (58.31%) of 256,235 methods reachable
7,787 types, 979 fields, and 6,877 methods registered for reflection
68 types, 82 fields, and 57 methods registered for JNI access
4 native libraries: dl, pthread, rt, z
[3/8] Building universe... (16.4s @ 3.28GB)
[4/8] Parsing methods... [***] (9.3s @ 2.32GB)
[5/8] Inlining methods... [****] (4.0s @ 2.39GB)
[6/8] Compiling methods... [********] (67.8s @ 2.69GB)
[7/8] Layouting methods... [***] (10.5s @ 2.88GB)
[8/8] Creating image... [****] (15.9s @ 3.58GB)
73.72MB (54.04%) for code area: 94,894 compilation units
57.31MB (42.01%) for image heap: 576,269 objects and 277 resources
5.38MB ( 3.95%) for other data
136.41MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
13.33MB calcite-core-1.35.0.jar 16.03MB byte[] for code metadata
13.17MB java.base 7.42MB byte[] for java.lang.String
5.79MB h2-2.2.224.jar 7.18MB java.lang.Class
5.66MB java.xml 5.80MB java.lang.String
5.29MB groovy-4.0.10.jar 4.52MB byte[] for general heap data
3.89MB org.apache.shardingsphere.sql.parser.autogen 2.45MB com.oracle.svm.core.hub.DynamicHubCompanion
2.96MB java.desktop 2.12MB byte[] for embedded resources
1.91MB jackson-databind-2.14.0.jar 1.74MB byte[] for reflection metadata
1.54MB avatica-core-1.23.0.jar 1.22MB java.lang.String[]
1.43MB janino-3.1.9.jar 1.05MB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
18.06MB for 727 more packages 7.56MB for 5238 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
AWT: Use the tracing agent to collect metadata for AWT.
HEAP: Set max heap for improved and more predictable memory usage.
CPU: Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
36.5s (12.0% of total time) in 181 GCs | Peak RSS: 5.27GB | CPU load: 4.20
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libawt.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libawt_headless.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libawt_xawt.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libfontmanager.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libjava.so (jdk_library_shim)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libjavajpeg.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libjvm.so (jdk_library_shim)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/liblcms.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/native-tests (executable)
========================================================================================================================
Finished generating 'native-tests' in 5m 1s.
[INFO] Executing: /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/native-tests --xml-output-dir /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/native-test-reports -Djunit.platform.listeners.uid.tracking.output.dir=/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/test-ids
JUnit Platform on Native Image - report
----------------------------------------
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - This is logback-classic version 1.3.11
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - No custom configurators were discovered as a service.
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Trying to configure with ch.qos.logback.classic.joran.SerializedModelConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Constructed configurator of type class ch.qos.logback.classic.joran.SerializedModelConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Could NOT find resource [logback-test.scmo]
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Could NOT find resource [logback.scmo]
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - ch.qos.logback.classic.joran.SerializedModelConfigurator.configure() call lasted 0 milliseconds. ExecutionStatus=INVOKE_NEXT_IF_ANY
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Trying to configure with ch.qos.logback.classic.util.DefaultJoranConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Constructed configurator of type class ch.qos.logback.classic.util.DefaultJoranConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Could NOT find resource [logback-test.xml]
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Found resource [logback.xml] at [resource:/logback.xml]
18:08:36,098 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@6b3ecc43 - Resource [logback.xml] occurs multiple times on the classpath.
18:08:36,098 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@6b3ecc43 - Resource [logback.xml] occurs at [resource:/logback.xml#1]
18:08:36,098 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@6b3ecc43 - Resource [logback.xml] occurs at [resource:/logback.xml]
18:08:36,098 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@3f7d7df1 - URL [resource:/logback.xml] is not of type file
18:08:36,099 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - <level> element is deprecated. Near [level] on line 31
18:08:36,099 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - Please use "level" attribute within <logger> or <root> elements instead.
18:08:36,099 |-INFO in ch.qos.logback.classic.model.processor.ContextNameModelHandler - Setting logger context name as [shardingsphere-infra-nativetest]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
18:08:36,099 |-INFO in ch.qos.logback.classic.model.processor.LevelModelHandler - ROOT level set to INFO
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@12459199 - End of configuration.
18:08:36,099 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@85c50cc - Registering current configuration as safe fallback point
18:08:36,099 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - ch.qos.logback.classic.util.DefaultJoranConfigurator.configure() call lasted 1 milliseconds. ExecutionStatus=DO_NOT_INVOKE_NEXT_IF_ANY
org.apache.shardingsphere.infra.nativetest.jdbc.features.ShardingTest > testShardingInLocalTransactions() FAILED
org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest > testReadWriteSplittingInLocalTransactions() FAILED
org.apache.shardingsphere.infra.nativetest.jdbc.features.EncryptTest > testEncryptInLocalTransactions() FAILED
org.apache.shardingsphere.infra.nativetest.jdbc.features.MaskTest > testMaskInLocalTransactions() FAILED
org.apache.shardingsphere.infra.nativetest.jdbc.features.ShadowTest > testShadowInLocalTransactions() FAILED
Failures (5):
JUnit Jupiter:ShardingTest:testShardingInLocalTransactions()
MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.ShardingTest', methodName = 'testShardingInLocalTransactions', methodParameterTypes = '']
=> java.nio.file.FileSystemNotFoundException
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
java.base@17.0.9/java.nio.file.Path.of(Path.java:208)
java.base@17.0.9/java.nio.file.Paths.get(Paths.java:98)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
[...]
JUnit Jupiter:ReadWriteSplittingTest:testReadWriteSplittingInLocalTransactions()
MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest', methodName = 'testReadWriteSplittingInLocalTransactions', methodParameterTypes = '']
=> java.lang.NullPointerException
java.base@17.0.9/java.io.Reader.<init>(Reader.java:168)
java.base@17.0.9/java.io.InputStreamReader.<init>(InputStreamReader.java:112)
org.apache.shardingsphere.infra.nativetest.FileTestUtils.readInputStream(FileTestUtils.java:52)
org.apache.shardingsphere.infra.nativetest.FileTestUtils.readFromFileURLString(FileTestUtils.java:46)
org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest.testReadWriteSplittingInLocalTransactions(ReadWriteSplittingTest.java:53)
java.base@17.0.9/java.lang.reflect.Method.invoke(Method.java:568)
java.base@17.0.9/java.util.ArrayList.forEach(ArrayList.java:1511)
java.base@17.0.9/java.util.ArrayList.forEach(ArrayList.java:1511)
JUnit Jupiter:EncryptTest:testEncryptInLocalTransactions()
MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.EncryptTest', methodName = 'testEncryptInLocalTransactions', methodParameterTypes = '']
=> java.nio.file.FileSystemNotFoundException
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
java.base@17.0.9/java.nio.file.Path.of(Path.java:208)
java.base@17.0.9/java.nio.file.Paths.get(Paths.java:98)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
[...]
JUnit Jupiter:MaskTest:testMaskInLocalTransactions()
MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.MaskTest', methodName = 'testMaskInLocalTransactions', methodParameterTypes = '']
=> java.nio.file.FileSystemNotFoundException
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
java.base@17.0.9/java.nio.file.Path.of(Path.java:208)
java.base@17.0.9/java.nio.file.Paths.get(Paths.java:98)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
[...]
JUnit Jupiter:ShadowTest:testShadowInLocalTransactions()
MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.ShadowTest', methodName = 'testShadowInLocalTransactions', methodParameterTypes = '']
=> java.nio.file.FileSystemNotFoundException
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
java.base@17.0.9/java.nio.file.Path.of(Path.java:208)
java.base@17.0.9/java.nio.file.Paths.get(Paths.java:98)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
[...]
Test run finished after 98 ms
[ 6 containers found ]
[ 0 containers skipped ]
[ 6 containers started ]
[ 0 containers aborted ]
[ 6 containers successful ]
[ 0 containers failed ]
[ 5 tests found ]
[ 0 tests skipped ]
[ 5 tests started ]
[ 0 tests aborted ]
[ 0 tests successful ]
[ 5 tests failed ]
This was referenced Nov 11, 2023
c9aac11 to
09dcdc8
Compare
09dcdc8 to
c8e5545
Compare
linghengqian
commented
Nov 12, 2023
Member
Author
There was a problem hiding this comment.
- Update in 2023.11.12: Fixes in #29014 .
- Blocked by oracle/graal#7804 .
6 tasks
c8e5545 to
7427de4
Compare
Closed
6 tasks
ecebd31 to
65e3ae0
Compare
…ShardingSphere JDBC and adds nativeTest-related unit test subsets
65e3ae0 to
138fb68
Compare
6 tasks
RaigorJiang
requested changes
Nov 14, 2023
strongduanmu
requested changes
Nov 14, 2023
RaigorJiang
reviewed
Nov 14, 2023
RaigorJiang
approved these changes
Nov 14, 2023
strongduanmu
approved these changes
Nov 16, 2023
strongduanmu
left a comment
Member
There was a problem hiding this comment.
Thank you for your perfect pr. @linghengqian Merged.
This was referenced Nov 21, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #21347.
Changes proposed in this pull request:
org.apache.shardingsphere:shardingsphere-infra-reachability-metadatafor hosting the GraalVM Reachability Metadata of ShardingSphere itself and some third-party libraries. This involves me personally not having much time in the central repository of GraalVM Reachability Metadata to handle PRs requested by the GraalVM Committer.org.apache.shardingsphere:shardingsphere-infra-reachability-metadatawill become an internal dependency oforg.apache.shardingsphere:shardingsphere-jdbc-coreto be passed out.org.apache.shardingsphere:shardingsphere-infra-nativetest.org.apache.shardingsphere:shardingsphere-infra-nativetestis adapted from the ShardingSphere Example. Use YAML instead of Java API to define ShardingSphere's DataSource because this is closest to the actual application scenario - this is the actual meaning of native Test. We only focus on core features in these unit test.org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTestseems to have hit a bug in ReadWrite Splitting. I will discuss the phenomenon I saw in a separate issue later.generateMetadataandnativeTestInShardingSphere. They both only managepluginManagement. Submodules that need to use the relevant Profile will automatically introduce the relevantplugin. This involves best practices defined by GraalVM Native Build Tools. Refer to Maven Native plugin fails on POM packaging projects graalvm/native-build-tools#490 .Before committing this PR, I'm sure that I have checked the following options:
./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.