Adds support for a custom SymbolProvider in NativeLibrary & Library by soywiz · Pull Request #1490 · java-native-access/jna

soywiz · 2022-12-10T02:19:28Z

Retake on #1483 I'm using a different branch, since I used master and that was problematic for syncing the original repo. I have also squashed all the commits into one.

This allows to do function hooking/replacement while still using direct mapping.

In addition to that, to properly using the OpenGL library on Windows, and access OpenGL >= 2.0, we need to use a custom getSymbol function: https://learn.microsoft.com/en-us/windows/win32/api/wingdi/nf-wingdi-wglgetprocaddress

In the test I'm switching cos and sin functions to show it works.

Started a thread here: https://groups.google.com/g/jna-users/c/RDt8sHAnTm0

soywiz · 2022-12-11T12:31:21Z

@matthiasblaesing can this be considered? is there something else I can do to move this forward?

matthiasblaesing · 2022-12-11T12:32:16Z

@soywiz I am/was busy with other things. I'll look into it.

soywiz · 2022-12-11T12:50:50Z

Sure. No problem and no rush. Just wanted to know if I missed something. Thanks for letting me know!

matthiasblaesing

Looks ok in general to me. What I noticed is:

This change also affects the InvocationHandler codepath, so this also needs to have a test for the non-direct invocation case.
The naming of the third parameter in SymbolProvider#getSymbolAddress makes me wonder if parent is really right here. Though I have not really reached a good conclusion
Please recheck the commit and ensure, that your real name is in the author field (git log -n 1 is your friend)

For the change in NativeLibrary - what do you think about this:

--- a/src/com/sun/jna/NativeLibrary.java
+++ b/src/com/sun/jna/NativeLibrary.java
@@ -87,17 +87,22 @@
 public class NativeLibrary implements Closeable {
 
     private static final Logger LOG = Logger.getLogger(NativeLibrary.class.getName());
-    private final static Level DEBUG_LOAD_LEVEL = DEBUG_LOAD ? Level.INFO : Level.FINE;
+    private static final Level DEBUG_LOAD_LEVEL = DEBUG_LOAD ? Level.INFO : Level.FINE;
+    private static final SymbolProvider NATIVE_SYMBOL_PROVIDER = new SymbolProvider() {
+        @Override
+        public long getSymbolAddress(long handle, String name, SymbolProvider parent) {
+            return Native.findSymbol(handle, name);
+        }
+    };
 
     private Cleaner.Cleanable cleanable;
     private long handle;
     private final String libraryName;
     private final String libraryPath;
     private final Map<String, Function> functions = new HashMap<String, Function>();
+    private final SymbolProvider symbolProvider;
     final int callFlags;
     private String encoding;
-    private SymbolProvider symbolProvider;
-    private SymbolProvider defaultSymbolProvider;
     final Map<String, ?> options;
 
     private static final Map<String, Reference<NativeLibrary>> libraries = new HashMap<String, Reference<NativeLibrary>>();
@@ -125,15 +130,11 @@
         this.callFlags = callingConvention;
         this.options = options;
         this.encoding = (String)options.get(Library.OPTION_STRING_ENCODING);
-        this.symbolProvider = (SymbolProvider)options.get(Library.OPTION_SYMBOL_PROVIDER);
-        if (this.symbolProvider != null) {
-            //noinspection Convert2Lambda
-            defaultSymbolProvider = new SymbolProvider() {
-                @Override
-                public long getSymbolAddress(long handle, String name, SymbolProvider parent) {
-                    return Native.findSymbol(handle, name);
-                }
-            };
+        SymbolProvider optionSymbolProvider = (SymbolProvider)options.get(Library.OPTION_SYMBOL_PROVIDER);
+        if (optionSymbolProvider == null) {
+            this.symbolProvider = NATIVE_SYMBOL_PROVIDER;
+        } else {
+            this.symbolProvider = optionSymbolProvider;
         }
 
         if (this.encoding == null) {
@@ -649,11 +650,7 @@
         if (handle == 0) {
             throw new UnsatisfiedLinkError("Library has been unloaded");
         }
-        if (symbolProvider != null) {
-            return symbolProvider.getSymbolAddress(handle, name, defaultSymbolProvider);
-        } else {
-            return Native.findSymbol(handle, name);
-        }
+        return this.symbolProvider.getSymbolAddress(handle, name, NATIVE_SYMBOL_PROVIDER);
     }
 
     @Override

soywiz · 2022-12-14T00:49:32Z

@matthiasblaesing applied your patch since it makes sense to me. Also added my real name in the git commit.

Regarding to "InvocationHandler codepath" I don't know what that really means. I mean: for me there are two code paths, the default one, and the one that provides a custom OPTION_SYMBOL_PROVIDER. The default one was already being tested, and the new one is also tested in the PR. And I can't see any InvocationHandler reference in the PR, so I think I need a small clarification on that.

matthiasblaesing · 2022-12-15T21:22:52Z

@matthiasblaesing applied your patch since it makes sense to me. Also added my real name in the git commit.

Regarding to "InvocationHandler codepath" I don't know what that really means. I mean: for me there are two code paths, the default one, and the one that provides a custom OPTION_SYMBOL_PROVIDER. The default one was already being tested, and the new one is also tested in the PR. And I can't see any InvocationHandler reference in the PR, so I think I need a small clarification on that.

I mean this construct (untested!):

    interface MathInterfaceWithSymbolProvider extends Library {

        public MathInterfaceWithSymbolProvider INSTANCE = Native.load(
                Platform.MATH_LIBRARY_NAME,
                MathInterfaceWithSymbolProvider.class, Collections.singletonMap(
                        Library.OPTION_SYMBOL_PROVIDER,
                        new SymbolProvider() {
                            @Override
                            public long getSymbolAddress(long handle, String name, SymbolProvider parent) {
                                if (name.equals("sin")) {
                                    return parent.getSymbolAddress(handle, "cos", null);
                                } else {
                                    return parent.getSymbolAddress(handle, "sin", null);
                                }
                            }
                        }
                )
        );

        double cos(double x);
        double sin(double x);
    }

Native#load takes an interface, that defines the mappings to the native functions as abstract methods. For this interface a java.lang.reflect.Proxy is created, which delegates invocations to an (java.lang.reflect.InvocationHandler).

The implementation can be found here:

jna/src/com/sun/jna/Library.java

Lines 117 to 268 in 9e7160e

    
               static class Handler implements InvocationHandler { 
        
                   static final Method OBJECT_TOSTRING; 
        
                   static final Method OBJECT_HASHCODE; 
        
                   static final Method OBJECT_EQUALS; 
        
                   static { 
        
                       try { 
        
                           OBJECT_TOSTRING = Object.class.getMethod("toString"); 
        
                           OBJECT_HASHCODE= Object.class.getMethod("hashCode"); 
        
                           OBJECT_EQUALS = Object.class.getMethod("equals", Object.class); 
        
                       } catch (Exception e) { 
        
                           throw new Error("Error retrieving Object.toString() method"); 
        
                       } 
        
                   } 
        
                   /** 
        
                    * FunctionInfo has to be immutable to to make the object visible 
        
                    * to other threads fully initialized. This is a prerequisite for 
        
                    * using the class in the double checked locking scenario of {@link Handler#invoke(Object, Method, Object[])} 
        
                    */ 
        
                   private static final class FunctionInfo { 
        
                       final InvocationHandler handler; 
        
                       final Function function; 
        
                       final boolean isVarArgs; 
        
                       final Object methodHandle; 
        
                       final Map<String, ?> options; 
        
                       final Class<?>[] parameterTypes; 
        
                       FunctionInfo(Object mh) { 
        
                           this.handler = null; 
        
                           this.function = null; 
        
                           this.isVarArgs = false; 
        
                           this.options = null; 
        
                           this.parameterTypes = null; 
        
                           this.methodHandle = mh; 
        
                       } 
        
                       FunctionInfo(InvocationHandler handler, Function function, Class<?>[] parameterTypes, boolean isVarArgs, Map<String, ?> options) { 
        
                           this.handler = handler; 
        
                           this.function = function; 
        
                           this.isVarArgs = isVarArgs; 
        
                           this.options = options; 
        
                           this.parameterTypes = parameterTypes; 
        
                           this.methodHandle = null; 
        
                       } 
        
                   } 
        
                   private final NativeLibrary nativeLibrary; 
        
                   private final Class<?> interfaceClass; 
        
                   // Library invocation options 
        
                   private final Map<String, Object> options; 
        
                   private final InvocationMapper invocationMapper; 
        
                   private final Map<Method, FunctionInfo> functions = new WeakHashMap<Method, FunctionInfo>(); 
        
                   public Handler(String libname, Class<?> interfaceClass, Map<String, ?> options) { 
        
                       if (libname != null && "".equals(libname.trim())) { 
        
                           throw new IllegalArgumentException("Invalid library name \"" + libname + "\""); 
        
                       } 
        
                       if (!interfaceClass.isInterface()) { 
        
                           throw new IllegalArgumentException(libname + " does not implement an interface: " + interfaceClass.getName()); 
        
                       } 
        
                       this.interfaceClass = interfaceClass; 
        
                       this.options = new HashMap<String, Object>(options); 
        
                       int callingConvention = AltCallingConvention.class.isAssignableFrom(interfaceClass) 
        
                                             ? Function.ALT_CONVENTION 
        
                                             : Function.C_CONVENTION; 
        
                       if (this.options.get(OPTION_CALLING_CONVENTION) == null) { 
        
                           this.options.put(OPTION_CALLING_CONVENTION, Integer.valueOf(callingConvention)); 
        
                       } 
        
                       if (this.options.get(OPTION_CLASSLOADER) == null) { 
        
                           this.options.put(OPTION_CLASSLOADER, interfaceClass.getClassLoader()); 
        
                       } 
        
                       this.nativeLibrary = NativeLibrary.getInstance(libname, this.options); 
        
                       invocationMapper = (InvocationMapper)this.options.get(OPTION_INVOCATION_MAPPER); 
        
                   } 
        
                   public NativeLibrary getNativeLibrary() { 
        
                       return nativeLibrary; 
        
                   } 
        
                   public String getLibraryName() { 
        
                       return nativeLibrary.getName(); 
        
                   } 
        
                   public Class<?> getInterfaceClass() { 
        
                       return interfaceClass; 
        
                   } 
        
                   @Override 
        
                   public Object invoke(Object proxy, Method method, Object[] inArgs) 
        
                       throws Throwable { 
        
                       // Intercept Object methods 
        
                       if (OBJECT_TOSTRING.equals(method)) { 
        
                           return "Proxy interface to " + nativeLibrary; 
        
                       } else if (OBJECT_HASHCODE.equals(method)) { 
        
                           return Integer.valueOf(hashCode()); 
        
                       } else if (OBJECT_EQUALS.equals(method)) { 
        
                           Object o = inArgs[0]; 
        
                           if (o != null && Proxy.isProxyClass(o.getClass())) { 
        
                               return Function.valueOf(Proxy.getInvocationHandler(o) == this); 
        
                           } 
        
                           return Boolean.FALSE; 
        
                       } 
        
                       // Using the double-checked locking pattern to speed up function calls 
        
                       FunctionInfo f = functions.get(method); 
        
                       if(f == null) { 
        
                           synchronized(functions) { 
        
                               f = functions.get(method); 
        
                               if (f == null) { 
        
                                   boolean isDefault = ReflectionUtils.isDefault(method); 
        
                                   if(! isDefault) { 
        
                                       boolean isVarArgs = Function.isVarArgs(method); 
        
                                       InvocationHandler handler = null; 
        
                                       if (invocationMapper != null) { 
        
                                           handler = invocationMapper.getInvocationHandler(nativeLibrary, method); 
        
                                       } 
        
                                       Function function = null; 
        
                                       Class<?>[] parameterTypes = null; 
        
                                       Map<String, Object> options = null; 
        
                                       if (handler == null) { 
        
                                           // Find the function to invoke 
        
                                           function = nativeLibrary.getFunction(method.getName(), method); 
        
                                           parameterTypes = method.getParameterTypes(); 
        
                                           options = new HashMap<String, Object>(this.options); 
        
                                           options.put(Function.OPTION_INVOKING_METHOD, method); 
        
                                       } 
        
                                       f = new FunctionInfo(handler, function, parameterTypes, isVarArgs, options); 
        
                                   } else { 
        
                                       f = new FunctionInfo(ReflectionUtils.getMethodHandle(method)); 
        
                                   } 
        
                                   functions.put(method, f); 
        
                               } 
        
                           } 
        
                       } 
        
                       if (f.methodHandle != null) { 
        
                           return ReflectionUtils.invokeDefaultMethod(proxy, f.methodHandle, inArgs); 
        
                       } else { 
        
                           if (f.isVarArgs) { 
        
                               inArgs = Function.concatenateVarArgs(inArgs); 
        
                           } 
        
                           if (f.handler != null) { 
        
                               return f.handler.invoke(proxy, method, inArgs); 
        
                           } 
        
                           return f.function.invoke(method, f.parameterTypes, method.getReturnType(), inArgs, f.options); 
        
                       } 
        
                   } 
        
               }

HTH
Matthias

soywiz · 2022-12-16T06:42:28Z

I see. So I guess incidencially this should be usable with the interface variant too. I have updated the title of the PR, updated the README and added a test so the scope of this PR is properly described

matthiasblaesing

Final nitpick: Please move the tests together into NativeCustomSymbolProviderTest. That way the test is focus on a single feature and not spread over two classes.

soywiz · 2022-12-18T16:41:40Z

@matthiasblaesing Done.

matthiasblaesing

Change looks good to me and Appveyor, Travis and github actions agree. Thank you.

This MR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [net.java.dev.jna:jna](https://github.com/java-native-access/jna) | compile | minor | `5.12.1` -> `5.13.0` | --- ### Release Notes <details> <summary>java-native-access/jna</summary> ### [`v5.13.0`](https://github.com/java-native-access/jna/blob/HEAD/CHANGES.md#Release-5130) [Compare Source](java-native-access/jna@5.12.1...5.13.0) \================ ## Features - [#1454](java-native-access/jna#1454): Add `c.s.j.p.win32.Psapi.QueryWorkingSetEx` and associated Types - [@crain-32](https://github.com/Crain-32). - [#1459](java-native-access/jna#1459): Add `VirtualLock` and `VirtualUnlock` in `c.s.j.p.win32.Kernel32` - [@matthiasblaesing](https://github.com/matthiasblaesing). - [#1471](java-native-access/jna#1471): Add `c.s.j.p.win32.Advapi32Util#isCurrentProcessElevated` and associated Types - [@dbwiddis](https://github.com/dbwiddis). - [#1474](java-native-access/jna#1474): Add `c.s.j.p.win32.WbemCli#IWbemClassObject.IWbemQualifierSet`, `IWbemServices.GetObject`, `IWbemContext.SetValue` and associated methods - [@rchateauneu](https://github.com/rchateauneu). - [#1482](java-native-access/jna#1482): Add multilingual support of `Kernel32Util.formatMessage` - [@overpathz](https://github.com/overpathz). - [#1490](java-native-access/jna#1490): Adds support for a custom `SymbolProvider` in `NativeLibrary` & `Library` - [@soywiz](https://github.com/soywiz). - [#1491](java-native-access/jna#1491): Update libffi to v3.4.4 - [@matthiasblaesing](https://github.com/matthiasblaesing). - [#1487](java-native-access/jna#1487): Add 'uses' information to OSGI metadata in MANIFEST.MF to improve stability of package resolution - [@sratz](https://github.com/sratz). ## Bug Fixes - [#1452](java-native-access/jna#1452): Fix memory allocation/handling for error message generation in native library code (`dispatch.c`) - [@matthiasblaesing](https://github.com/matthiasblaesing). - [#1460](java-native-access/jna#1460): Fix win32 variant date conversion in DST offest window and with millisecond values - [@eranl](https://github.com/eranl). - [#1472](java-native-access/jna#1472): Fix incorrect bitmask in `c.s.j.Pointer#createConstant(int)` - [@dbwiddis](https://github.com/dbwiddis). - [#1481](java-native-access/jna#1481): Fix NPE in NativeLibrary when unpacking from classpath is disabled - [@trespasserw](https://github.com/trespasserw). - [#1489](java-native-access/jna#1489): Fixes typo in `OpenGL32Util#wglGetProcAddress`, instead of parameter `procName` the hardcoded value `wglEnumGpusNV` was used - [@soywiz](https://github.com/soywiz). </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

soywiz force-pushed the feature/direct.mapping.custom.symbol.provider branch from d3198b2 to fcd4408 Compare December 10, 2022 02:21

soywiz mentioned this pull request Dec 11, 2022

DirectGL on Win32. Hugely reduces OpenGL overhead on windows JVM korlibs/korge#1140

Closed

matthiasblaesing requested changes Dec 13, 2022

View reviewed changes

soywiz force-pushed the feature/direct.mapping.custom.symbol.provider branch 2 times, most recently from 2b4c6f5 to df762de Compare December 14, 2022 00:42

soywiz force-pushed the feature/direct.mapping.custom.symbol.provider branch 2 times, most recently from e9f597f to de7dd2e Compare December 16, 2022 06:41

soywiz changed the title ~~Adds support for a custom SymbolProvider in NativeLibrary~~ Adds support for a custom SymbolProvider in NativeLibrary & Library Dec 16, 2022

soywiz force-pushed the feature/direct.mapping.custom.symbol.provider branch from de7dd2e to cd8474c Compare December 16, 2022 08:42

matthiasblaesing reviewed Dec 18, 2022

View reviewed changes

Adds support for a custom SymbolProvider in NativeLibrary & Library

5a6a9f2

soywiz force-pushed the feature/direct.mapping.custom.symbol.provider branch from cd8474c to 5a6a9f2 Compare December 18, 2022 16:26

matthiasblaesing approved these changes Dec 20, 2022

View reviewed changes

matthiasblaesing merged commit db1b553 into java-native-access:master Dec 20, 2022

soywiz mentioned this pull request Jan 14, 2023

DirectGL on Win32. Hugely reduces OpenGL overhead on windows JVM: Use JNA 5.13 Library.OPTION_SYMBOL_PROVIDER to implement fast direct OpenGL mapping on Windows korlibs/korge#1195

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support for a custom SymbolProvider in NativeLibrary & Library#1490

Adds support for a custom SymbolProvider in NativeLibrary & Library#1490
matthiasblaesing merged 1 commit intojava-native-access:masterfrom
korlibs:feature/direct.mapping.custom.symbol.provider

soywiz commented Dec 10, 2022 •

edited

Loading

Uh oh!

soywiz commented Dec 11, 2022

Uh oh!

matthiasblaesing commented Dec 11, 2022

Uh oh!

soywiz commented Dec 11, 2022

Uh oh!

matthiasblaesing left a comment •

edited

Loading

Uh oh!

soywiz commented Dec 14, 2022

Uh oh!

matthiasblaesing commented Dec 15, 2022

Uh oh!

soywiz commented Dec 16, 2022 •

edited

Loading

Uh oh!

matthiasblaesing left a comment

Uh oh!

soywiz commented Dec 18, 2022 •

edited

Loading

Uh oh!

matthiasblaesing left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

soywiz commented Dec 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

soywiz commented Dec 11, 2022

Uh oh!

matthiasblaesing commented Dec 11, 2022

Uh oh!

soywiz commented Dec 11, 2022

Uh oh!

matthiasblaesing left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soywiz commented Dec 14, 2022

Uh oh!

matthiasblaesing commented Dec 15, 2022

Uh oh!

soywiz commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matthiasblaesing left a comment

Choose a reason for hiding this comment

Uh oh!

soywiz commented Dec 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matthiasblaesing left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

soywiz commented Dec 10, 2022 •

edited

Loading

matthiasblaesing left a comment •

edited

Loading

soywiz commented Dec 16, 2022 •

edited

Loading

soywiz commented Dec 18, 2022 •

edited

Loading