Skip to content

access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability#2113

Merged
davidzollo merged 3 commits intoapache:devfrom
tisonkun:GH-2111
Mar 9, 2020
Merged

access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability#2113
davidzollo merged 3 commits intoapache:devfrom
tisonkun:GH-2111

Conversation

@tisonkun
Copy link
Copy Markdown
Member

@tisonkun tisonkun commented Mar 8, 2020

What is the purpose of the pull request

Current implementation relies on sun.misc.JavaIOFileDescriptorAccess
which is only accessible on oraclejdk8.

Basically the demand is getting & setting handle field of FileDescriptor,
so we can directly do that with reflection.

Though, I suspect the necessity we introduce ProcessImplForWin32. Maybe
we could have a better way to support worker server to run bat script.

Brief change log

Replace JavaIOFileDescriptorAccess with reflections to FileDescriptor.

Verify this pull request

This pull request is already covered by existing tests, such as

  • ProcessImplForWin32Test

This closes #2111 .

FD_ACCESS = FileDescriptor.class.getDeclaredField("fd");
FD_ACCESS.setAccessible(true);
} catch (NoSuchFieldException e) {
throw new RuntimeException(e);
Copy link
Copy Markdown
Member Author

@tisonkun tisonkun Mar 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unfamiliar with exception strategy in DolphinScheduler so it would be better our committers can check whether the exception thrown obeys conventions.

try {
FD_ACCESS.set(obj, fd);
} catch (IllegalAccessException e) {
throw new RuntimeException(e);
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

try {
return (Long) FD_ACCESS.get(obj);
} catch (IllegalAccessException e) {
throw new RuntimeException(e);
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@codecov-io
Copy link
Copy Markdown

Codecov Report

❗ No coverage uploaded for pull request base (dev@0e1dd8d). Click here to learn what that means.
The diff coverage is 13.04%.

Impacted file tree graph

@@          Coverage Diff           @@
##             dev    #2113   +/-   ##
======================================
  Coverage       ?   29.64%           
  Complexity     ?     1607           
======================================
  Files          ?      336           
  Lines          ?    17833           
  Branches       ?     2250           
======================================
  Hits           ?     5287           
  Misses         ?    11990           
  Partials       ?      556
Impacted Files Coverage Δ Complexity Δ
...uler/common/utils/process/ProcessImplForWin32.java 1.66% <13.04%> (ø) 1 <1> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e1dd8d...31be59d. Read the comment docs.

@tisonkun
Copy link
Copy Markdown
Member Author

tisonkun commented Mar 8, 2020

This PR contains only refactor changes, I don't think any further tests should be added. Please take a look.

Copy link
Copy Markdown
Member

@liwenhe1993 liwenhe1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Copy Markdown
Member

@liwenhe1993 liwenhe1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UT failed in Windows.

Copy link
Copy Markdown
Member

@liwenhe1993 liwenhe1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please run org.apache.dolphinscheduler.common.utils.process.ProcessBuilderForWin32Test.

@tisonkun
Copy link
Copy Markdown
Member Author

tisonkun commented Mar 8, 2020

@liwenhe1993 could you share the error message and check that it is running with this patch? I cannot reproduce and it seems our ci system doesn't complain about it or report something helps.

@liwenhe1993
Copy link
Copy Markdown
Member

[ERROR] 2020-03-08 16:59:27.524 org.apache.dolphinscheduler.common.utils.process.ProcessBuilderForWin32Test:[53] - Can not set int field java.io.FileDescriptor.fd to java.lang.Long
[ERROR] 2020-03-08 16:59:27.570 org.apache.dolphinscheduler.common.utils.process.ProcessBuilderForWin32Test:[66] - Can not set int field java.io.FileDescriptor.fd to java.lang.Long

@tisonkun tisonkun changed the title access field fd of FileDescriptor in ProcessImplForWin32 by reflection for portability access field handle of FileDescriptor in ProcessImplForWin32 by reflection for portability Mar 8, 2020
…ction for portability

Current implementation relies on `sun.misc.JavaIOFileDescriptorAccess`
which is only accessible on oraclejdk8.

Basically the demand is getting & setting `handle` field of
`FileDescriptor`, so we can directly do that with reflection.

Though, I suspect the necessity we introduce ProcessImplForWin32. Maybe
we could have a better way to support worker server to run bat script.
@tisonkun
Copy link
Copy Markdown
Member Author

tisonkun commented Mar 8, 2020

@liwenhe1993 thanks for your information. It seems I misspell the field name in the patch. Could you please verify the latest patch?

@liwenhe1993
Copy link
Copy Markdown
Member

liwenhe1993 commented Mar 8, 2020

Yeah, I will do, but UT run failed.

Tests in error: 
  ShellTaskTest.testHandleForWindows:179 NullPointer

Maybe ocurred error by:

static {
    try {
        FD_HANDLE = FileDescriptor.class.getDeclaredField("handle");
        FD_HANDLE.setAccessible(true);
    } catch (NoSuchFieldException e) {
        throw new RuntimeException(e);
    }
}

So, UT should be updated.

@tisonkun
Copy link
Copy Markdown
Member Author

tisonkun commented Mar 8, 2020

@liwenhe1993 Thanks for your updated. I don't have a debug environment now. It is hard to debug with single NPE message, could you checkout the patch and fix the issue? The reflection direction looks viable and I think the authority of the patch could then belong to you :-)

@liwenhe1993
Copy link
Copy Markdown
Member

org.apache.dolphinscheduler.server.worker.task.shell.ShellTaskTest file should be updated.

Code:

    /**
     * Method: handle() for Windows
     */
    @Test
    public void testHandleForWindows() throws Exception {
        try {
            PowerMockito.when(OSUtils.isWindows()).thenReturn(true);
            shellTask.handle();
            Assert.assertTrue(true);
        } catch (Error | Exception e) {
            // if (!e.getMessage().contains("process error . exitCode is :  -1")) {
                logger.error(e.getMessage());
            // }
        }
    }

UT can run successfully when you updated this file.

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Mar 9, 2020

SonarCloud Quality Gate failed.

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 3 Security Hotspots to review)
Code Smell A 115 Code Smells

17.1% 17.1% Coverage
1.0% 1.0% Duplication

Copy link
Copy Markdown
Member

@liwenhe1993 liwenhe1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@dailidong

@davidzollo
Copy link
Copy Markdown
Contributor

good job

thanks for your contribution, looking forward to your next contribution

@davidzollo davidzollo merged commit 9224b49 into apache:dev Mar 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] sun.misc.JavaIOFileDescriptorAccess is not portable

4 participants