Skip to content

Fix descriptor bugs and support dup() on regular files for Python#1257

Merged
stevenengler merged 4 commits intoshadow:devfrom
stevenengler:python
Apr 5, 2021
Merged

Fix descriptor bugs and support dup() on regular files for Python#1257
stevenengler merged 4 commits intoshadow:devfrom
stevenengler:python

Conversation

@stevenengler
Copy link
Copy Markdown
Contributor

@stevenengler stevenengler commented Apr 3, 2021

Fixes a few descriptor bugs and implements dup() for regular files, which is required for Python. (Python uses dup() on stdin, stdout, and stderr to check if they are valid descriptors. See create_stdio() in CPython.)

Shadow is still missing syscalls to use things like the built-in http server (I see futex errors, missing socket options, and IPv6 issues). Python also make heavy use of select(), so things like time.sleep() don't work in Shadow.

options:
  stoptime: 10
topology:
- graphml: |
    <graphml xmlns="http://graphml.graphdrawing.org/xmlns" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
      <key attr.name="packetloss" attr.type="double" for="edge" id="d4" />
      <key attr.name="latency" attr.type="double" for="edge" id="d3" />
      <key attr.name="bandwidthup" attr.type="int" for="node" id="d2" />
      <key attr.name="bandwidthdown" attr.type="int" for="node" id="d1" />
      <key attr.name="countrycode" attr.type="string" for="node" id="d0" />
      <graph edgedefault="undirected">
        <node id="poi-1">
          <data key="d0">US</data>
          <data key="d1">10240</data>
          <data key="d2">10240</data>
        </node>
        <edge source="poi-1" target="poi-1">
          <data key="d3">50.0</data>
          <data key="d4">0.0</data>
        </edge>
      </graph>
    </graphml>
plugins:
- id: python
  path: /usr/bin/python3
hosts:
- id: test
  quantity: 1
  processes:
  - plugin: python
    starttime: 1
    arguments: -c print(1234)

@stevenengler stevenengler added the Component: Main Composing the core Shadow executable label Apr 3, 2021
@stevenengler stevenengler self-assigned this Apr 3, 2021
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 3, 2021

Codecov Report

Merging #1257 (8f547cc) into dev (22d4722) will decrease coverage by 0.03%.
The diff coverage is 46.87%.

❗ Current head 8f547cc differs from pull request most recent head 28c56e7. Consider uploading reports for the commit 28c56e7 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##              dev    #1257      +/-   ##
==========================================
- Coverage   55.68%   55.64%   -0.04%     
==========================================
  Files         142      142              
  Lines       20508    20562      +54     
  Branches     5031     5053      +22     
==========================================
+ Hits        11419    11442      +23     
- Misses       5984     5992       +8     
- Partials     3105     3128      +23     
Flag Coverage Δ
tests 55.64% <46.87%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/main/host/syscall/file.c 23.41% <0.00%> (ø)
src/main/host/syscall/fileat.c 9.65% <0.00%> (ø)
src/main/host/syscall/timerfd.c 36.11% <0.00%> (ø)
src/test/file/test_file.c 49.45% <26.31%> (-2.67%) ⬇️
src/main/host/syscall/unistd.c 44.89% <47.05%> (+0.39%) ⬆️
src/main/host/descriptor/file.c 32.51% <61.90%> (+1.28%) ⬆️
src/main/host/descriptor/descriptor.c 73.43% <100.00%> (+0.20%) ⬆️
src/main/host/process.c 70.29% <100.00%> (ø)
src/main/core/worker.c 75.37% <0.00%> (-1.00%) ⬇️
src/main/host/syscall_handler.c 51.89% <0.00%> (+0.42%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 22d4722...28c56e7. Read the comment docs.

@stevenengler stevenengler force-pushed the python branch 3 times, most recently from 911900b to 2c2c405 Compare April 5, 2021 13:46
@stevenengler stevenengler requested a review from robgjansen April 5, 2021 13:47
Copy link
Copy Markdown
Member

@robgjansen robgjansen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for organizing the commits - that made it easy to review :)

Requesting changes, primarily to address the NULL pointer issue.

descriptor_adjustStatus(&file->super, STATUS_DESCRIPTOR_ACTIVE, TRUE);

return _file_getFD(file);
return 0;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment here, or document in the header file what file_open and file_openat are expected to return. (It's useful since the return logic is slightly different from open and openat.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the return value back, but made it an error to call file_openat() if you haven't registered the descriptor yet.

Comment on lines +302 to +310
LegacyDescriptor* desc = process_getRegisteredLegacyDescriptor(sys->process, fd);

LegacyDescriptorType dType = descriptor_getType(desc);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the plugin passes in a bad FD? I think desc will be NULL then, and descriptor_getType will fail the magic assertion.

Instead, do we want to return -EBADF if desc is NULL?

For example, _syscallhandler_readHelper does this:

    /* Get the descriptor. */
    LegacyDescriptor* desc = process_getRegisteredLegacyDescriptor(sys->process, fd);
    if (!desc) {
        return (SysCallReturn){.state = SYSCALL_DONE, .retval.as_i64 = -EBADF};
    }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I forgot to check this.

warning("Cannot dup legacy descriptors");
return (SysCallReturn){.state = SYSCALL_DONE, .retval.as_i64 = -EOPNOTSUPP};
const SysCallArgs* args) {
gint fd = args->args[0].as_i64;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I generally like to add a debug message right after setting the fd in this line, to log the fd that we got on the syscall, before doing other stuff that may cause problems.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

Comment on lines +266 to +273
assert_nonneg_errno(fd = open(adf.name, O_RDONLY));
assert_nonneg_errno(fd2 = dup(fd));
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to expand this test, to make sure that something we wrote to fd before the dup is actually readable from fd2 after the dup?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added more to this test.

@stevenengler stevenengler requested a review from robgjansen April 5, 2021 17:06
Copy link
Copy Markdown
Member

@robgjansen robgjansen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks!

@stevenengler stevenengler merged commit a0ef5ac into shadow:dev Apr 5, 2021
@stevenengler stevenengler deleted the python branch April 5, 2021 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: Main Composing the core Shadow executable

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants