I think the biggest downside of adding something like this is that it'd be another place to keep in sync with the config format.

really half-baked idea: I wonder if we could make this python module more-or-less replace the specification doc. If we added the text from there as comments it'd be pretty close. Potentially we could have a script that generates the markdown/html from the python module.

I write many shadow config generators in python. Usually I'm reading in a config created by tornettools and then extending/modifying it, so I've never found it that cumbersome to check the docs for config names and then use the untyped python dicts that the yaml reader return. But maybe this module would improve QoL and help cut down on errors?

I wonder:

• Does it make sense to create the module in its own repo? Maybe not (yet) because it is too simple right now and would be more likely to become stale? (Though I could imagine a shadowtools python module where we could add future python extensions, like parsing/plotting new data formats, monitoring simulation progress, etc.)
• Would we want to push the module to pypi.org, to enable pip install without cloning the shadow repo first? If we did this, does that contribute to an argument for a separate shadowtools repo?

I would argue against putting the Shadow config documentation inside this somewhat extraneous python tool. That seems kind of backwards to me. I would prefer instead to put the config docs as close to the rust config code as possible (i.e., configuration.rs) to make it as easy as possible to keep in sync as things change. Then I could imagine that shadow (or maybe a separate rust executable) implements a subcommand that auto-generates this python module from the latest state of the rust code. Similarly, we could dump the rust docs into the mdbook format for the web. I'm not exactly sure how feasible this would be though, e.g., is there a rust element for everything in the configuration specification?

Does it make sense to create the module in its own repo?

Yeah, I'm not sure. Keeping it in this repo is generally simpler and less work overall, and we can ensure it's always in sync with shadow itself. From a user perspective the only downside I can think of is in a CI job where you want to run a pre-built shadow binary using this python module, now you need to clone the relatively large shadow repo just to get the module. Overall that's probably better though than having to track and version the two repos independently.

Would we want to push the module to pypi.org, to enable pip install without cloning the shadow repo first? If we did this, does that contribute to an argument for a separate shadowtools repo?

It seems a little weird to push shadow-related-packages to pypy.org when we don't package shadow itself anywhere. We can define a package though, and use pip to install it. e.g. in arti's CI we install chutney with python3 -m pip install git+https://gitlab.torproject.org/tpo/core/chutney.git@"$CHUTNEY_COMMIT". I'm not sure whether the pyproject.toml needs to be in the root of the repo for that to work, or if there's someway to specify a subdirectory of the repo.

I would argue against putting the Shadow config documentation inside this somewhat extraneous python tool.

Fair enough.

I would prefer instead to put the config docs as close to the rust config code as possible (i.e., configuration.rs) to make it as easy as possible to keep in sync as things change. Then I could imagine that shadow (or maybe a separate rust executable) implements a subcommand that auto-generates this python module from the latest state of the rust code.

Interesting. I think we do something a little bit like this in arti to scrape rustdoc comments to generate the RPC documentation.

By the way I don't know if you saw that later commits, but I did end up fleshing out the runner tool a bit more. It's kind of convenient, and maybe a useful gateway to show people how to use shadow before forcing them to write a config file. Right now it's in the same python module, but probably belongs in its own module. (Though could maybe be part of the same python package)

Example:

$ time ~/projects/shadow/src/tools/shadow.py PATH="$PATH" CHUTNEY_ENABLE_CONTROLSOCKET=0 CHUTNEY_DISABLE_IPV6=1 CHUTNEY_PATH="$(pwd)" tools/test-network.sh --flavor basic-min --tor tor --tor-gencert tor-gencert
tail: cannot open '/tmp/run-shadow4y13361b/shadow.data/hosts/host/bash.1000.stdout' for reading: No such file or directory
** Starting Shadow 3.2.0
tail: '/tmp/run-shadow4y13361b/shadow.data/hosts/host/bash.1000.stdout' has appeared;  following new file
test-network.sh: using CHUTNEY_DNS_CONF '/dev/null'
test-network.sh: $TOR_DIR has no src/app or src/or, looking elsewhere
test-network.sh: Assuming $CHUTNEY_TOR is a binary name in $PATH
test-network.sh: Assuming $CHUTNEY_TOR_GENCERT is a binary name in $PATH
test-network.sh: Using $CHUTNEY_TOR: 'tor' and $CHUTNEY_TOR_GENCERT: 'tor-gencert'
==== Running tests: bootstrap attempt 1/1

Launching chutney using Python 3.10.12
Sending SIGINT to nodes
Waiting for nodes to finish.

Launching chutney using Python 3.10.12
bootstrap-network.sh: bootstrapping network: basic-min

Launching chutney using Python 3.10.12
NOTE: creating '/home/jnewsome/projects/chutney/nodes.946684801.15', linking to '/home/jnewsome/projects/chutney/nodes'
Creating identity key for test000a     with tor-gencert
Creating identity key for test001a     with tor-gencert
Creating identity key for test002a     with tor-gencert
Creating identity key for test003a     with tor-gencert
======= phase 1

Launching chutney using Python 3.10.12
Starting nodes

Launching chutney using Python 3.10.12
test000a     is running with PID  1057: Tor 0.4.9.0-alpha-dev (git-a56350abc8a8c18b)
test001a     is running with PID  1062: Tor 0.4.9.0-alpha-dev (git-a56350abc8a8c18b)
test002a     is running with PID  1067: Tor 0.4.9.0-alpha-dev (git-a56350abc8a8c18b)
test003a     is running with PID  1072: Tor 0.4.9.0-alpha-dev (git-a56350abc8a8c18b)
test004r     is running with PID  1077: Tor 0.4.9.0-alpha-dev (git-a56350abc8a8c18b)
test005c     is running with PID  1082: Tor 0.4.9.0-alpha-dev (git-a56350abc8a8c18b)
6/6 nodes are running
Waiting up to 300 seconds for all nodes in phase 1 to bootstrap...

Launching chutney using Python 3.10.12
** Shadow completed successfully
Waiting for nodes to bootstrap...

Bootstrap in progress: 10 seconds
Node status:
test000a     :    0, starting                 , Starting
test001a     :    0, starting                 , Starting
test002a     :    0, starting                 , Starting
test003a     :    0, starting                 , Starting
test004r     :    0, starting                 , Starting
test005c     :   30, loading_status           , Loading networkstatus consensus
Published dir info:
test000a     : -400, caches and clients       , all formats                   , No dir file
test001a     : -400, caches and clients       , all formats                   , No dir file
test002a     : -400, caches and clients       , all formats                   , No dir file
test003a     : -400, caches and clients       , all formats                   , No dir file
test004r     : -400, caches and clients       , all formats                   , No dir file

Bootstrap in progress: 20 seconds
Node status:
test000a     :    0, starting                 , Starting
test001a     :    0, starting                 , Starting
test002a     :    0, starting                 , Starting
test003a     :  100, done                     , Done
test004r     :    0, starting                 , Starting
test005c     :   75, enough_dirinfo           , Loaded enough directory info to build circuits
Published dir info:
test000a     : -400, 004r                     , all formats                   , No dir file
test001a     : -400, 004r                     , all formats                   , No dir file
test002a     : -400, 004r                     , all formats                   , No dir file
test003a     : -400, 004r                     , all formats                   , No dir file
test004r     : -400, 004r                     , all formats                   , No dir file

Bootstrap in progress: 30 seconds
Node status:
test000a     :  100, done                     , Done
test001a     :  100, done                     , Done
test002a     :  100, done                     , Done
test003a     :  100, done                     , Done
test004r     :  100, done                     , Done
test005c     :  100, done                     , Done
Published dir info:
test000a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test001a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test002a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test003a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test004r     :    0, all nodes                , desc md md_cons ns_cons       , Not in dir file

Bootstrap in progress: 40 seconds
Node status:
test000a     :  100, done                     , Done
test001a     :  100, done                     , Done
test002a     :  100, done                     , Done
test003a     :  100, done                     , Done
test004r     :  100, done                     , Done
test005c     :  100, done                     , Done
Published dir info:
test000a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test001a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test002a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test003a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test004r     :    0, all nodes                , desc md md_cons ns_cons       , Not in dir file

Bootstrap in progress: 50 seconds
Node status:
test000a     :  100, done                     , Done
test001a     :  100, done                     , Done
test002a     :  100, done                     , Done
test003a     :  100, done                     , Done
test004r     :  100, done                     , Done
test005c     :  100, done                     , Done
Published dir info:
test000a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test001a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test002a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test003a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test004r     :    0, all nodes                , desc md md_cons ns_cons       , Not in dir file

Bootstrap in progress: 60 seconds
Node status:
test000a     :  100, done                     , Done
test001a     :  100, done                     , Done
test002a     :  100, done                     , Done
test003a     :  100, done                     , Done
test004r     :  100, done                     , Done
test005c     :  100, done                     , Done
Published dir info:
test000a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test001a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test002a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test003a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test004r     :    0, 005c                     , md md_cons                    , Not in dir file

Bootstrap in progress: 70 seconds
Node status:
test000a     :  100, done                     , Done
test001a     :  100, done                     , Done
test002a     :  100, done                     , Done
test003a     :  100, done                     , Done
test004r     :  100, done                     , Done
test005c     :  100, done                     , Done
Published dir info:
test000a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test001a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test002a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test003a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test004r     :    0, 005c                     , md md_cons                    , Not in dir file

Everything bootstrapped after 72 sec
Bootstrap finished: 72 seconds
Node status:
test000a     :  100, done                     , Done
test001a     :  100, done                     , Done
test002a     :  100, done                     , Done
test003a     :  100, done                     , Done
test004r     :  100, done                     , Done
test005c     :  100, done                     , Done
Published dir info:
test000a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test001a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test002a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test003a     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached
test004r     :  100, all nodes                , desc md md_cons ns_cons       , Dir info cached

Waiting 0 seconds for the network to be ready...

Running 1 verify rounds for this bootstrap...

Launching chutney using Python 3.10.12
NOTE: Registering send-data-1
NOTE: Registering check-2
NOTE: Success for send-data-1
NOTE: Success for check-2
Verifying data transmission: (retrying for up to 60 seconds)
Connecting:
  Exit to 127.0.0.1:4747 via client localhost:9005
Transmitting Data:

NOTE: Status:
{'send-data-1': 'Flushed', 'check-2': 'successful verification'}
Transmission: Success
Completed verify round 1/1 in this bootstrap

Launching chutney using Python 3.10.12
Sending SIGINT to nodes
Waiting for nodes to finish.
Summary: nodes
Detail: chutney/tools/warnings.sh /home/jnewsome/projects/chutney/nodes.946684801.15
Warning: Could not find a node that matches the configured _HSLayer2Nodes set Number: 3
==== Chutney succeeded after 1 attempt(s).

real	0m9.055s
user	0m7.013s
sys	0m1.402s

I'm not sure whether the pyproject.toml needs to be in the root of the repo for that to work, or if there's someway to specify a subdirectory of the repo.

It looks like it can be in a subdirectory. From https://pip.pypa.io/en/stable/cli/pip_install/#examples :

python -m pip install -e 'git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path' # install a python package from a repo subdirectory

Ok, I think this is about ready for proper review.

If you're not crazy about the shadow-shell tool living here, I'm happy to put in its own repo, e.g. on tor's gitlab. I imagine I'm going to be hacking on it a bit more.

I'm also open to better names than "shadowsim" and "shadow-shell".

Here are my first impressions before reviewing. Just typing them out now before I get some food :)

Looks cool!

• config: This seems useful for people writing scripts to generate new configs from scratch. It would be nice to have a good way to keep this in sync with shadow, but I can't think of a good way.
• shell: I think this is a nice way for people to experiment with shadow. I'm not super clear on what the use cases are outside of that though. Running chutney on a single host in shadow seems like a niche use case. That doesn't mean I don't think we should add it, just that I'm worried if it doesn't have a clear use case it might not be used and then slowly forgotten about. Maybe this could be useful in our shadow tests if we want to run existing programs like ip addr to check the output, and don't want to write an entire yaml file to test the one command. (Though maybe it would be nicer to use rust unit tests for these instead.)

I think having it in the shadow repo makes sense to keep things in sync. Having it in the shadow repo also means that you don't need to worry about if the version of the tool from its repo will be compatible with a specific version of shadow from the shadow repo.

I think it should have tests in the CI to make sure it continues to work and doesn't become stale over time.

I have mixed feelings about generating the docs from code in general. I think generating documentation from code doesn't usually turn out that great. There ends up being cases that aren't generated properly or could be done better by a person, and then we need some way to override the documentation generation script. I had tried doing this when originally moving the config code to rust, but realized after a while that it would take more time and effort to maintain the script than to just update the documentation manually when needed. It can be painful when you know what you want the documentation to say, but you need to spend an hour modifying the script to do what you could do by hand in 10 seconds. On the other hand it's not difficult to accidentally forget to update the docs, and then the docs become stale. But I think we've been doing a pretty good job at keeping the docs up to date.

Another option is something like schemars which lets you derive a json schema from the rust code which you could use to build documentation, but it needs a bunch of manual editing to get human readable types, default values, etc. Anyways my thoughts are that if we can generate the docs like we have it now, and without needing a bunch of overrides for cases that the docs generation script doesn't handle well, then it might be nice to have it regardless of if it's being generated from the rust serde types or the new python config types.

shadowsim --> I like shadowpy or shadowtools better

Ah, so I was originally thinking just "shadow", but there are a lot packages referencing various shadows in pypi: https://pypi.org/search/?q=shadow. Even if we never publish to pypi I figured it'd be best to avoid name collisions there. So the "sim" suffix was to distinguish it from all the other shadows being referenced there.

IMO having 'py' in the name of a python package feels redundant, but it's certainly not unprecedented 🤷

Maybe also adding the "tools" suffix makes sense though since the python module doesn't completely wrap "the shadow simulator", but is indeed more of a toolset for working with it.

So maybe "shadowsimtools"?

I was thinking why not just make tools/. the base directory rather than tools/shadowsim. But that maybe only makes sense if the python package name is shadowtools (in which case the base directory for shadowtools would be shadow/tools :)). But also maybe that's a stretch.

I figured I'd have the intermediate tools directory in case there's anything else we end up wanting to add there later, but maybe without immediate plans to do so it's over-engineering.

So maybe make the base directory shadowsimtools, if we go with that package name?

config: This seems useful for people writing scripts to generate new configs from scratch. It would be nice to have a good way to keep this in sync with shadow, but I can't think of a good way.

Agreed :/

shell: I think this is a nice way for people to experiment with shadow. I'm not super clear on what the use cases are outside of that though. Running chutney on a single host in shadow seems like a niche use case. That doesn't mean I don't think we should add it, just that I'm worried if it doesn't have a clear use case it might not be used and then slowly forgotten about. Maybe this could be useful in our shadow tests if we want to run existing programs like ip addr to check the output, and don't want to write an entire yaml file to test the one command. (Though maybe it would be nicer to use rust unit tests for these instead.)

Yeah, it might end up being too niche to be generally useful. I definitely intend to use it in tor's CI though, so it'll have at least one user :). Maybe more use-cases will crop up; e.g. in some ways it's a more powerful alternative to faketime.

I think we could use this to run a lot of our "unit" tests and delete the checked-in configs. In addition to getting rid of the configs, the simulated stdout would be a bit more visible since you wouldn't have to hunt down the data dir. I'm not sure it makes sense to go that "all in" on it yet, though. e.g.

$ shadow-exec ./build/src/target/debug/test_sleep 
*** Basic sleep tests ***
    sleep,    call_clock_gettime -- Duration: 1s (sleep_duration 1s, tolerance +/- 30ms)
    sleep, syscall_clock_gettime -- Duration: 1s (sleep_duration 1s, tolerance +/- 30ms)
   usleep,    call_clock_gettime -- Duration: 1s (sleep_duration 1s, tolerance +/- 30ms)
   usleep, syscall_clock_gettime -- Duration: 1s (sleep_duration 1s, tolerance +/- 30ms)
*** Interrupted sleep tests. Sleeping for 3s but interrupted after 1s ***
    sleep -- Slept for: 1s, rem: Some(2s)
   usleep -- Slept for: 1s, rem: Some(2s)
Success.

I think it should have tests in the CI to make sure it continues to work and doesn't become stale over time.

Yeah, I'll add some tests. (Though I suppose using it to run our other tests as above would kill 2 birds with one stone 🤔 )

I have mixed feelings about generating the docs from code in general.

Yeah, same. It's probably not worth it. I feel a little bad about adding yet another place to keep synchronized with the config spec, but OTOH it shouldn't really generate that much more work, and isn't that big a deal if it falls out of sync.

Added mypy static type checking to the CI. Still thinking about how best to add additional tests for shadow-exec.

On a whim, this gets surprisingly far:

$ shadow-exec -- bash -c "PATH='$PATH' ./setup test -j1 'linux'"
2000-01-01 00:00:00,001 [INFO] calling ['ctest', '-j1', '--timeout', '20', '-R', 'linux']
Test project /home/jnewsome/projects/shadow/build
      Start   3: bindc-linux
 1/66 Test   #3: bindc-linux .............................   Passed    0.00 sec
      Start   5: capget-linux
 2/66 Test   #5: capget-linux ............................   Passed    0.00 sec
      Start   7: capset-linux
 3/66 Test   #7: capset-linux ............................   Passed    0.00 sec
      Start  13: clone_leader_exits_early-linux
 4/66 Test  #13: clone_leader_exits_early-linux ..........   Passed    0.00 sec
      Start  17: fork-linux
 5/66 Test  #17: fork-linux ..............................   Passed    4.35 sec
      Start  19: close_range-linux
 6/66 Test  #19: close_range-linux .......................   Passed    0.00 sec
      Start  38: cpp-linux
 7/66 Test  #38: cpp-linux ...............................   Passed    1.00 sec
      Start  47: dup-linux
 8/66 Test  #47: dup-linux ...............................   Passed    0.00 sec
      Start  50: epoll-linux
 9/66 Test  #50: epoll-linux .............................   Passed    1.90 sec
      Start  53: epoll-rs-linux
10/66 Test  #53: epoll-rs-linux ..........................   Passed    0.45 sec
      Start  55: epoll-edge-rs-linux
11/66 Test  #55: epoll-edge-rs-linux .....................   Passed    9.08 sec
      Start  58: eventfd-linux
12/66 Test  #58: eventfd-linux ...........................   Passed    0.00 sec
      Start  61: exit-linux
13/66 Test  #61: exit-linux ..............................   Passed    0.00 sec
      Start  65: file-linux
14/66 Test  #65: file-linux ..............................   Passed    0.00 sec
      Start  67: futex-linux
15/66 Test  #67: futex-linux .............................   Passed    3.10 sec
      Start  69: ifaddrs-linux
16/66 Test  #69: ifaddrs-linux ...........................   Passed    0.00 sec
      Start  71: mmap-linux
17/66 Test  #71: mmap-linux ..............................   Passed    0.00 sec
      Start  73: unaligned-linux
18/66 Test  #73: unaligned-linux .........................   Passed    1.00 sec
      Start  75: netlink-send-limit-linux
19/66 Test  #75: netlink-send-limit-linux ................   Passed    0.00 sec
      Start  77: netlink-bind-linux
20/66 Test  #77: netlink-bind-linux ......................   Passed    0.00 sec
      Start  82: pipe-linux
21/66 Test  #82: pipe-linux ..............................   Passed    4.00 sec
      Start  84: poll-linux
22/66 Test  #84: poll-linux ..............................   Passed    3.18 sec
      Start  86: prctl-linux
23/66 Test  #86: prctl-linux .............................   Passed    0.00 sec
      Start  88: random-linux
24/66 Test  #88: random-linux ............................   Passed    0.00 sec
      Start  94: flush_after_exit-linux
25/66 Test  #94: flush_after_exit-linux ..................   Passed    0.00 sec
      Start  96: busy_wait-linux
26/66 Test  #96: busy_wait-linux .........................   Passed    0.00 sec
      Start 100: getaddrinfo-linux
27/66 Test #100: getaddrinfo-linux .......................   Passed    0.00 sec
      Start 102: sched_affinity-linux
28/66 Test #102: sched_affinity-linux ....................   Passed    0.00 sec
      Start 104: select-linux
29/66 Test #104: select-linux ............................   Passed    0.28 sec
      Start 106: signal-linux
30/66 Test #106: signal-linux ............................   Passed    0.30 sec
      Start 108: signals-extra-linux
31/66 Test #108: signals-extra-linux .....................   Passed    0.30 sec
      Start 111: sleep-linux
32/66 Test #111: sleep-linux .............................   Passed    6.00 sec
      Start 113: sockbuf-linux
33/66 Test #113: sockbuf-linux ...........................   Passed    0.02 sec
      Start 115: socket-linux
34/66 Test #115: socket-linux ............................***Failed    0.00 sec
      Start 118: bind-linux
35/66 Test #118: bind-linux ..............................   Passed    0.00 sec
      Start 122: listen-linux
36/66 Test #122: listen-linux ............................   Passed    9.90 sec
      Start 124: getsockname-linux
37/66 Test #124: getsockname-linux .......................   Passed    0.00 sec
      Start 127: accept-linux
38/66 Test #127: accept-linux ............................   Passed    4.81 sec
      Start 130: connect-linux

As far as I can tell shadow is deadlocked at that point, in the connect-linux test. It looks like that one actually does set up multiple hosts normally, so it's unsurprising that it doesn't work, though maybe not great that it just deadlocks. (Running the tests this way also passes --libc-passing to the test bins instead of the --shadow-passing that we'd want)

I am thinking a bit more about adding a add_shadow_exec_test cmake macro and converting some of the simpler add_shadow_test tests to use that instead. There could even be a add_linux_and_shadow_exec macro that creates both a linux and shadow test. It would simplify those tests and get rid of some boilerplate, at the expense of having more macros and another level of indirection to think through when things go wrong.

Probably for now I'll just add a few tests to the python package dir itself rather than trying to migrate our other tests to use it.

So maybe "shadowsimtools"?

It's the "sim" part that I was hoping to drop.

I guess I'm selfishly thinking Shadow existed long before most of those python packages so we have as much claim to the base shadow name as anyone else. I still like shadowtools. It probably doesn't matter too much since we don't really want to push to pypi, shadowtools is shorter and slightly less complex, and if we someday want to push to pypi and that name is taken (it is not taken as of now) we could consider renaming.

We can also have Steve cast a vote :)

So maybe "shadowsimtools"?

It's the "sim" part that I was hoping to drop.

I guess I'm selfishly thinking Shadow existed long before most of those python packages so we have as much claim to the base shadow name as anyone else. I still like shadowtools. It probably doesn't matter too much since we don't really want to push to pypi, shadowtools is shorter and slightly less complex, and if we someday want to push to pypi and that name is taken (it is not taken as of now) we could consider renaming.

We can also have Steve cast a vote :)

I don't have strong feelings here. I think I'd lean slightly towards "shadowtools" over "shadowsim" or "shadowsimtools". IMO we shouldn't care too much about what's on pypy since a name that's available today might not be available tomorrow (either coincidentally or maliciously).

I don't have strong feelings here. I think I'd lean slightly towards "shadowtools" over "shadowsim" or "shadowsimtools".

Sounds like "shadowtools" it is!

IMO we shouldn't care too much about what's on pypy since a name that's available today might not be available tomorrow (either coincidentally or maliciously).

Yeah, my main concern is that I don't think a user can install this package locally AND a package of the same name from pypi at the same time. But maybe it's not that big of a deal, particularly with virtual envs.

On a whim, this gets surprisingly far:

It's crazy to me that it gets that far, considering it's running random cmake stuff. I wonder if the rust-unit-tests test works, which needs to run the rust compiler...

I have mixed feelings about generating the docs from code in general.

Yeah, same. It's probably not worth it. I feel a little bad about adding yet another place to keep synchronized with the config spec, but OTOH it shouldn't really generate that much more work, and isn't that big a deal if it falls out of sync.

Yeah I don't think it's a big deal. It might be good for us to (at some point, doesn't need to be now) have a documentation page about how to add new config options (update rust code, update documentation, update python code, and update expected --help text in the cli test).

I see the PR to drop support for Debian 10, but it also looks like the PR is failing on ubuntu 20.04. If ubuntu 20.04 has too old of a python version (3.8), my vote would be to just disable testing of this python code on that platform rather than trying to support the old ubuntu version that we're dropping support for in ~5 months.

I see the PR to drop support for Debian 10, but it also looks like the PR is failing on ubuntu 20.04. If ubuntu 20.04 has too old of a python version (3.8), my vote would be to just disable testing of this python code on that platform rather than trying to support the old ubuntu version that we're dropping support for in ~5 months.

Yeah I think it only needs a small change, but it if it turns out to be too onerous I'm fine with just not supporting it.

Ok, I think that about does it! I'm sure I'll be iterating a bit more on the shadow-exec but I think this is a reasonable starting point.

@robgjansen could you approve #3451 and remove debian:10 from the required tests?

Thanks for the reviews!

Oh, @robgjansen I wasn't sure what milestone to use. Maybe a new one along the lines of "usability"?

Maybe we can avoid adding a milestone for now to prevent "milestone creep"? ... IMO, those typically imply some concerted effort toward a goal, or that we put more thought into where we are headed than perhaps we did here. That being said, if we start adding lots of work in this direction, we could think more about where we are wanting to get to in the shortish term, and then it could make sense to encode that into a milestone.

Maybe we can avoid adding a milestone for now to prevent "milestone creep"? ... IMO, those typically imply some concerted effort toward a goal, or that we put more thought into where we are headed than perhaps we did here. That being said, if we start adding lots of work in this direction, we could think more about where we are wanting to get to in the shortish term, and then it could make sense to encode that into a milestone.

That's fine with me if you're fine with me continuing to flagrantly ignore the "check milestone" test failures for now :)

flagrantly ignore the "check milestone" test failures

:( well that sounds like you have some more work in mind in this direction. I made a milestone for this now. But I really want to avoid having this be a generic "usability" milestone that will be open for the rest of eternity. So if you could think a bit about when we should consider this line of work mature enough to close the milestone, that would be nice. And please update the milestone description to document your current plans :)

https://github.com/shadow/shadow/milestone/56

Thanks; this LGTM! Yeah I expect minimally I'll be tinkering a bit more with the shadow-exec script.

Another way of looking at the goal here is to improve the "shadow as CI tool" use-case. Generally I think shadow-exec is useful to get time compression and determinism for CI-like use-cases, but maybe a bit too "yolo" for academic paper results. e.g. this is why I ended up defaulting --model-unblocked-syscall-latency to true for shadow-exec while continuing to think false is probably the right default for shadow itself: for the CI use-case it's better if it Just Works by default, but for paper results you want to be sure you understand the exact time model.

Another way of looking at the goal here is to improve the "shadow as CI tool" use-case.

Maybe also the "reproducible build" use-case? Though if we really wanted to support that I suppose we should fix file metadata to have the simulated dates.

I'm not sure how much you care about these use-cases in general, but OTOH supporting more use-cases would hopefully help draw interest and ultimately funding.

gcc seems to work, though it seems we have a bug preventing setting +x.

jnewsome@pop-os:~/tmp$ shadow-exec -- bash -c "export PATH='$PATH'; gcc hello-world.c; chmod +x ./a.out; ./a.out"
shadow: ** Starting Shadow 3.2.0
shadow: Error: process 'host.bash.1000' exited with status Normal(126); expected end state was exited: 0 but was exited: 126
shadow: Error: Failed to run the simulation
shadow: Error: 
shadow: Error: Caused by:
shadow: Error:     1 managed processes in unexpected final state
shadow: ** Shadow did not complete successfully: Failed to run the simulation
shadow: **   1 managed processes in unexpected final state
shadow: ** See the log for details
bash: line 1: ./a.out: Permission denied
jnewsome@pop-os:~/tmp$ chmod +x ./a.out 
jnewsome@pop-os:~/tmp$ ./a.out 
hello world

@sporksmith I added the project 141 label, but feel free to remove if it shouldn't be.

@sporksmith I added the project 141 label, but feel free to remove if it shouldn't be.

Oh thanks, I'd forgotten we'd created that label.

I added the alt use-case examples to the milestone