Documented PID numbering, and closed #1491 since that enables graceful shutdown.

Supporting killall workflow tracked in #1986

Supporting fork+exec tracked in #1987

We discussed this some more and haven't yet decided on a path forward. There are a few independent questions:

1. Do users expect Shadow to return an error status if a managed process exits with a non-zero error code? What about when Shadow intentionally kills a managed process due to stop_time?

2. What should Shadow's default behaviour be? Should Shadow even care about the exit codes of managed processes? If a managed process exits with a non-zero exit code, but Shadow itself didn't have any errors, then should Shadow return a zero or non-zero exit code? In this case Shadow executed everything as specified by the user so the simulation itself didn't have any errors, but the simulation results may not be what the user expected. If we were to classify these as "logical errors" or "technical errors" in the simulation, which of these two should affect the simulation's exit code?

3. How can we be more clear to the user about the end result of a simulation, especially if the answer to (1) conflicts with the answer to (2)? Currently when Shadow finishes, it writes a "Shadow completed successfully" or "Shadow did not complete successfully" message depending on Shadow's exit code.

4. Sometimes the "user" cannot read and understand Shadow's log messages to look for logical errors in their simulation. For example if we want to make sure that our tests are working correctly in the CI, we need a way to automate checking the output from (3). If the answer to (2) is to ignore exit codes, we still need a way to check these exit codes in our tests.

5. Should the configuration format support options to change how Shadow handles different managed process exit states? Or can we wait until Shadow supports fork/exec, and require users to wrap managed processes in bash scripts which check exit states?

From discussion:

1. We don't know exactly what users expect from Shadow, but they probably expect some consistency about when Shadow returns a non-zero exit code. Shadow's current rules about when to return a non-zero exit code are confusing and not documented anywhere.

2. We have conflicting opinions on this.

3. Instead of printing a success/failure message, we can print a summary table of processes and their exit states. There are different ways we could summarize these states, but one would be to group them based on (plugin_path, exit_state). We should also document Shadow's behaviour somewhere.

4. We could log exit codes to a json file, and have a script iterate over that json to look for managed processes that didn't return a zero exit code. But this probably isn't ideal.

5. We could have a per-process configuration option that changes how Shadow handles different managed process exit states. The default for this option would depend on the answer to (2). A backwards-compatible option could be a "runs_forever" option that has the following rules:

   • if false: It is an error if the plugin exits with a non-zero error code, either during the simulation, or at the end of the simulation after the sigkill. If it's running at the end of the simulation, it will always be an error.
   • if true: It is an error if the plugin exits before the end of the simulation. The exit code is always ignored (both during and at the end of the simulation).

   This might still not be clear to users though. Another option could be to have a "exit_state" option that allows users to specify exactly what exit state the process should have (an exit code, an exit signal, or if it should be killed by Shadow's "stop time").