This is part of a series of periodic updates of development in Shadow. This work is sponsored by the NSF. Previous updates:

• 2020-02 https://lists.torproject.org/pipermail/tor-project/2020-February/002718.html
• 2020-03 https://lists.torproject.org/pipermail/tor-project/2020-March/002785.html
• 2020-06 https://lists.torproject.org/pipermail/tor-project/2020-June/002859.html
• 2020-07 https://lists.torproject.org/pipermail/tor-project/2020-July/002947.html

We've continued to make progress on the new process-based architecture in our dev branch. In particular:

• Functionality:

  • Partial support for threaded plugins. Support multi-threaded plugins #826
  • Implemented more of the syscalls known to be needed to support tor. Implement syscalls known to be used by tor and tgen #849

• Performance:

  • Now supports multiple worker threads, allowing Shadow to take advantage of multi-core hosts. Support multi-threaded Shadow #909
  • Completed the MemoryManager (in Rust!), which makes most IPC (notably including syscalls) between Shadow and its plugin processes zero-copy. Make thread_getWriteablePointer and thread_getReadablePointer zero-copy #888
  • We now use spin-locks for faster control-transfers between Shadow and its plugins. With this change, preload-based interposition is now significantly faster than ptrace-based interposition, and is getting close to uniprocess Shadow's performance. Use shared memory and spinlocks for Shadow-to-plugin IPC #894
  • Added preload-ptrace (aka Hybrid) mode: This mode extends ptrace-based interposition mode to use the faster preload-based mechanism for supported libc interfaces. This brings ptrace mode much closer to preload mode's performance when the plugin "hot paths" go through supported libc interfaces, while ensuring that ptrace-based interposition catches any syscalls that might otherwise be missed. Add a fallback mechanism for interposing syscalls in ThreadPreload #991

Additionally, we've begun prototyping a new high-level abstraction for descriptors (file, socket, pipe, etc) in Rust. This will pave the way for more comprehensive and accurate descriptor support.

• Our new descriptor design allows us to support new features such as file descriptor cloning, and improve the accuracy of existing features such as epoll monitoring.
• New descriptor and syscall features can now be written in Rust, furthering our goal of improving the memory safety, stability, and maintainability of Shadow.

Finally, we've made progress on tools that support the running of simulations with private tor networks.

• Our tornettools tool can help guide users through the process of creating tor network configurations, running simulations, and processing/visualizing the results.
• We started experimenting with running nightly tor experiments to help us benchmark performance and better track regressions.

Thanks, and Happy Holidays!

The Shadow team