Livelock with curl on Debian 10 #1794
Description
I am trying to get shadow to work (ultimately, for Tor research, but staying with a toy example for now), but experience the following issue: Whenever the shadow simulation is started, it appears to go to some livelock, using full CPU and not getting the actual simulation done. I have tried to investigate the problem, but digging into the shadow internals is somewhat cumbersome due to my lack of knowledge of the code base.
Describe the issue
Shadow goes to livelock, takes full CPU resources and simulation does not continue. Both with the toy example from the user guide (curl) as well as with the Tor example (edit: The Tor example failed due to issue #1788 instead).
To Reproduce
command line:
shadow shadow.yaml > shadow.log
shadow.yaml:
general:
# stop after 10 simulated seconds
stop_time: 10s
log_level: debug
network:
graph:
# use a built-in network graph containing
# a single vertex with a bandwidth of 1 Gbit
type: 1_gbit_switch
hosts:
# a host with the hostname 'server'
server:
network_node_id: 0
processes:
- path: /usr/bin/python3
args: -m http.server 80
start_time: 3s
# one host
client:
network_node_id: 0
quantity: 1
processes:
- path: /usr/bin/curl
args: -s server
start_time: 5s
Operating System (please complete the following information):
lsb_release -d:
Description: Debian GNU/Linux 10 (buster)
uname -a:
Linux cdoepmann-debian 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux
Shadow (please complete the following information):
shadow --version:
Shadow v2.0.0-88-g0a686c9b 2021-11-29--19:57:10
(same for original v2.0.0)
Additional context
backtrace when interupting the livelock (shadow process):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00007ffff7cc813f in g_cond_wait () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00005555567bae82 in countdownlatch_await (latch=0x555556cc3620)
at /home/christoph/tools/shadow/src/main/utility/count_down_latch.c:39
#3 0x000055555679a5d3 in workerpool_awaitTaskFn (pool=0x555556cae480)
at /home/christoph/tools/shadow/src/main/core/worker.c:296
#4 0x00005555567bef3b in scheduler_awaitNextRound (scheduler=0x555556cb3c00)
at /home/christoph/tools/shadow/src/main/core/scheduler/scheduler.c:434
#5 0x0000555556799610 in manager_run (manager=0x555556cb3680)
at /home/christoph/tools/shadow/src/main/core/manager.c:778
#6 0x0000555556796bbc in controller_run (controller=0x555556cc3d10)
at /home/christoph/tools/shadow/src/main/core/controller.c:517
#7 0x0000555555f24f9f in shadow_rs::core::main::run_shadow (args=...) at core/main.rs:153
#8 0x0000555555c13842 in shadow_rs::core::main::export::main_runShadow (argc=2, argv=0x7fffffffdd08)
at core/main.rs:235
#9 0x0000555555c1374b in main (argc=2, argv=0x7fffffffdd08) at /home/christoph/tools/shadow/src/main/main.c:9
backtrace when interupting the livelock (curl process):
0x00007ffff76e6b47 in shim_native_syscallv (n=202, args=0x7ffff76fd0e0 <_tlss+39488>) at /home/christoph/tools/shadow/src/lib/shim/shim_syscall.c:74
#1 0x00007ffff76e73a8 in shim_syscallv (n=202, args=0x7ffff76fd0e0 <_tlss+39488>) at /home/christoph/tools/shadow/src/lib/shim/shim_syscall.c:262
#2 0x00007ffff76e4f91 in shim_api_syscallv (n=202, args=0x7ffff76fd0e0 <_tlss+39488>) at /home/christoph/tools/shadow/src/lib/shim/shim_api_syscall.c:24
#3 0x00007ffff7fc2391 in syscall (n=202) at /home/christoph/tools/shadow/src/lib/libc_preload/libc_impls.c:36
#4 0x00007ffff76e7b00 in _futex_wait_abs (s=0x7fffee112668, timeout=0x0) at /home/christoph/tools/shadow/src/lib/shim/shadow_sem.c:30
#5 0x00007ffff76e7d9f in _shadow_sem_timedwait (sem=0x7fffee112668, abs_timeout=0x0) at /home/christoph/tools/shadow/src/lib/shim/shadow_sem.c:130
#6 0x00007ffff76e7e65 in shadow_sem_wait (sem=0x7fffee112668) at /home/christoph/tools/shadow/src/lib/shim/shadow_sem.c
#7 0x00007ffff76e80f0 in BinarySpinningSem::wait (this=0x7fffee112668, spin=true) at /home/christoph/tools/shadow/src/l
#8 0x00007ffff76e77c8 in shimevent_recvEventFromShadow (data=0x7fffee112408, e=0x7ffff76fd2e0 <_tlss+40000>, spin=true)
#9 0x00007ffff76e6d08 in _shim_emulated_syscall_event (syscall_event=0x7fffffffd890) at /home/christoph/tools/shadow/sr
#10 0x00007ffff76e716b in shim_emulated_syscallv (n=13, args=0x7fffffffda50) at /home/christoph/tools/shadow/src/lib/shi
#11 0x00007ffff76e734d in shim_syscallv (n=13, args=0x7fffffffda50) at /home/christoph/tools/shadow/src/lib/shim/shim_sy
#12 0x00007ffff76e7451 in shim_syscall (n=13) at /home/christoph/tools/shadow/src/lib/shim/shim_syscall.c:271
#13 0x00007ffff76e5ce1 in _shim_seccomp_handle_sigsys (sig=31, info=0x7fffffffdcf0, voidUcontext=0x7fffffffdbc0) at /hom
#14 <signal handler called>
#15 0x00007ffff7cca800 in __libc_sigaction (sig=13, act=<optimized out>, oact=0x0) at ../sysdeps/unix/sysv/linux/sigacti
#16 0x00007ffff7f2c429 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#17 0x00007ffff7f2fc13 in curl_multi_perform () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#18 0x00007ffff7f26742 in curl_easy_perform () from /usr/lib/x86_64-linux-gnu/libcurl.so.4
#19 0x0000555555569eb0 in ?? ()
#20 0x000055555556b3aa in ?? ()
#21 0x000055555555d8b4 in ?? ()
#22 0x00007ffff7b1909b in __libc_start_main (main=0x55555555d760, argc=3, argv=0x7fffffffec08, init=<optimized out>, fin
#23 0x000055555555d9ca in ?? ()
There were no build errors and the tests also pass.
I would greatly appreciate any help towards getting shadow to run.