-
|
I'm trying to simulate the network of 5,000 nodes of gossipsub protocol Since running a lot processes requires a lot of memory, I created some swap file of 230G as described here However, it seems not easy to just use the swap file as the memory. As you can see in the figure, I have 230G of swap space, but it's not used at all. As a result, I can run only 400 nodes (with 2GB of memory). Does anyone have experience running thousands of nodes? I think even though each process uses very little memory, in the end the total sum of memory will be very high if you simulate a lot of nodes. What I'm guessing now is that most memory used by each process is shared memory and the shared memory probably cannot be swapped out? The following is the htop result. Let me know if you need any clarification or some source code.
|
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
Thanks for your question!
Yes! Shadow is routinely used for simulations with many thousands of nodes. However, it's usually done on machines with many more resources than the one you've posted.
I do believe that Linux kernel memory can't be paged out [0], and simulating many processes can consume a decent amount of memory in the kernel per process (a
To the broader question, I am worried about the strategy of running a large experiment and trying to fit all the state into swap space. Even if you worked out the paging, the experiment would likely progress too slowly to be useful. There are a couple of ways you might consider adjusting your process:
[0] https://tldp.org/LDP/khg/HyperNews/get/memory/linuxmm.html |
Beta Was this translation helpful? Give feedback.
-
|
+1 to all of that, but one small thing you could try is disabling the memory manager, which otherwise maps all of the managed process heaps and stacks into shared memory for faster IPC, and yeah I'm not sure whether that memory is eligible to be swapped.
|
Beta Was this translation helpful? Give feedback.
-
|
I tried using
|
Beta Was this translation helpful? Give feedback.
-
|
Glad to hear it! I think One thing I forgot is that you'd probably need to remount Since you already have something working it's probably not worth messing with unless you're curious or it's worth your time to slightly improve simulation performance. It used to be a bigger win, but we've improved the non memory-mapped code path to the point that it's only about a ~1% performance gain #2581 |
Beta Was this translation helpful? Give feedback.
Thanks for your question!
Yes! Shadow is routinely used for simulations with many thousands of nodes. However, it's usually done on machines with many more resources than the one you've posted.
I do believe that Linux kernel memory can't be paged out [0], and simulating many processes can consume a decent amount of memory in the kernel per process (a
task_structkernel object alone might consume 1-2 kilobytes per process). Also, you might need to adjust your machine's virtual memory parameters before the right swapping behavior is achieved [1].