TGen error causes #2134
-
|
I have been running large-scale Tor simulations using tornettools, which simulates clients with TGen. After some analysis, I've found that most of the errors that occur in TGen clients are "READEOF," and this seems to happen very often (up to 30% error rate with the 5 MiB stream with no padding machines enabled). However, I haven't been able to determine the source of this error. I took a brief look through the TGen source code, and there doesn't seem to be any reason why sockets are being closed early. This might make more sense if the errors were happening during a setup phase (such as SOCKS connection), but they tend to occur after a fair number of bytes have already been sent. I was hoping that someone might have some insight into this, or at least be able to point me to some resource that might be useful in figuring out the reason for these errors. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 9 replies
-
I hope you're finding our tools useful :)
Just so we're on the same page, tornettools will create two main client types: perf clients and Markov clients. I think when you stated you had "no padding machines enabled" I think you meant that you had no Markov clients running. And so I think you're asking about errors on the fixed 5MiB transfer perf clients. 30% (Also, note that timeouts are more common for the Markov model clients because the probabilistic delay model can eventually lead to a timeout.) |
Beta Was this translation helpful? Give feedback.
-
Tor monitors its memory usage and if it becomes near this limit, it will start destroying circuits. This would probably show up as an EOF for tgen streams. But if you aren't seeing any "We're low on memory" log messages, then this doesn't appear to be happening. When the tornettools simulation finishes, what does the tonettools log say the returned code was? (What's the code in the log line "Done simulating; shadow returned code '_'")? |
Beta Was this translation helpful? Give feedback.
-
|
Can you also run one of your memory-intensive simulations and check for oom-killed processes immediately after with something like |
Beta Was this translation helpful? Give feedback.
-
|
Shadow returns code 0 for all of these 1% simulations, even the ones which exhaust all of the system's memory, but if I go above about 1.5% it starts returning error codes. I checked again to confirm that none of the Tor logs have any messages about lack of memory. Unfortunately, I don't have read access to /var/log/syslog on this machine, but it might be somewhat telling that the simulation that uses the most memory always outputs a similar minimum value of available memory in free.log, perhaps some sort of threshold below which processes begin to be killed? |
Beta Was this translation helpful? Give feedback.
-
|
That's what I'm thinking as well. I also don't think Shadow is correctly handling processes that being killed. I made an issue for that in #2151. |
Beta Was this translation helpful? Give feedback.
-
|
For now, I'll try running some scaled-down simulations to see if that changes much about the errors. It seems a bit weird that increasing the scale further results in Shadow returning an error code, maybe because memory issues are happening earlier, but there definitely seems to be a correlation. |
Beta Was this translation helpful? Give feedback.
I hope you're finding our tools useful :)
Just so we're on the same page, tornettools will create two main client types: perf clients and Markov clients. I think when you stated you had "no padding machines enabled" I think you meant that you had no Markov clients running. And so I think you're asking about errors on the fixed 5MiB transfer perf clients.
30%
READEOFerrors does indeed sound bad. I don't generally get error rates that high. Something that could possibly make sense could be a timeout, since it's only happening on 5MiB downloads an…