Need a limited Threadpool in consumer side #2013
Description
Environment
- Dubbo version: 2.5.3
- Java version: 1.7
in some circumstances, too many threads will be created and thus process will suffer from OOM.Here is the related issue #1932
After analyzing the problem, I found that the consumer used a cached thread pool which has no limited thread size, which is the root cause. So a choice for users to limit the max size of consumer thread is needed.
Here is one case to reproduce the problem:
consumer A call service from provider B in a very big tps, but provider B is not responding very quickly in some times due to some pause, and the calls timeout, but after a short time, provider B becomes quick so the large number of responses send back to consumer A nearly in the same time. In this case, you will find many many consumer threads are created in consumer side due to the current design.
You can easily reproduce this issue with the following provider sample to simulate the provider problem which timeout response send back in the same time to consumer side, and then consumer created many threads in this case.
public class MockImpl implements Mock {
private static final Logger logger = LoggerFactory.getLogger(MockImpl.class);
public void sleep(long ms) {
long span = computeNextMinuteSpan();
logger.info("begin to sleep "+ms+" ms");
try {
Thread.sleep(ms);
} catch (InterruptedException e) {
e.printStackTrace();
}
logger.info("after sleep " + ms + " ms");
}
public void sleepToNextMinute() {
long span = computeNextMinuteSpan();
sleep(span);
}
public static long computeNextMinuteSpan() {
long now = System.currentTimeMillis();
Calendar cal = Calendar.getInstance();
cal.setTimeInMillis(System.currentTimeMillis());
cal.add(Calendar.MINUTE, 1);
cal.set(Calendar.SECOND, 0);
cal.set(Calendar.MILLISECOND, 0);
return cal.getTimeInMillis() - now;
}
}
- export this provider with a short timeout say 10ms, with big thread count say 1000. And if you export many providers instead of only one, the problem will be more obvious.
- in a for loop, continuously consume the service
sleepToNextMinutein a single thread or in thread pool. - Observe the thread count of DubboClient .
Here is a sample for consumer
logger.info("sleeping till next minute ......");
Thread.sleep(computeNextMinuteSpan());
Executors.newScheduledThreadPool(1).scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
int i = 0;
Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
for (Thread t : threadSet) {
if (t.getName().startsWith("DubboClientHandler")) {
logger.info("dubbo thread: {}",t.getName());
i++;
}
}
logger.info("=================================Dubbo Thread {}===================================",i);
}
},0,1000, TimeUnit.MILLISECONDS);
logger.info("mocking...");
for (int i =0;i<10000;i++) {
try {
mock.sleepToNextMinute();
} catch (Exception e) {
//logger.error(e.getMessage());
}
}
logger.info("mocking ends");
You can easily find that the consumer threads increase heavily in a very short time even if you are calling the provider service with only one thread