-
Notifications
You must be signed in to change notification settings - Fork 5k
[Bug] [MasterServer] With MySQL 8.0, Druid make thread blocked to wait lock when load class #6483
Description
Search before asking
- I had searched in the issues and found no similar issues.
What happened
"MasterEventExecution" #130 daemon prio=5 os_prio=0 tid=0x00007f3590024000 nid=0x355414 waiting for monitor entry [0x00007f35843fe000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.ClassLoader.loadClass(ClassLoader.java:404) - waiting to lock <0x0000000740022240> (a java.lang.Object) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.alibaba.druid.util.Utils.loadClass(Utils.java:212) at com.alibaba.druid.util.MySqlUtils.getLastPacketReceivedTimeMs(MySqlUtils.java:372) at com.alibaba.druid.pool.DruidAbstractDataSource.testConnectionInternal(DruidAbstractDataSource.java:1471) at com.alibaba.druid.pool.DruidDataSource.recycle(DruidDataSource.java:1938) at com.alibaba.druid.pool.DruidPooledConnection.recycle(DruidPooledConnection.java:324) at com.alibaba.druid.pool.DruidPooledConnection.close(DruidPooledConnection.java:269) at org.springframework.jdbc.datasource.DataSourceUtils.doCloseConnection(DataSourceUtils.java:360) at org.springframework.jdbc.datasource.DataSourceUtils.doReleaseConnection(DataSourceUtils.java:347) at org.springframework.jdbc.datasource.DataSourceUtils.releaseConnection(DataSourceUtils.java:314) at org.mybatis.spring.transaction.SpringManagedTransaction.close(SpringManagedTransaction.java:115) at org.apache.ibatis.executor.BaseExecutor.close(BaseExecutor.java:90) at org.apache.ibatis.session.defaults.DefaultSqlSession.close(DefaultSqlSession.java:263) at org.mybatis.spring.SqlSessionUtils.closeSqlSession(SqlSessionUtils.java:195) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:448) at com.sun.proxy.$Proxy90.selectOne(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159) at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:89) at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61) at com.sun.proxy.$Proxy95.selectById(Unknown Source) at org.apache.dolphinscheduler.service.process.ProcessService.findProcessInstanceById(ProcessService.java:379) at org.apache.dolphinscheduler.service.process.ProcessService$$FastClassBySpringCGLIB$$ed138739.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687) at org.apache.dolphinscheduler.service.process.ProcessService$$EnhancerBySpringCGLIB$$f0a248d9.findProcessInstanceById(<generated>) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.updateProcessInstanceState(WorkflowExecuteThread.java:1085) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.submitPostNode(WorkflowExecuteThread.java:839) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.taskFinished(WorkflowExecuteThread.java:384) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.taskStateChangeHandler(WorkflowExecuteThread.java:353) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.stateEventHandler(WorkflowExecuteThread.java:302) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.handleEvents(WorkflowExecuteThread.java:250) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.run(WorkflowExecuteThread.java:231) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
branch: dev
database: mysql 8.0.21
The MasterServer jstack show that when druid load class, many threads is blocked to wait lock.
I found that when I change durid config like that:
spring.datasource.testOnBorrow=false spring.datasource.testOnReturn=false
can solve this problem, but I think it's not the right way.
See druid issue: alibaba/druid#3808.
Maybe we should upgrade the mysql connector version and druid version.
What you expected to happen
thread not blocked by druid when loadClass.
How to reproduce
run by complete data mode, and use jstack MasterServer
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct