Search before asking
What happened
"MasterEventExecution" #130 daemon prio=5 os_prio=0 tid=0x00007f3590024000 nid=0x355414 waiting for monitor entry [0x00007f35843fe000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.ClassLoader.loadClass(ClassLoader.java:404) - waiting to lock <0x0000000740022240> (a java.lang.Object) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.alibaba.druid.util.Utils.loadClass(Utils.java:212) at com.alibaba.druid.util.MySqlUtils.getLastPacketReceivedTimeMs(MySqlUtils.java:372) at com.alibaba.druid.pool.DruidAbstractDataSource.testConnectionInternal(DruidAbstractDataSource.java:1471) at com.alibaba.druid.pool.DruidDataSource.recycle(DruidDataSource.java:1938) at com.alibaba.druid.pool.DruidPooledConnection.recycle(DruidPooledConnection.java:324) at com.alibaba.druid.pool.DruidPooledConnection.close(DruidPooledConnection.java:269) at org.springframework.jdbc.datasource.DataSourceUtils.doCloseConnection(DataSourceUtils.java:360) at org.springframework.jdbc.datasource.DataSourceUtils.doReleaseConnection(DataSourceUtils.java:347) at org.springframework.jdbc.datasource.DataSourceUtils.releaseConnection(DataSourceUtils.java:314) at org.mybatis.spring.transaction.SpringManagedTransaction.close(SpringManagedTransaction.java:115) at org.apache.ibatis.executor.BaseExecutor.close(BaseExecutor.java:90) at org.apache.ibatis.session.defaults.DefaultSqlSession.close(DefaultSqlSession.java:263) at org.mybatis.spring.SqlSessionUtils.closeSqlSession(SqlSessionUtils.java:195) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:448) at com.sun.proxy.$Proxy90.selectOne(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159) at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:89) at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61) at com.sun.proxy.$Proxy95.selectById(Unknown Source) at org.apache.dolphinscheduler.service.process.ProcessService.findProcessInstanceById(ProcessService.java:379) at org.apache.dolphinscheduler.service.process.ProcessService$$FastClassBySpringCGLIB$$ed138739.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687) at org.apache.dolphinscheduler.service.process.ProcessService$$EnhancerBySpringCGLIB$$f0a248d9.findProcessInstanceById(<generated>) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.updateProcessInstanceState(WorkflowExecuteThread.java:1085) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.submitPostNode(WorkflowExecuteThread.java:839) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.taskFinished(WorkflowExecuteThread.java:384) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.taskStateChangeHandler(WorkflowExecuteThread.java:353) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.stateEventHandler(WorkflowExecuteThread.java:302) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.handleEvents(WorkflowExecuteThread.java:250) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.run(WorkflowExecuteThread.java:231) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
branch: dev
database: mysql 8.0.21
The MasterServer jstack show that when druid load class, many threads is blocked to wait lock.
I found that when I change durid config like that:
spring.datasource.testOnBorrow=false spring.datasource.testOnReturn=false
can solve this problem, but I think it's not the right way.
See druid issue: alibaba/druid#3808.
Maybe we should upgrade the mysql connector version and druid version.
What you expected to happen
thread not blocked by druid when loadClass.
How to reproduce
run by complete data mode, and use jstack MasterServer
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Search before asking
What happened
"MasterEventExecution" #130 daemon prio=5 os_prio=0 tid=0x00007f3590024000 nid=0x355414 waiting for monitor entry [0x00007f35843fe000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.ClassLoader.loadClass(ClassLoader.java:404) - waiting to lock <0x0000000740022240> (a java.lang.Object) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.alibaba.druid.util.Utils.loadClass(Utils.java:212) at com.alibaba.druid.util.MySqlUtils.getLastPacketReceivedTimeMs(MySqlUtils.java:372) at com.alibaba.druid.pool.DruidAbstractDataSource.testConnectionInternal(DruidAbstractDataSource.java:1471) at com.alibaba.druid.pool.DruidDataSource.recycle(DruidDataSource.java:1938) at com.alibaba.druid.pool.DruidPooledConnection.recycle(DruidPooledConnection.java:324) at com.alibaba.druid.pool.DruidPooledConnection.close(DruidPooledConnection.java:269) at org.springframework.jdbc.datasource.DataSourceUtils.doCloseConnection(DataSourceUtils.java:360) at org.springframework.jdbc.datasource.DataSourceUtils.doReleaseConnection(DataSourceUtils.java:347) at org.springframework.jdbc.datasource.DataSourceUtils.releaseConnection(DataSourceUtils.java:314) at org.mybatis.spring.transaction.SpringManagedTransaction.close(SpringManagedTransaction.java:115) at org.apache.ibatis.executor.BaseExecutor.close(BaseExecutor.java:90) at org.apache.ibatis.session.defaults.DefaultSqlSession.close(DefaultSqlSession.java:263) at org.mybatis.spring.SqlSessionUtils.closeSqlSession(SqlSessionUtils.java:195) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:448) at com.sun.proxy.$Proxy90.selectOne(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159) at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:89) at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61) at com.sun.proxy.$Proxy95.selectById(Unknown Source) at org.apache.dolphinscheduler.service.process.ProcessService.findProcessInstanceById(ProcessService.java:379) at org.apache.dolphinscheduler.service.process.ProcessService$$FastClassBySpringCGLIB$$ed138739.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687) at org.apache.dolphinscheduler.service.process.ProcessService$$EnhancerBySpringCGLIB$$f0a248d9.findProcessInstanceById(<generated>) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.updateProcessInstanceState(WorkflowExecuteThread.java:1085) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.submitPostNode(WorkflowExecuteThread.java:839) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.taskFinished(WorkflowExecuteThread.java:384) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.taskStateChangeHandler(WorkflowExecuteThread.java:353) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.stateEventHandler(WorkflowExecuteThread.java:302) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.handleEvents(WorkflowExecuteThread.java:250) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.run(WorkflowExecuteThread.java:231) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)branch: dev
database: mysql 8.0.21
The MasterServer jstack show that when druid load class, many threads is blocked to wait lock.
I found that when I change durid config like that:
spring.datasource.testOnBorrow=false spring.datasource.testOnReturn=falsecan solve this problem, but I think it's not the right way.
See druid issue: alibaba/druid#3808.
Maybe we should upgrade the mysql connector version and druid version.
What you expected to happen
thread not blocked by druid when loadClass.
How to reproduce
run by complete data mode, and use jstack MasterServer
Anything else
No response
Are you willing to submit PR?
Code of Conduct