Skip to content

[Bug] [MasterServer] With MySQL 8.0, Druid make thread blocked to wait lock when load class  #6483

@caishunfeng

Description

@caishunfeng

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

"MasterEventExecution" #130 daemon prio=5 os_prio=0 tid=0x00007f3590024000 nid=0x355414 waiting for monitor entry [0x00007f35843fe000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.ClassLoader.loadClass(ClassLoader.java:404) - waiting to lock <0x0000000740022240> (a java.lang.Object) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.alibaba.druid.util.Utils.loadClass(Utils.java:212) at com.alibaba.druid.util.MySqlUtils.getLastPacketReceivedTimeMs(MySqlUtils.java:372) at com.alibaba.druid.pool.DruidAbstractDataSource.testConnectionInternal(DruidAbstractDataSource.java:1471) at com.alibaba.druid.pool.DruidDataSource.recycle(DruidDataSource.java:1938) at com.alibaba.druid.pool.DruidPooledConnection.recycle(DruidPooledConnection.java:324) at com.alibaba.druid.pool.DruidPooledConnection.close(DruidPooledConnection.java:269) at org.springframework.jdbc.datasource.DataSourceUtils.doCloseConnection(DataSourceUtils.java:360) at org.springframework.jdbc.datasource.DataSourceUtils.doReleaseConnection(DataSourceUtils.java:347) at org.springframework.jdbc.datasource.DataSourceUtils.releaseConnection(DataSourceUtils.java:314) at org.mybatis.spring.transaction.SpringManagedTransaction.close(SpringManagedTransaction.java:115) at org.apache.ibatis.executor.BaseExecutor.close(BaseExecutor.java:90) at org.apache.ibatis.session.defaults.DefaultSqlSession.close(DefaultSqlSession.java:263) at org.mybatis.spring.SqlSessionUtils.closeSqlSession(SqlSessionUtils.java:195) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:448) at com.sun.proxy.$Proxy90.selectOne(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:159) at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:89) at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61) at com.sun.proxy.$Proxy95.selectById(Unknown Source) at org.apache.dolphinscheduler.service.process.ProcessService.findProcessInstanceById(ProcessService.java:379) at org.apache.dolphinscheduler.service.process.ProcessService$$FastClassBySpringCGLIB$$ed138739.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687) at org.apache.dolphinscheduler.service.process.ProcessService$$EnhancerBySpringCGLIB$$f0a248d9.findProcessInstanceById(<generated>) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.updateProcessInstanceState(WorkflowExecuteThread.java:1085) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.submitPostNode(WorkflowExecuteThread.java:839) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.taskFinished(WorkflowExecuteThread.java:384) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.taskStateChangeHandler(WorkflowExecuteThread.java:353) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.stateEventHandler(WorkflowExecuteThread.java:302) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.handleEvents(WorkflowExecuteThread.java:250) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteThread.run(WorkflowExecuteThread.java:231) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

branch: dev
database: mysql 8.0.21

The MasterServer jstack show that when druid load class, many threads is blocked to wait lock.
I found that when I change durid config like that:
spring.datasource.testOnBorrow=false spring.datasource.testOnReturn=false
can solve this problem, but I think it's not the right way.

See druid issue: alibaba/druid#3808.
Maybe we should upgrade the mysql connector version and druid version.

What you expected to happen

thread not blocked by druid when loadClass.

How to reproduce

run by complete data mode, and use jstack MasterServer

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions