Skip to content

[fix][flaky-test] Fix ClassCastException: BrokerService cannot be cast to class PulsarResources#16821

Merged
codelipenghui merged 4 commits into
apache:masterfrom
poorbarcode:fix/flacky/classCast
Jul 28, 2022
Merged

[fix][flaky-test] Fix ClassCastException: BrokerService cannot be cast to class PulsarResources#16821
codelipenghui merged 4 commits into
apache:masterfrom
poorbarcode:fix/flacky/classCast

Conversation

@poorbarcode

@poorbarcode poorbarcode commented Jul 27, 2022

Copy link
Copy Markdown
Contributor

Motivation

Such as PersistentTopicTest, ServerCnxAuthorizationTest, PersistentDispatcherFailoverConsumerTest and MessageCumulativeAckTest is very flaky, will throws ClassCastException like this:

class-case

I found the cause of the problem: while executing doReturn(pulsarResources).when(pulsar).getPulsarResources(), Meta Store Thread also accesses variable PulsarService.getPulsarResources() asynchronously in logic: notification by zk-watcher(Concurrent access will be problematic if the object is being mock bound).

Modifications

Execute mock-cmd in meta-thread (The executor of MetaStore is a single thread pool executor, so all things will be thread safety).

Documentation

Check the box below or label this PR directly.

Need to update docs?

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-complete
    (Docs have been already added)

@shibd

shibd commented Jul 27, 2022

Copy link
Copy Markdown
Member

@poorbarcode The same problem is happening here, can you take a look at it together? #16666

@poorbarcode

Copy link
Copy Markdown
Contributor Author

Hi @shibd

The same problem is happening here, can you take a look at it together? #16666

Already included, it is PersistentTopicTest( the super class of PersistentTopicStreamingDispatcherTest ).

@shibd shibd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch! I tested it locally and it can work.

@poorbarcode

Copy link
Copy Markdown
Contributor Author

Nice catch! I tested it locally and it can work.

Thanks.

@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Jul 27, 2022
@poorbarcode

Copy link
Copy Markdown
Contributor Author

/pulsarbot rerun-failure-checks

@mattisonchao mattisonchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

@codelipenghui codelipenghui merged commit 4ee3466 into apache:master Jul 28, 2022
@nodece nodece mentioned this pull request Jul 28, 2022
@mattisonchao

Copy link
Copy Markdown
Member

It looks like Mockito's bug, but I'm still not sure how Mockito causes these things. So, I'll try to find the root cause, and I'll post back here when I find out.

Gleiphir2769 pushed a commit to Gleiphir2769/pulsar that referenced this pull request Aug 4, 2022
codelipenghui pushed a commit that referenced this pull request Aug 8, 2022
…t to class PulsarResources (#16821)

(cherry picked from commit 4ee3466)
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Aug 16, 2022
…t to class PulsarResources (apache#16821)

(cherry picked from commit 4ee3466)
(cherry picked from commit 41d7cf1)
@poorbarcode poorbarcode deleted the fix/flacky/classCast branch September 17, 2022 02:51
congbobo184 pushed a commit that referenced this pull request Nov 10, 2022
…t to class PulsarResources (#16821)

(cherry picked from commit 4ee3466)
@congbobo184 congbobo184 added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label Nov 10, 2022
congbobo184 pushed a commit that referenced this pull request Dec 1, 2022
…t to class PulsarResources (#16821)

(cherry picked from commit 4ee3466)
@lhotari

lhotari commented Jan 9, 2023

Copy link
Copy Markdown
Member

I don't see how this could fix the issue. Here's an example of a recent problem

Error:  Tests run: 45, Failures: 1, Errors: 0, Skipped: 32, Time elapsed: 41.251 s <<< FAILURE! - in org.apache.pulsar.broker.service.persistent.PersistentTopicStreamingDispatcherTest
  Error:  setup(org.apache.pulsar.broker.service.persistent.PersistentTopicStreamingDispatcherTest)  Time elapsed: 0.35 s  <<< FAILURE!
  org.mockito.exceptions.base.MockitoException: Unable to create mock instance of type 'ServerCnx'
  	at org.apache.pulsar.broker.BrokerTestUtil.spyWithClassAndConstructorArgsRecordingInvocations(BrokerTestUtil.java:61)
  	at org.apache.pulsar.broker.service.PersistentTopicTest.setup(PersistentTopicTest.java:233)
  	at org.apache.pulsar.broker.service.persistent.PersistentTopicStreamingDispatcherTest.setup(PersistentTopicStreamingDispatcherTest.java:34)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
  	at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
  	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:590)
  	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
  	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
  	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
  	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
  	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
  	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
  	at org.testng.TestRunner.privateRun(TestRunner.java:829)
  	at org.testng.TestRunner.run(TestRunner.java:602)
  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
  	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
  	at org.testng.TestNG.runSuites(TestNG.java:[1099](https://github.com/apache/pulsar/actions/runs/3875096798/jobs/6609869449#step:11:1100))
  	at org.testng.TestNG.run(TestNG.java:1067)
  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90)
  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
  	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
  Caused by: org.mockito.creation.instance.InstantiationException: 
  Unable to create instance of 'ServerCnx'.
  Please ensure the target class has a constructor that matches these argument types: [org.apache.pulsar.broker.PulsarService] and executes cleanly.
  	... 41 more
  Caused by: java.lang.reflect.InvocationTargetException
  	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.newInstance(InstrumentationMemberAccessor.java:198)
  	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.newInstance(InstrumentationMemberAccessor.java:161)
  	at org.mockito.internal.util.reflection.ModuleMemberAccessor.newInstance(ModuleMemberAccessor.java:42)
  	at org.mockito.internal.creation.instance.ConstructorInstantiator.invokeConstructor(ConstructorInstantiator.java:70)
  	at org.mockito.internal.creation.instance.ConstructorInstantiator.withParams(ConstructorInstantiator.java:53)
  	at org.mockito.internal.creation.instance.ConstructorInstantiator.newInstance(ConstructorInstantiator.java:39)
  	at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.doCreateMock(InlineDelegateByteBuddyMockMaker.java:360)
  	at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.createMock(InlineDelegateByteBuddyMockMaker.java:330)
  	at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.createMock(InlineByteBuddyMockMaker.java:58)
  	at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:53)
  	at org.mockito.internal.MockitoCore.mock(MockitoCore.java:84)
  	at org.mockito.Mockito.mock(Mockito.java:1964)
  	... 41 more
  Caused by: java.lang.ClassCastException: class org.apache.pulsar.broker.service.BrokerService cannot be cast to class org.apache.pulsar.broker.resources.PulsarResources (org.apache.pulsar.broker.service.BrokerService and org.apache.pulsar.broker.resources.PulsarResources are in unnamed module of loader 'app')
  	at org.apache.pulsar.broker.PulsarService.getPulsarResources(PulsarService.java:267)
  	at org.apache.pulsar.broker.service.TopicListService.<init>(TopicListService.java:103)
  	at org.apache.pulsar.broker.service.ServerCnx.<init>(ServerCnx.java:297)
  	at org.apache.pulsar.broker.service.ServerCnx.<init>(ServerCnx.java:255)
  	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
  	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor$Dispatcher$ByteBuddy$NFN5n4xN.invokeWithArguments(Unknown Source)
  	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.lambda$newInstance$0(InstrumentationMemberAccessor.java:191)
  	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.newInstance(InstrumentationMemberAccessor.java:188)
  	... 52 more

happens in https://github.com/apache/pulsar/actions/runs/3875096798/jobs/6609869449#step:11:1088

@lhotari

lhotari commented Jan 25, 2023

Copy link
Copy Markdown
Member

@poorbarcode This solution didn't fix the issue. Your analysis of the problem was correct. I have created a solution where the getters are no longer overridden using Mockito. The PR is #19323, please review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants