Add compacted topic metrics for TopicStats in CLI #11564

Technoboy- · 2021-08-05T08:01:51Z

Motivation

Add below metrics to help track potential flows or examine the overall condition of compacted topics .

lastCompactionRemovedEventCount : the removed event count of last compaction
lastCompactionSucceedTimestamp : the timestamp of last succeed compaction
lastCompactionFailedTimestamp : the timestamp of last failed compaction
lastCompactionDurationTimeInMills: the duration time of last compaction

These 4 metrics will be displayed in topic stats CLI :

./pulsar-admin topics stats persistent://tenant/ns/topic

Documentation

This patch will add metrics in CLI , which would generate doc automatically.

Technoboy- · 2021-08-06T10:07:58Z

/pulsarbot run-failure-checks

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

michaeljmarshall

I suggested a few changes and asked some questions about design. I think the Sum class needs tests before this can be merged.

pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java

pulsar-common/src/main/java/org/apache/pulsar/common/stats/Sum.java

Technoboy- · 2021-08-10T01:11:37Z

I suggested a few changes and asked some questions about design. I think the Sum class needs tests before this can be merged.

Thanks for reviewing. Yesterday, we find that compacted topic has metrics in InternalStats which would be shown by:

./pulsar-admin topics stats-internal persistent://tenant/ns/topic

I may have to record other metrics except the above two compactedTopicSize ,compactedTopicMsgCount

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/CompactStats.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java

pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/CompactStats.java

pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java

pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBean.java

gaoran10

LGTM

codelipenghui · 2021-08-18T02:40:28Z

@michaeljmarshall Please help review the PR again.

codelipenghui · 2021-08-19T05:04:26Z

@michaeljmarshall Please help review the PR again.

michaeljmarshall

I am concerned that the 2 hour fixed rate for cleanup will lead to incorrect metrics. I think it could be possible to implement the cleanup in a way that removes the possibility of serving incorrect metrics. Please let me know what you think.

Otherwise, this PR looks good to me.

pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java

pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/CompactStats.java

Technoboy- · 2021-08-19T14:07:51Z

@michaeljmarshall @gaoran10 @codelipenghui thanks for reviewing.

michaeljmarshall

LGTM. Good work! I left a couple comments. It's up to you if you want to address them or leave the PR as is.

...r-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/CompactionStats.java

pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java

codelipenghui · 2021-08-21T04:12:06Z

Error:  Tests run: 28, Failures: 12, Errors: 0, Skipped: 13, Time elapsed: 601.287 s <<< FAILURE! - in org.apache.pulsar.functions.source.batch.BatchSourceExecutorTest
Error:  testPushLifeCycle(org.apache.pulsar.functions.source.batch.BatchSourceExecutorTest)  Time elapsed: 300.011 s  <<< FAILURE!
org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.functions.source.batch.BatchSourceExecutorTest.testPushLifeCycle() didn't finish within the time-out 300000
	at org.testng.internal.MethodInvocationHelper.invokeWithTimeoutWithNewExecutor(MethodInvocationHelper.java:371)
	at org.testng.internal.MethodInvocationHelper.invokeWithTimeout(MethodInvocationHelper.java:282)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:605)
	at org.testng.internal.TestInvoker.retryFailed(TestInvoker.java:214)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:58)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.testng.TestRunner.privateRun(TestRunner.java:764)
	at org.testng.TestRunner.run(TestRunner.java:585)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
	at org.testng.TestNG.runSuites(TestNG.java:1069)
	at org.testng.TestNG.run(TestNG.java:1037)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

Technoboy- · 2021-08-21T05:53:56Z

/pulsarbot run-failure-checks

Technoboy- · 2021-08-21T12:59:33Z

/pulsarbot run-failure-checks

Technoboy- · 2021-08-21T13:53:59Z

/pulsarbot run-failure-checks

### Motivation As #11564 has involved compaction metrics in CLI, it's extremely useful to expose relative metrics to Prometheus. - pulsar_compaction_removed_event_count : the removed event count of compaction . - pulsar_compaction_succeed_count : the succeed count of compaction. - pulsar_compaction_failed_count : the failed count of compaction. - pulsar_compaction_duration_time_in_millis : the duration time of compaction. - pulsar_compaction_read_throughput : the read throughput of compaction. - pulsar_compaction_write_throughput : the write throughput of compaction. - pulsar_compaction_compacted_entries_count: the compacted entries count. - pulsar_compaction_compacted_entries_size: the compacted entries size; if users enable the topic level metrics and the topic has been compacted or doing the compaction, we should expose the compaction metrics for the topic level. If users disable the topic level metrics, we should expose the aggregated compaction metrics for the namespace level.

Add below metrics to help track potential flows or examine the overall condition of compacted topics . - lastCompactionRemovedEventCount : the removed event count of last compaction - lastCompactionSucceedTimestamp : the timestamp of last succeed compaction - lastCompactionFailedTimestamp : the timestamp of last failed compaction - lastCompactionDurationTimeInMills: the duration time of last compaction These 4 metrics will be displayed in topic stats CLI : ``` ./pulsar-admin topics stats persistent://tenant/ns/topic ``` This patch will add metrics in CLI , which would generate doc automatically. (cherry picked from commit c0ef593)

### Motivation As #11564 has involved compaction metrics in CLI, it's extremely useful to expose relative metrics to Prometheus. - pulsar_compaction_removed_event_count : the removed event count of compaction . - pulsar_compaction_succeed_count : the succeed count of compaction. - pulsar_compaction_failed_count : the failed count of compaction. - pulsar_compaction_duration_time_in_millis : the duration time of compaction. - pulsar_compaction_read_throughput : the read throughput of compaction. - pulsar_compaction_write_throughput : the write throughput of compaction. - pulsar_compaction_compacted_entries_count: the compacted entries count. - pulsar_compaction_compacted_entries_size: the compacted entries size; if users enable the topic level metrics and the topic has been compacted or doing the compaction, we should expose the compaction metrics for the topic level. If users disable the topic level metrics, we should expose the aggregated compaction metrics for the namespace level. (cherry picked from commit 656635e)

Add below metrics to help track potential flows or examine the overall condition of compacted topics . - lastCompactionRemovedEventCount : the removed event count of last compaction - lastCompactionSucceedTimestamp : the timestamp of last succeed compaction - lastCompactionFailedTimestamp : the timestamp of last failed compaction - lastCompactionDurationTimeInMills: the duration time of last compaction These 4 metrics will be displayed in topic stats CLI : ``` ./pulsar-admin topics stats persistent://tenant/ns/topic ``` This patch will add metrics in CLI , which would generate doc automatically. (cherry picked from commit c0ef593) (cherry picked from commit 71cc13b)

### Motivation As apache#11564 has involved compaction metrics in CLI, it's extremely useful to expose relative metrics to Prometheus. - pulsar_compaction_removed_event_count : the removed event count of compaction . - pulsar_compaction_succeed_count : the succeed count of compaction. - pulsar_compaction_failed_count : the failed count of compaction. - pulsar_compaction_duration_time_in_millis : the duration time of compaction. - pulsar_compaction_read_throughput : the read throughput of compaction. - pulsar_compaction_write_throughput : the write throughput of compaction. - pulsar_compaction_compacted_entries_count: the compacted entries count. - pulsar_compaction_compacted_entries_size: the compacted entries size; if users enable the topic level metrics and the topic has been compacted or doing the compaction, we should expose the compaction metrics for the topic level. If users disable the topic level metrics, we should expose the aggregated compaction metrics for the namespace level. (cherry picked from commit 656635e) (cherry picked from commit a80f1e8)

### Motivation Add below metrics to help track potential flows or examine the overall condition of compacted topics . - lastCompactionRemovedEventCount : the removed event count of last compaction - lastCompactionSucceedTimestamp : the timestamp of last succeed compaction - lastCompactionFailedTimestamp : the timestamp of last failed compaction - lastCompactionDurationTimeInMills: the duration time of last compaction These 4 metrics will be displayed in topic stats CLI : ``` ./pulsar-admin topics stats persistent://tenant/ns/topic ``` ### Documentation This patch will add metrics in CLI , which would generate doc automatically.

### Motivation As apache#11564 has involved compaction metrics in CLI, it's extremely useful to expose relative metrics to Prometheus. - pulsar_compaction_removed_event_count : the removed event count of compaction . - pulsar_compaction_succeed_count : the succeed count of compaction. - pulsar_compaction_failed_count : the failed count of compaction. - pulsar_compaction_duration_time_in_millis : the duration time of compaction. - pulsar_compaction_read_throughput : the read throughput of compaction. - pulsar_compaction_write_throughput : the write throughput of compaction. - pulsar_compaction_compacted_entries_count: the compacted entries count. - pulsar_compaction_compacted_entries_size: the compacted entries size; if users enable the topic level metrics and the topic has been compacted or doing the compaction, we should expose the compaction metrics for the topic level. If users disable the topic level metrics, we should expose the aggregated compaction metrics for the namespace level.

Technoboy- added a commit to Technoboy-/pulsar that referenced this pull request Aug 5, 2021

Add compacted topic metrics for TopicStats in CLI (apache#11564)

81a7b49

Technoboy- force-pushed the add-compacted-topic-metric-1 branch from c49abed to 81a7b49 Compare August 5, 2021 08:45

zymap requested review from eolivelli, merlimat and zymap August 5, 2021 09:47

zymap assigned Technoboy- Aug 5, 2021

zymap added area/admin area/metrics labels Aug 5, 2021

zymap added this to the 2.9.0 milestone Aug 5, 2021

zymap requested review from codelipenghui and sijie and removed request for zymap August 5, 2021 10:08

Anonymitaet added the doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. label Aug 5, 2021

codelipenghui requested review from 315157973 and hangc0276 August 7, 2021 03:10

michaeljmarshall reviewed Aug 9, 2021

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java Outdated Show resolved Hide resolved

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java Outdated Show resolved Hide resolved

michaeljmarshall requested changes Aug 9, 2021

View reviewed changes

Technoboy- changed the title ~~Add compacted topic metrics for TopicStats in CLI~~ [WIP]Add compacted topic metrics for TopicStats in CLI Aug 10, 2021

Technoboy- added a commit to Technoboy-/pulsar that referenced this pull request Aug 10, 2021

Add compacted topic metrics for TopicStats in CLI (apache#11564)

c6beb44

Technoboy- force-pushed the add-compacted-topic-metric-1 branch from f729c2c to c6beb44 Compare August 10, 2021 10:18

Technoboy- changed the title ~~[WIP]Add compacted topic metrics for TopicStats in CLI~~ Add compacted topic metrics for TopicStats in CLI Aug 10, 2021

michaeljmarshall requested changes Aug 10, 2021

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java Outdated Show resolved Hide resolved

pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/CompactStats.java Show resolved Hide resolved

Technoboy- added a commit to Technoboy-/pulsar that referenced this pull request Aug 11, 2021

Add compacted topic metrics for TopicStats in CLI (apache#11564)

89c90f9

Technoboy- force-pushed the add-compacted-topic-metric-1 branch from e8b4d0d to 89c90f9 Compare August 11, 2021 05:28

michaeljmarshall reviewed Aug 11, 2021

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java Outdated Show resolved Hide resolved

pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/CompactStats.java Show resolved Hide resolved

codelipenghui reviewed Aug 13, 2021

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java Outdated Show resolved Hide resolved

codelipenghui reviewed Aug 15, 2021

View reviewed changes

Technoboy- added a commit to Technoboy-/pulsar that referenced this pull request Aug 16, 2021

Add compacted topic metrics for TopicStats in CLI (apache#11564)

1a8bdac

gaoran10 approved these changes Aug 17, 2021

View reviewed changes

codelipenghui approved these changes Aug 19, 2021

View reviewed changes

michaeljmarshall requested changes Aug 19, 2021

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java Outdated Show resolved Hide resolved

pulsar-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/CompactStats.java Outdated Show resolved Hide resolved

michaeljmarshall approved these changes Aug 20, 2021

View reviewed changes

...r-client-admin-api/src/main/java/org/apache/pulsar/common/policies/data/CompactionStats.java Outdated Show resolved Hide resolved

pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactorMXBeanImpl.java Outdated Show resolved Hide resolved

sijie approved these changes Aug 21, 2021

View reviewed changes

Add compacted topic metrics for TopicStats in CLI (apache#11564)

c4b6e71

Technoboy- force-pushed the add-compacted-topic-metric-1 branch from 6cbbb12 to c4b6e71 Compare August 21, 2021 02:50

Technoboy- added 2 commits August 21, 2021 10:54

rename from compact to compaction

04d300e

fix test

96732c0

codelipenghui merged commit c0ef593 into apache:master Aug 21, 2021

Technoboy- mentioned this pull request Aug 22, 2021

Expose compaction metrics to Prometheus #11739

Merged

codelipenghui added the release/2.8.2 label Aug 23, 2021

codelipenghui added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Sep 2, 2021

hangc0276 added release/2.8.1 and removed release/2.8.2 labels Sep 4, 2021

Technoboy- deleted the add-compacted-topic-metric-1 branch August 10, 2022 05:51

Add compacted topic metrics for TopicStats in CLI #11564

Add compacted topic metrics for TopicStats in CLI #11564

Uh oh!

Conversation

Technoboy- commented Aug 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Documentation

Uh oh!

Technoboy- commented Aug 6, 2021

Uh oh!

Uh oh!

Uh oh!

michaeljmarshall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Technoboy- commented Aug 10, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaoran10 left a comment

Choose a reason for hiding this comment

Uh oh!

codelipenghui commented Aug 18, 2021

Uh oh!

codelipenghui commented Aug 19, 2021

Uh oh!

michaeljmarshall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Technoboy- commented Aug 19, 2021

Uh oh!

michaeljmarshall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codelipenghui commented Aug 21, 2021

Uh oh!

Technoboy- commented Aug 21, 2021

Uh oh!

Technoboy- commented Aug 21, 2021

Uh oh!

Technoboy- commented Aug 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Technoboy- commented Aug 5, 2021 •

edited

Loading