feat: Defer cleanup for log/index compactions, add debug log by devanbenz · Pull Request #26511 · influxdata/influxdb

devanbenz · 2025-06-10T22:03:32Z

I believe that there is something happening which causes CurrentCompactionN() to always be greater than 0. Thus making Partition.Wait() hang forever.

Taking a look at some profiles where this issue occurs. I'm seeing a consistent one where we're stuck on Partition.Wait()

-----------+-------------------------------------------------------
         1   runtime.gopark
             runtime.chanrecv
             runtime.chanrecv1
             github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).Wait
             github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).Close
             github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).close
             github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).Close
             github.com/influxdata/influxdb/tsdb.(*Shard).closeNoLock
             github.com/influxdata/influxdb/tsdb.(*Shard).Close
             github.com/influxdata/influxdb/tsdb.(*Store).DeleteShard
             github.com/influxdata/influxdb/services/retention.(*Service).DeletionCheck.func3
             github.com/influxdata/influxdb/services/retention.(*Service).DeletionCheck
             github.com/influxdata/influxdb/services/retention.(*Service).run
             github.com/influxdata/influxdb/services/retention.(*Service).Open.func1
-----------+-------------------------------------------------------

Defer'ing compaction count cleanup inside goroutines should help with any hanging current compaction counts.

Modify currentCompactionN to be a sync atomic.

Adding a debug level log within Compaction.Wait() should aid in debugging.

I believe that there is something happening which causes CurrentCompactionN() to alway be greater than 0. Thus making Partition.Wait() hang forever.

davidby-influx

More ticker/timer fun.

davidby-influx · 2025-06-13T22:54:08Z

tsdb/index/tsi1/partition.go

+			}
+			p.logger.Debug("Partition.Wait() timed out waiting for compactions to complete",
+				zap.Int32("stuck_compactions", p.CurrentCompactionN()), zap.Duration("timeout", timeoutDuration),
+				zap.Strings("files", files))


Did we decide earlier that we should not exit on timeout?

If we return on timeout, then the timeout should be a timer not a ticker.

If we don't return on a timeout, then the timeout code could be in the ticker with a check if the timeoutDuration has passed since the wait was entered.

I don't think we should return on timeout. I do think we should log, I was actually thinking about changing this to a warn since it will only occur every 24 hours if a log compaction is stuck. This is mostly to ensure the issue is not happening after my go routine cleanup code is in place. I will add a comment about it.

davidby-influx

time.After changes.

davidby-influx · 2025-06-18T18:40:37Z

tsdb/index/tsi1/partition.go

 		}
-		<-ticker.C
+		select {
+		case <-time.After(timeoutDuration):


time.After creates a new channel every call. You need to move it outside the loop. See here: https://stackoverflow.com/a/39212626

This only prints the warning once., though. If you want to print the warning repeatedly (every timeoutDuration) then you should use an if in the ticker case and check elapsed time.

davidby-influx

Looking better, but one more simplification

davidby-influx · 2025-06-18T23:09:54Z

tsdb/index/tsi1/partition.go

-		<-ticker.C
+		select {
+		case <-ticker.C:
+			elapsed := time.Since(startTime)


It seems like lastWarningTime and startTime could be collapsed to a singe variable for simplicity.

lastTime := time.Now() for { if p.CurrentCompactionN() == 0 { return } select { case <-ticker.C: elapsed := time.Since(lastTime) if elapsed >= timeoutDuration { lastTime = time.Now() files := make([]string, 0) for _, v := range p.fileSet.Files() { files = append(files, v.Path())

davidby-influx

LGTM

I believe that there is something happening which causes CurrentCompactionN() to always be greater than 0. Thus making Partition.Wait() hang forever. Taking a look at some profiles where this issue occurs. I'm seeing a consistent one where we're stuck on Partition.Wait() ``` -----------+------------------------------------------------------- 1 runtime.gopark runtime.chanrecv runtime.chanrecv1 github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).Wait github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).Close github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).close github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).Close github.com/influxdata/influxdb/tsdb.(*Shard).closeNoLock github.com/influxdata/influxdb/tsdb.(*Shard).Close github.com/influxdata/influxdb/tsdb.(*Store).DeleteShard github.com/influxdata/influxdb/services/retention.(*Service).DeletionCheck.func3 github.com/influxdata/influxdb/services/retention.(*Service).DeletionCheck github.com/influxdata/influxdb/services/retention.(*Service).run github.com/influxdata/influxdb/services/retention.(*Service).Open.func1 -----------+------------------------------------------------------- ``` Defer'ing compaction count cleanup inside goroutines should help with any hanging current compaction counts. Modify currentCompactionN to be a sync atomic. Adding a debug level log within Compaction.Wait() should aid in debugging. (cherry picked from commit 149fb47)

…#26629)

…g log (#26511) (#26629)" This reverts commit 6f526cd.

I believe that there is something happening which causes CurrentCompactionN() to always be greater than 0. Thus making Partition.Wait() hang forever. Taking a look at some profiles where this issue occurs. I'm seeing a consistent one where we're stuck on Partition.Wait() ``` -----------+------------------------------------------------------- 1 runtime.gopark runtime.chanrecv runtime.chanrecv1 github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).Wait github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).Close github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).close github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).Close github.com/influxdata/influxdb/tsdb.(*Shard).closeNoLock github.com/influxdata/influxdb/tsdb.(*Shard).Close github.com/influxdata/influxdb/tsdb.(*Store).DeleteShard github.com/influxdata/influxdb/services/retention.(*Service).DeletionCheck.func3 github.com/influxdata/influxdb/services/retention.(*Service).DeletionCheck github.com/influxdata/influxdb/services/retention.(*Service).run github.com/influxdata/influxdb/services/retention.(*Service).Open.func1 -----------+------------------------------------------------------- ``` Defer'ing compaction count cleanup inside goroutines should help with any hanging current compaction counts. Modify currentCompactionN to be a sync atomic. Adding a debug level log within Compaction.Wait() should aid in debugging. (cherry picked from commit 149fb47)

…#26871)

I believe that there is something happening which causes CurrentCompactionN() to always be greater than 0. Thus making Partition.Wait() hang forever. Taking a look at some profiles where this issue occurs. I'm seeing a consistent one where we're stuck on Partition.Wait() ``` -----------+------------------------------------------------------- 1 runtime.gopark runtime.chanrecv runtime.chanrecv1 github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).Wait github.com/influxdata/influxdb/tsdb/index/tsi1.(*Partition).Close github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).close github.com/influxdata/influxdb/tsdb/index/tsi1.(*Index).Close github.com/influxdata/influxdb/tsdb.(*Shard).closeNoLock github.com/influxdata/influxdb/tsdb.(*Shard).Close github.com/influxdata/influxdb/tsdb.(*Store).DeleteShard github.com/influxdata/influxdb/services/retention.(*Service).DeletionCheck.func3 github.com/influxdata/influxdb/services/retention.(*Service).DeletionCheck github.com/influxdata/influxdb/services/retention.(*Service).run github.com/influxdata/influxdb/services/retention.(*Service).Open.func1 -----------+------------------------------------------------------- ``` Defer'ing compaction count cleanup inside goroutines should help with any hanging current compaction counts. Modify currentCompactionN to be a sync atomic. Adding a debug level log within Compaction.Wait() should aid in debugging. (cherry picked from commit 149fb47)

…#27174)

devanbenz added 5 commits June 10, 2025 17:02

feat: Add a debug level log for Parititon.Wait()

1234104

I believe that there is something happening which causes CurrentCompactionN() to alway be greater than 0. Thus making Partition.Wait() hang forever.

fix: update the timeout to be longer

e1a315e

feat: remove debugging log

fab7ee3

fix: remove p.closing check

3f17667

feat: add better logging

40ddbc4

devanbenz changed the title ~~feat: Add a debug level log for Parititon.Wait()~~ feat: Add a debug level log for Partition.Wait() Jun 10, 2025

feat: Add defer to current compaction decrements

996ff99

devanbenz changed the title ~~feat: Add a debug level log for Partition.Wait()~~ feat: Defer cleanup for log/index compactions, add debug log Jun 11, 2025

devanbenz added 2 commits June 13, 2025 12:40

feat: use atomics for currentCompactionN

9dd2db7

feat: Fixes incorrect import for atomic

de6b137

devanbenz marked this pull request as ready for review June 13, 2025 18:21

devanbenz requested a review from davidby-influx June 13, 2025 18:21

devanbenz self-assigned this Jun 13, 2025

devanbenz requested a review from gwossum June 13, 2025 18:21

devanbenz added kind/bug area/tsi 1.x labels Jun 13, 2025

davidby-influx requested changes Jun 13, 2025

View reviewed changes

feat: Use time.After()

39ec6f1

devanbenz requested a review from davidby-influx June 16, 2025 18:17

davidby-influx requested changes Jun 18, 2025

View reviewed changes

devanbenz added 2 commits June 18, 2025 17:36

feat: Use a conditional to check for timeouts

e17080f

Merge branch 'master-1.x' into db/shards_persisting

2459540

davidby-influx requested changes Jun 18, 2025

View reviewed changes

devanbenz added 2 commits June 20, 2025 10:54

feat: combine startTime and lastWarningTime

82f762e

feat: Simplify if branch

f34b1e2

davidby-influx approved these changes Jun 20, 2025

View reviewed changes

devanbenz merged commit 149fb47 into master-1.x Jun 20, 2025
9 checks passed

devanbenz deleted the db/shards_persisting branch June 20, 2025 18:18

devanbenz added a commit that referenced this pull request Jul 25, 2025

feat: Defer cleanup for log/index compactions, add debug log (#26511) (…

6f526cd

…#26629)

devanbenz added a commit that referenced this pull request Jul 31, 2025

feat: Revert "feat: Defer cleanup for log/index compactions, add debu…

c6dac96

…g log (#26511) (#26629)" This reverts commit 6f526cd.

devanbenz mentioned this pull request Aug 26, 2025

[Port to 1.12] Defer cleanup for log/index compactions, add debug log #26745

Closed

devanbenz added a commit that referenced this pull request Sep 30, 2025

feat: Defer cleanup for log/index compactions, add debug log (#26511) (…

f748c5c

…#26871)

This was referenced Dec 23, 2025

Defer cleanup for log/index compactions, add debug log [port to main-2.x] #27052

Closed

OSS 2.9.0 cherry-picks #27048

Closed

devanbenz added a commit that referenced this pull request Feb 2, 2026

feat: Defer cleanup for log/index compactions, add debug log (#26511) (…

25c2f56

…#27174)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Defer cleanup for log/index compactions, add debug log#26511

feat: Defer cleanup for log/index compactions, add debug log#26511
devanbenz merged 13 commits intomaster-1.xfrom
db/shards_persisting

devanbenz commented Jun 10, 2025 •

edited

Loading

Uh oh!

davidby-influx left a comment

Uh oh!

davidby-influx Jun 13, 2025

Uh oh!

devanbenz Jun 16, 2025

Uh oh!

davidby-influx left a comment

Uh oh!

davidby-influx Jun 18, 2025

Uh oh!

davidby-influx left a comment

Uh oh!

davidby-influx Jun 18, 2025

Uh oh!

davidby-influx left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

devanbenz commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidby-influx left a comment

Choose a reason for hiding this comment

Uh oh!

davidby-influx Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

devanbenz Jun 16, 2025

Choose a reason for hiding this comment

Uh oh!

davidby-influx left a comment

Choose a reason for hiding this comment

Uh oh!

davidby-influx Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

davidby-influx left a comment

Choose a reason for hiding this comment

Uh oh!

davidby-influx Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

davidby-influx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

devanbenz commented Jun 10, 2025 •

edited

Loading