Skip to content

Optimization panic status cleanup#7783

Merged
xzfc merged 1 commit intodevfrom
optimization-panic
Dec 16, 2025
Merged

Optimization panic status cleanup#7783
xzfc merged 1 commit intodevfrom
optimization-panic

Conversation

@xzfc
Copy link
Member

@xzfc xzfc commented Dec 15, 2025

Depends on #7766 because it touches the same part of code.

Problem

When an optimization task panics, its telemetry reports are stuck as never-ending "status": "optimizing":

curl '127.0.0.1:6333/telemetry?details_level=10' \
| jq '.result.collections.collections[].shards[].local.optimizations'
{
  "status": {
    "error": "Service internal error: Optimization task panicked: Simulated panic for testing purposes"
  },
  "optimizations": {
    "count": 0,
    "fail_count": 2,
    "total_duration_micros": 0,
    "last_responded": "2025-12-15T22:16:29.440Z"
  },
  "log": [
    {
      "name": "indexing",
      "segment_ids": [
        1,
        2
      ],
      "status": "optimizing", // ⚠️ should be "error"
      "start_at": "2025-12-15T22:16:29.422213214Z",
      "end_at": null          // ⚠️ should be non-null
    },
    {
      "name": "indexing",
      "segment_ids": [
        3,
        0
      ],
      "status": "optimizing", // ⚠️
      "start_at": "2025-12-15T22:16:28.987590070Z",
      "end_at": null          // ⚠️
    }
  ]
}

Additionally, the optimizing panic handler is not reliable. Its StoppableTaskHandle::panic_handler relies on the caller to call join_and_handle_panic explicitly. (what if the supposed caller will be cancelled/skipped due to errors/early return).

And it's awkward to use: in order to "jump" into it, the stoppable task calls panic!(). But how to pass a variable to the panic handler other than string (e.g. TrackerStatus)?

Solution

Remove panic handling from StoppableTaskHandle, instead use std::panic::catch_unwind. Handle all cases under a single match so the compiler won't let us to forget to provide a value for tracker_handle.update() or report_optimizer_error().

Also, reliable completion handling would be useful for tracking currently optimized segments.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants