Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #7766 because it touches the same part of code.
Problem
When an optimization task panics, its telemetry reports are stuck as never-ending
"status": "optimizing":{ "status": { "error": "Service internal error: Optimization task panicked: Simulated panic for testing purposes" }, "optimizations": { "count": 0, "fail_count": 2, "total_duration_micros": 0, "last_responded": "2025-12-15T22:16:29.440Z" }, "log": [ { "name": "indexing", "segment_ids": [ 1, 2 ], "status": "optimizing", // ⚠️ should be "error" "start_at": "2025-12-15T22:16:29.422213214Z", "end_at": null // ⚠️ should be non-null }, { "name": "indexing", "segment_ids": [ 3, 0 ], "status": "optimizing", // ⚠️ "start_at": "2025-12-15T22:16:28.987590070Z", "end_at": null // ⚠️ } ] }Additionally, the optimizing panic handler is not reliable. Its
StoppableTaskHandle::panic_handlerrelies on the caller to calljoin_and_handle_panicexplicitly. (what if the supposed caller will be cancelled/skipped due to errors/early return).And it's awkward to use: in order to "jump" into it, the stoppable task calls
panic!(). But how to pass a variable to the panic handler other than string (e.g.TrackerStatus)?Solution
Remove panic handling from
StoppableTaskHandle, instead usestd::panic::catch_unwind. Handle all cases under a singlematchso the compiler won't let us to forget to provide a value fortracker_handle.update()orreport_optimizer_error().Also, reliable completion handling would be useful for tracking currently optimized segments.