Hi,
I would like to be able to peek at the contents of the MpscArrayQueue or MpscUnboundedArrayQueue from a non-consumer thread.
I'm working on Apache Storm, which has the following queue setup to process a stream of data:
Inbound thread -> inputQueue -> user code -> outputQueue -> Outbound thread
I would like for a thread that is not the user code or the outbound thread to be able to peek at, but not modify, the elements of inputQueue and outputQueue. Essentially I would like a public Collection<E> unorderedSnapshot() method that just returns a copy of the contents of the queue's buffer, filtering out any nulls or SKIP instructions.
Just to justify why I need this, feel free to skip if it's not important:
Storm is a system for distributed processing of data streams. When a new element in a data stream is submitted for processing, it may go to many physical machines before it is considered fully processed.
Storm puts a timeout on how long the full processing sequence is allowed to take, in order to offer at-least-once processing guarantees in the presence of e.g. crashing machines or network issues.
I would like to be able to keep this timeout from expiring as long as any machines are still actively processing an element. One of the places an element is at risk of timing out is when it is stuck in the inputQueue or outputQueue, either due to slow user code, or due to backpressure from downstream processing steps.
To keep the timeout from expiring, I would like for a heartbeat thread to occasionally reset the timeout for messages that are stuck in the inputQueue or outputQueue. In order to do this, I need this thread to be able to read the queue contents.
Hi,
I would like to be able to peek at the contents of the MpscArrayQueue or MpscUnboundedArrayQueue from a non-consumer thread.
I'm working on Apache Storm, which has the following queue setup to process a stream of data:
Inbound thread -> inputQueue -> user code -> outputQueue -> Outbound thread
I would like for a thread that is not the user code or the outbound thread to be able to peek at, but not modify, the elements of inputQueue and outputQueue. Essentially I would like a
public Collection<E> unorderedSnapshot()method that just returns a copy of the contents of the queue's buffer, filtering out any nulls or SKIP instructions.Just to justify why I need this, feel free to skip if it's not important:
Storm is a system for distributed processing of data streams. When a new element in a data stream is submitted for processing, it may go to many physical machines before it is considered fully processed.
Storm puts a timeout on how long the full processing sequence is allowed to take, in order to offer at-least-once processing guarantees in the presence of e.g. crashing machines or network issues.
I would like to be able to keep this timeout from expiring as long as any machines are still actively processing an element. One of the places an element is at risk of timing out is when it is stuck in the inputQueue or outputQueue, either due to slow user code, or due to backpressure from downstream processing steps.
To keep the timeout from expiring, I would like for a heartbeat thread to occasionally reset the timeout for messages that are stuck in the inputQueue or outputQueue. In order to do this, I need this thread to be able to read the queue contents.