ModalConnection closed when running on modal through TURN server #1858
Conversation
| "LOG_LEVEL": LOG_LEVEL, | ||
| "ONNXRUNTIME_EXECUTION_PROVIDERS": "[CUDAExecutionProvider,CPUExecutionProvider]", | ||
| "PROJECT": PROJECT, | ||
| "PYTHONASYNCIODEBUG": str(os.getenv("PYTHONASYNCIODEBUG", "0")), |
There was a problem hiding this comment.
adds the possibility to enable asyncio debugging
| ) | ||
| cls_with_options = cls_with_options.with_options( | ||
| ram=requested_ram_mb, | ||
| memory=requested_ram_mb, |
There was a problem hiding this comment.
discovered the right parameter is memory
https://modal.com/docs/reference/modal.Cls#with_options
There was a problem hiding this comment.
Interesting! Modal was definitely allocating requested amount of RAM - maybe something changed in their docs
| return frame | ||
| except StopIteration: | ||
| loop = asyncio.get_running_loop() | ||
| frame = await loop.run_in_executor(None, lambda: next(self._iterator, None)) |
There was a problem hiding this comment.
efforts to reduce main loop cpu pressure.
| heartbeat_callback() | ||
| await asyncio.sleep(WEBRTC_DATA_CHANNEL_BUFFER_DRAINING_DELAY) | ||
|
|
||
| async def wait_for_buffer_drain() -> None: |
There was a problem hiding this comment.
efforts to reduce main loop cpu pressure.
| self.realtime_processing = realtime_processing | ||
|
|
||
| # Optional receiver-paced flow control (enabled only after first ACK is received) | ||
| self._ack_last: int = 0 |
There was a problem hiding this comment.
Properties related to the ack window behaviour.
| workflow_output: Dict[str, Any], | ||
| data_output_mode: DataOutputMode, | ||
| ) -> Tuple[Dict[str, Any], List[str]]: | ||
| """Serialize workflow outputs in a thread to avoid blocking the event loop.""" |
There was a problem hiding this comment.
efforts to reduce main loop cpu pressure.
This one was that improved the asyncio warning the most.
I believe because of the base64 jpeg conversion
| ) | ||
| cls_with_options = cls_with_options.with_options( | ||
| ram=requested_ram_mb, | ||
| memory=requested_ram_mb, |
There was a problem hiding this comment.
Interesting! Modal was definitely allocating requested amount of RAM - maybe something changed in their docs
Description
Connection closes unexpectedly when running on modal through TURN server bug fix.
Added an ACK mechanism, where the client is responsible to sending the last frame it received.
This should limit the number of frames/data that are in transit on the TURN server.
The belief is that when sending frames through the datachannel, we hit some limits on the TURN server, for example the inference data sending is probably faster than the client can download from it.
With the ACK, we know an information from the client on really where it is reading currently.
Details:
Type of change
How has this change been tested, please provide a testcase or example of how you tested the change?
locally/modal
Any specific deployment considerations
modal
Docs