Local scheduler crashes when job is killed. #1925
Description
System information
- OS Platform and Distribution: Ubuntu
- Ray installed from (source or binary): pip
- Ray version: 0.4.0
- Python version: Python 3.6.1 :: Continuum Analytics, Inc.
Describe the problem
I don't have a minimal example at the moment, but I can reproduce the issue by running a specific application and then hitting Ctrl-C in the middle. The local scheduler crashes with
/ray/src/common/state/redis.cc:979 Check failed: reply->type != REDIS_REPLY_ERROR reply->str is Cannot update a task that doesn't exist yet
Also
/opt/conda/lib/python3.6/site-packages/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x43a0f5]
/opt/conda/lib/python3.6/site-packages/ray/local_scheduler/../core/src/local_scheduler/local_scheduler(_ZN3ray8internal7CerrLogD1Ev+0x8e)[0x44633e]
/opt/conda/lib/python3.6/site-packages/ray/local_scheduler/../core/src/local_scheduler/local_scheduler(_Z32redis_task_table_update_callbackP17redisAsyncContextPvS1_+0x216)[0x463136]
/opt/conda/lib/python3.6/site-packages/ray/local_scheduler/../core/src/local_scheduler/local_scheduler(redisProcessCallbacks+0x83)[0x4b1833]
/opt/conda/lib/python3.6/site-packages/ray/local_scheduler/../core/src/local_scheduler/local_scheduler(aeProcessEvents+0x227)[0x477f47]
/opt/conda/lib/python3.6/site-packages/ray/local_scheduler/../core/src/local_scheduler/local_scheduler(aeMain+0x2b)[0x47826b]
/opt/conda/lib/python3.6/site-packages/ray/local_scheduler/../core/src/local_scheduler/local_scheduler(main+0x562)[0x43a9b2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd936d55830]
/opt/conda/lib/python3.6/site-packages/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x43c921]
cc @unixpickle