test_memory_efficient fails during startup. It fails to connect to the cluster. We may need to look into Coiled to understand what's the cause. @ntabris is this something you can help with?

@fjetter not sure what you mean by "fails at startup", since I can see that cluster started and ran code. From what I can see, this also looks like a dask issue, but I'd be happy to get some help understanding this better.

Here's some information from this exact failure (not a re-run, this is from the linked CI run).

https://cloud.coiled.io/dask-engineering/clusters/93429/details

http://35.86.202.18:3000/d/eU1bT-nVz/cluster-metrics-prometheus?orgId=1&from=1665448336084&to=1665449682572&var-datasource=AWS+Prometheus+-+Sandbox+%28us+east+2%29&var-env=All&var-account=dask-engineering&var-cluster=test_futures-d09a04c5&var-cluster_id=93429

Scheduler logs show thousands of lines of

Task inc-[...] marked as failed because 3 workers died while trying to run it

and then python on scheduler gets OOMKilled:

Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816716] python invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816724] CPU: 1 PID: 2109 Comm: python Not tainted 5.15.0-1020-aws #24~20.04.1-Ubuntu
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816727] Hardware name: Amazon EC2 t3.xlarge/, BIOS 1.0 10/16/2017
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816728] Call Trace:
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816730]  <TASK>
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816732]  dump_stack_lvl+0x4a/0x63
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816739]  dump_stack+0x10/0x16
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816741]  dump_header+0x53/0x225
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816746]  oom_kill_process.cold+0xb/0x10
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816748]  out_of_memory+0x1dc/0x530
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816752]  mem_cgroup_out_of_memory+0xf7/0x110
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816756]  try_charge_memcg+0x690/0x740
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816759]  ? get_page_from_freelist+0x2f8/0x450
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816763]  charge_memcg+0x45/0xb0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816765]  __mem_cgroup_charge+0x2d/0x90
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816768]  __add_to_page_cache_locked+0x2e1/0x360
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816772]  ? scan_shadow_nodes+0x40/0x40
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816776]  add_to_page_cache_lru+0x4d/0xd0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816778]  pagecache_get_page+0x18f/0x530
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816781]  ? page_cache_ra_unbounded+0x16a/0x220
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816784]  filemap_fault+0x527/0xb60
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816786]  ? filemap_map_pages+0x138/0x640
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816789]  __do_fault+0x40/0x120
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816793]  do_fault+0x1f9/0x420
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816795]  __handle_mm_fault+0x63d/0x850
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816798]  handle_mm_fault+0xd8/0x2c0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816801]  do_user_addr_fault+0x1c2/0x660
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816805]  exc_page_fault+0x77/0x170
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816810]  asm_exc_page_fault+0x26/0x30
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816814] RIP: 0033:0x7fc6a1506ad8
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816820] Code: Unable to access opcode bytes at RIP 0x7fc6a1506aae.
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816821] RSP: 002b:00007fc6771c24f0 EFLAGS: 00010246
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816823] RAX: ffffffffffffff92 RBX: 00007fc6771c2550 RCX: 00007fc6a1506ad8
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816824] RDX: 0000000000000000 RSI: 0000000000000089 RDI: 00005589926a3d88
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816825] RBP: 00005589926a3d60 R08: 0000000000000000 R09: 00000000ffffffff
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816826] R10: 00007fc6771c2600 R11: 0000000000000246 R12: 0000000000000000
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816828] R13: 00005589926a3d90 R14: 00007fc6771c2600 R15: 00005589926a3d88
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816830]  </TASK>
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816830] memory: usage 15663768kB, limit 15663768kB, failcnt 3812
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816832] memory+swap: usage 15663768kB, limit 31327536kB, failcnt 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816833] kmem: usage 34104kB, limit 9007199254740988kB, failcnt 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816835] Memory cgroup stats for /docker/a7f7645ecc5375b249859a49b34e8f90d8708fa43f7762b15b0b556319489677:
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] anon 16004775936
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] file 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] kernel_stack 294912
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] pagetables 31891456
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] percpu 144
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] sock 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] shmem 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] file_mapped 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] file_dirty 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] file_writeback 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] swapcached 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] anon_thp 44040192
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] file_thp 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] shmem_thp 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] inactive_anon 16004739072
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] active_anon 32768
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] inactive_file 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] active_file 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] unevictable 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] slab_reclaimable 1834552
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] slab_unreclaimable 813968
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] slab 2648520
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] workingset_refault_anon 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] workingset_refault_file 21419
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] workingset_activate_anon 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] workingset_activate_file 2345
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] workingset_restore_anon 0
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816847] workingset_restore_file 1859
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816850] Tasks state (memory values in pages):
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816851] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816852] [   1122]  1000  1122    25933      511    86016        0             0 micromamba
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816855] [   1135]  1000  1135     1433       84    49152        0             0 entry.sh
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816857] [   2097]  1000  2097    25941      519    90112        0             0 micromamba
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816860] [   2099]  1000  2099  4255380  3906089 31703040        0             0 python
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816861] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=a7f7645ecc5375b249859a49b34e8f90d8708fa43f7762b15b0b556319489677,mems_allowed=0,oom_memcg=/docker/a7f7645ecc5375b249859a49b34e8f90d8708fa43f7762b15b0b556319489677,task_memcg=/docker/a7f7645ecc5375b249859a49b34e8f90d8708fa43f7762b15b0b556319489677,task=python,pid=2099,uid=1000
Oct 11 00:51:10 ip-10-0-9-207 kernel: [  972.816910] Memory cgroup out of memory: Killed process 2099 (python) total-vm:17021520kB, anon-rss:15624356kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:30960kB oom_score_adj:0

And here's memory util on the scheduler, copied from Grafana dashboard:

⚠️ CI failed ⚠️ #441

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions