Skip to content

Containers with memory limit < 6M fail with obscure error message #41167

New issue
New issue
Closed
Closed
Containers with memory limit < 6M fail with obscure error message#41167
Labels
area/runtimeRuntimekind/enhancementEnhancements are not bugs or new features but can improve usability or performance.
@thaJeztah

Description

@thaJeztah
thaJeztah
opened on Jul 1, 2020

TL;DR

  • Likely related to the security fix in runc, containers now require a bigger memory-footprint to start (see [CVE-2019-5736]: Runc uses more memory during start up after the fix opencontainers/runc#1980).
  • Trying to start a container with less than 6m of memory will fail, and produce a confusing error message.
  • Containers that fail to start because of this limit, will not be marked as OOMKilled, which makes it harder for users to find the cause of the failure
    • can we detect this situation? is it because the shim was killed?
  • Docker defines a minimum memory-limit of 4m, which no longer is adequate

We should probably look into the memory-consumption needed, and try to reduce this overhead. Possibly alternative approaches could help (see "possible enhancements for consideration" at the end).

Test environment

Steps were executed on a DigitalOcean machine (1 CPU, 2GB memory). Details below

Note: this was running on a machine without swap, so each of the below was printing

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.

I removed those warnings for readability. I later re-configured the machine to enable swap (described further down)

docker version and docker info 
Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:45:36 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:44:07 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Client:
 Debug Mode: false

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 10
 Server Version: 19.03.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-66-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.947GiB
 Name: ubuntu-s-1vcpu-2gb-ams3-01
 ID: 7VHE:PVY6:PXSQ:APMB:RC3E:ITUA:RWAJ:6EZK:QLLC:PHT6:DIIJ:5A6K
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Problem description

For some time, we defined a minimum limit for --memory limits (mostly based on some "trial and error" to see what would be a reasonable minimum). Currently that limit is 4MB

docker run --rm --memory=3m alpine echo success
docker: Error response from daemon: Minimum memory limit allowed is 4MB.
See 'docker run --help'.

While looking at docker/for-linux#1054 on a Ubuntu 18.04 machine (kernel 4.15.0-66-generic), I noticed that this limit is too low, and that low values now fail with a confusing error message (possibly related to security fixes in runc now causing more memory to be consumed during creation/starting the container).

Trying to find what the minimum was, here's trying to set 4m, 5m, 6m limits:

run --rm --memory=4m alpine echo success
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"4194304\\\\\\\" to \\\\\\\"/sys/fs/cgroup/memory/docker/1254c8d63f85442e599b17dff895f4543c897755ee3bd9b56d5d3d17724b38d7/memory.limit_in_bytes\\\\\\\": write /sys/fs/cgroup/memory/docker/1254c8d63f85442e599b17dff895f4543c897755ee3bd9b56d5d3d17724b38d7/memory.limit_in_bytes: device or resource busy\\\"\"": unknown.
ERRO[0000] error waiting for container: context canceled

docker run --rm --memory=5m alpine
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"5242880\\\\\\\" to \\\\\\\"/sys/fs/cgroup/memory/docker/723b06ffb440d3b3629ba80cc160ee1aacb8aaa6e583f4b2289d4f649404c0e9/memory.limit_in_bytes\\\\\\\": write /sys/fs/cgroup/memory/docker/723b06ffb440d3b3629ba80cc160ee1aacb8aaa6e583f4b2289d4f649404c0e9/memory.limit_in_bytes: device or resource busy\\\"\"": unknown.
ERRO[0000] error waiting for container: context canceled

docker run --rm --memory=6m alpine echo success
success

Cleaned up error message:

setting cgroup config for procHooks process caused
failed to write "4194304" to "/sys/fs/cgroup/memory/docker/1254c8d63f85442e599b17dff895f4543c897755ee3bd9b56d5d3d17724b38d7/memory.limit_in_bytes":
write /sys/fs/cgroup/memory/docker/1254c8d63f85442e599b17dff895f4543c897755ee3bd9b56d5d3d17724b38d7/memory.limit_in_bytes: device or resource busy "": unknown.

Next, finding a more "exact" limit (is it 6m exactly?), some interesting bits:

5.1m did not print the cgroup-failure, but looks like the process did not come up; notice that no container-ID is printed (which could be a bug in our code?)

docker run --rm --memory=5.1m alpine echo success
docker: Error response from daemon: cannot start a stopped process: unknown.

Trying again, without --rm, so that I can inspect the container's state:

docker run --memory=5.1m alpine echo success

docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
96cf0c635fd2        alpine              "echo success"      3 seconds ago       Created                                 objective_bohr

docker container inspect --format='{{json .State}}' objective_bohr | jq .

{
  "Status": "created",
  "Running": false,
  "Paused": false,
  "Restarting": false,
  "OOMKilled": false,
  "Dead": false,
  "Pid": 0,
  "ExitCode": 128,
  "Error": "cannot start a stopped process: unknown",
  "StartedAt": "0001-01-01T00:00:00Z",
  "FinishedAt": "0001-01-01T00:00:00Z"
}

So, the container was not marked as OOMKilled

In the 5.1m ... 5.8m range, I also often get a context canceled

docker run --rm --memory=5.1m alpine echo success
docker: Error response from daemon: cannot start a stopped process: unknown.
ERRO[0000] error waiting for container: context canceled

And the corresponding logs (/var/log/syslog). Relevant information;

Task in /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e killed as a result of limit of /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e
memory: usage 5220kB, limit 5220kB, failcnt 26
memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
kmem: usage 440kB, limit 9007199254740988kB, failcnt 0
Memory cgroup stats for /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e: cache:0KB rss:4780KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:4768KB inactive_file:0KB active_file:0KB unevictable:0KB
[ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[15121]     0 15121   103754     2473   143360        0             0 runc:[2:INIT]
Memory cgroup out of memory: Kill process 15121 (runc:[2:INIT]) score 1864 or sacrifice child
Killed process 15121 (runc:[2:INIT]) total-vm:415016kB, anon-rss:4720kB, file-rss:5172kB, shmem-rss:0kB
oom_reaper: reaped process 15121 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Details 
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 dockerd[7455]: time="2020-07-01T08:12:09.867865476Z" level=warning msg="Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap."
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.323768] docker0: port 1(veth6de25f3) entered blocking state
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.323771] docker0: port 1(veth6de25f3) entered disabled state
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.324087] device veth6de25f3 entered promiscuous mode
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 networkd-dispatcher[854]: WARNING:Unknown index 115 seen, reloading interface list
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15081]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15081]: Could not generate persistent MAC address for veth551ed8f: No such file or directory
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15084]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.333033] IPv6: ADDRCONF(NETDEV_UP): veth6de25f3: link is not ready
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15084]: Could not generate persistent MAC address for veth6de25f3: No such file or directory
Jul  1 08:12:09 ubuntu-s-1vcpu-2gb-ams3-01 containerd[7309]: time="2020-07-01T08:12:09.981585296Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e/shim.sock" debug=false pid=15095
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-networkd[3946]: veth6de25f3: Gained carrier
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-networkd[3946]: docker0: Gained carrier
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.650910] eth0: renamed from veth551ed8f
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.651164] IPv6: ADDRCONF(NETDEV_CHANGE): veth6de25f3: link becomes ready
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.651188] docker0: port 1(veth6de25f3) entered blocking state
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.651190] docker0: port 1(veth6de25f3) entered forwarding state
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663278] runc:[2:INIT] invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663279] runc:[2:INIT] cpuset=4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e mems_allowed=0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663285] CPU: 0 PID: 15121 Comm: runc:[2:INIT] Not tainted 4.15.0-66-generic #75-Ubuntu
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663286] Hardware name: DigitalOcean Droplet, BIOS 20171212 12/12/2017
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663286] Call Trace:
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663296]  dump_stack+0x63/0x8e
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663299]  dump_header+0x71/0x285
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663301]  oom_kill_process+0x21f/0x420
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663302]  out_of_memory+0x2b6/0x4d0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663306]  mem_cgroup_out_of_memory+0xbb/0xd0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663307]  mem_cgroup_oom_synchronize+0x2e8/0x320
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663309]  ? mem_cgroup_css_online+0x40/0x40
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663310]  pagefault_out_of_memory+0x36/0x7b
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663314]  mm_fault_error+0x90/0x180
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663315]  __do_page_fault+0x46b/0x4b0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663317]  ? __schedule+0x256/0x880
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663319]  do_page_fault+0x2e/0xe0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663321]  ? async_page_fault+0x2f/0x50
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663324]  do_async_page_fault+0x51/0x80
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663325]  async_page_fault+0x45/0x50
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663327] RIP: 0033:0x7f77a63f30dd
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663328] RSP: 002b:00007fff84b6ba40 EFLAGS: 00010202
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663329] RAX: 000000000000ef61 RBX: 00000000000000f1 RCX: 000055e3eeff8fb0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663330] RDX: 00000000000000f1 RSI: 0000000000000000 RDI: 000055e3eeff90a0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663330] RBP: ffffffffffffffa8 R08: 000055e3eeff59e0 R09: 0000000000000000
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663331] R10: 000055e3eefe7010 R11: 0000000000000000 R12: 000000000000000d
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663331] R13: 00007f77a6749ca0 R14: 00007f77a6749c40 R15: 0000000000000000
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663332] Task in /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e killed as a result of limit of /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663337] memory: usage 5220kB, limit 5220kB, failcnt 26
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663337] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663338] kmem: usage 440kB, limit 9007199254740988kB, failcnt 0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663338] Memory cgroup stats for /docker/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e: cache:0KB rss:4780KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:4768KB inactive_file:0KB active_file:0KB unevictable:0KB
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663362] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663408] [15121]     0 15121   103754     2473   143360        0             0 runc:[2:INIT]
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.663409] Memory cgroup out of memory: Kill process 15121 (runc:[2:INIT]) score 1864 or sacrifice child
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.666109] Killed process 15121 (runc:[2:INIT]) total-vm:415016kB, anon-rss:4720kB, file-rss:5172kB, shmem-rss:0kB
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.671888] oom_reaper: reaped process 15121 (runc:[2:INIT]), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 containerd[7309]: time="2020-07-01T08:12:10.316214199Z" level=info msg="shim reaped" id=4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 dockerd[7455]: time="2020-07-01T08:12:10.328018040Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-networkd[3946]: veth6de25f3: Lost carrier
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.742548] docker0: port 1(veth6de25f3) entered disabled state
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.745452] veth551ed8f: renamed from eth0
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.757493] docker0: port 1(veth6de25f3) entered disabled state
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 networkd-dispatcher[854]: WARNING:Unknown index 114 seen, reloading interface list
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.762798] device veth6de25f3 left promiscuous mode
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 kernel: [49223.762808] docker0: port 1(veth6de25f3) entered disabled state
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 networkd-dispatcher[854]: ERROR:Unknown interface index 114 seen even after reload
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 dockerd[7455]: time="2020-07-01T08:12:10.402854115Z" level=error msg="4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e cleanup: failed to delete container from containerd: no such container"
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 dockerd[7455]: time="2020-07-01T08:12:10.412135381Z" level=error msg="Handler for POST /v1.40/containers/4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e/start returned error: cannot start a stopped process: unknown"
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 dockerd[7455]: time="2020-07-01T08:12:10.426702914Z" level=error msg="failed to process event" container=4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e error="could not find container 4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e: No such container: 4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e" event=exit event-info="{4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e 4d42d19b32596a652c03346bbe2d437d09b7b6cf5218bb31329b70938eaca88e 15121 137 2020-07-01 08:12:10.285461759 +0000 UTC false <nil>}" module=libcontainerd namespace=moby
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15167]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15167]: link_config: could not get ethtool features for veth551ed8f
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-udevd[15167]: Could not set offload features of veth551ed8f: No such device
Jul  1 08:12:10 ubuntu-s-1vcpu-2gb-ams3-01 systemd-networkd[3946]: docker0: Lost carrier

Trying with --network=host (skipping creation of a network namespace), didn't help; memory limit still looks to be too low:

docker run --rm --network=host --memory=5.1m alpine echo success
docker: Error response from daemon: cannot start a stopped process: unknown.
ERRO[0000] error waiting for container: context canceled

For 5.9m, I got an interesting one. Unfortunately I wasn't able to reproduce it after, and didn't capture logs (weren't able to find them back);

docker run --rm --memory=5.9m alpine echo success
standard_init_linux.go:211: exec user process caused "argument list too long"

(whaaa?)

With 5.9m or up, things went mostly ok;

docker run --rm --memory=5.9m alpine echo success
success

updating memory limit after starting

So, the problem looks to be the memory consumption when starting the container.

docker run -dit --name mycontainer --memory=8m alpine
acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86

Once the container is started, it's consuming little memory:

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
acdd326419f0        mycontainer         0.00%               460KiB / 8MiB       89.84%              1.12kB / 0B         0B / 0B             1

And we can reduce the memory-limit to go as low as 512k:

echo 4194304 > /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
echo 2097152 > /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
echo 1048576 > /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
echo 524288  > /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes

At this point, the container is still running fine (within limits);

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
acdd326419f0        mycontainer         0.00%               460KiB / 512KiB     89.84%              1.12kB / 0B         0B / 0B             1

Going to 256k, an error occurred, and the memory limit cannot be updated;

echo 262144 > /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
-bash: echo: write error: Device or resource busy

cat /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes
524288

So, looks like the kernel refuses to set a limit < amount of actual memory used in the cgroup

Trying if stopping and re-starting a container with the low limit works (or if memory use is needed to start the container);

docker kill mycontainer
mycontainer

docker start mycontainer
mycontainer

That didn't work, because the cgroup is reset to the container's config when starting again;

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
acdd326419f0        mycontainer         0.00%               468KiB / 8MiB       5.71%               906B / 0B           0B / 0B             1

Trying to update the container's limit to 4M before killing;

docker container update --memory=4m mycontainer
mycontainer

docker kill mycontainer
mycontainer

docker start mycontainer
Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"4194304\\\\\\\" to \\\\\\\"/sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes\\\\\\\": write /sys/fs/cgroup/memory/docker/acdd326419f0898be63b0463cfc81cd17fb34d2dae6f8aa3768ee6a075ca5c86/memory.limit_in_bytes: device or resource busy\\\"\"": unknown
Error: failed to start containers: mycontainer

So, looks like starting the container requires more than 4M, so this is not a solution.

Does swap help?

No, enabling swap does not help. I tried enabling swap limit support:

cat /etc/default/grub |grep GRUB_CMDLINE_LINUX=
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

After enabling swap and rebooting the machine, the problem is the same;

docker run --rm --memory=4m alpine echo success
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"4194304\\\\\\\" to \\\\\\\"/sys/fs/cgroup/memory/docker/10ef86ea7cd32cd775f0e6b1a9fd882a0b0d4191f1817b1375d478ce2cbac90d/memory.limit_in_bytes\\\\\\\": write /sys/fs/cgroup/memory/docker/10ef86ea7cd32cd775f0e6b1a9fd882a0b0d4191f1817b1375d478ce2cbac90d/memory.limit_in_bytes: device or resource busy\\\"\"": unknown.

docker run --rm --memory=4m --memory-swap=16m alpine echo success
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:415: setting cgroup config for procHooks process caused \\\"failed to write \\\\\\\"4194304\\\\\\\" to \\\\\\\"/sys/fs/cgroup/memory/docker/6772c052387d1a4a6321c97cc2a4d1f2e5a3908e89510e433eae3aecc36b7438/memory.limit_in_bytes\\\\\\\": write /sys/fs/cgroup/memory/docker/6772c052387d1a4a6321c97cc2a4d1f2e5a3908e89510e433eae3aecc36b7438/memory.limit_in_bytes: device or resource busy\\\"\"": unknown.

possible enhancements for consideration

Wondering if it would be useful to allow lower memory limits, and to achieve that, we could;

  1. create and start container (with a minimum limit of (e.g.) memory-limit as defined by user + 6m (accounting for the overhead of starting the container))
  2. after container is started update the limit to the specified value (e.g. 512k)

Not sure if worth looking into (and if it would be racy)

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/runtimeRuntimekind/enhancementEnhancements are not bugs or new features but can improve usability or performance.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions