The memory footprint of kata containers is pretty high even inside the guest. Comparing to what runv+hyperstart has for a 256MB busybox container:

/ # free -m
             total       used       free     shared    buffers     cached
Mem:           235         33        202         18          0         19
-/+ buffers/cache:         13        222
Swap:            0          0          0

kata container only leaves less than 120MB free memory for the container app:

/ # free -m
             total       used       free     shared    buffers     cached
Mem:           195         82        113         26          0         26
-/+ buffers/cache:         55        139
Swap:            0          0          0

There are two places we need to look into:

1. total memory: both cases are specifying 256 MB in qemu arguments but kata container guest only sees around 200MB of total memory, as also confirmed by agent log system-memory=\"200572 kB.
2. used memory: likely this is caused by C vs. go. IWO, the go runtime costs around 50MB more than a plain C agent. I guess it's a price we'll have to pay with having a go agent. I wonder if there is some way to still reduce the agent memory footprint?

@sboeuf @WeiZhang555 @amshinde @devimc @egernst @laijs any thought?