The cardinality aggregation currently reserves a fixed amount of memory per bucket. While this makes memory management easier, this also has the downside of making buckets that don't collect many docs very memory-intensive.