logs filling up dockerd memory #1046
Description
In our prometheus setup we see a lot of
time="2015-09-01T10:51:02Z" level=warning msg="Ignoring sample with out-of-order timestamp for fingerprint 8ecd53ef091d7a0f .....: 1441104582.533 is not after 1441104582.533" file=storage.go line=564
time="2015-09-01T10:51:02Z" level=warning msg="Ignoring sample with out-of-order timestamp for fingerprint 60499958e3784aaa ....1441104417.533 is not after 1441104417.533" file=storage.go line=564
And with "a lot" I mean that dockerd fills to 32GB and then gets OOM-killed. This is partly due to docker being stupid and actually caching this output or maybe this isn't worthy of outputting?