• Tendermint version: 0.31.5 (d2eab53)
• ABCI app: self-written, not currently publicly available
• OS: Ubuntu 16.04.5 LTS x86_64

What happened: The block header time starts drifting into the future. The below graph shows now - BlockHeaderTime, so if the value is negative, Block.Header.Time is ahead of real time.

What was expected to happen: The block header time should match real time, to within a couple of seconds.

Tried the latest version: no

How to reproduce it: I have yet to create a minimal ABCI app that reproduces this problem.

Logs: At the default log levels (main:info,state:info,*:error) there are no useful errors. I have changed the config to *:debug and am waiting for the next occurrence of time drift.

Config: Default config, as generated by tendermint testnet

node command runtime flags: Just tendermint node

Our tendermint cluster consists of 3 nodes, in GB, SG and US. Transactions are submitted at a rate of under 10 per second. If under heavy load, it would be reasonable to expect the block header time to lag real time (ie drift into the past), but this issue is drift in the opposite direction.