LogFile_STD (LogFileImpl) fails to recover from getting out of space #2084
Description
Expected behavior
When there is no space on disk, the LogFileImpl::writeImpl should throw exception. If some space on the disk is freed, the LogFileImpl::writeImpl should recover and write normally to the log file.
Actual behavior
When there is no space on disk, the LogFileImpl::writeImpl should throw exception. If some space on the disk is freed, the LogFileImpl::writeImpl still throws an exception and does not recover. The reason is that the stream flags are never cleared. Thus the check
if (!_str.good()) throw WriteFileException(_path);
always throws the exception and the writes do not work because the stream is not in a good state.
Steps to reproduce the problem
- create a small loop device with some reserved space, eg.
dd if=/dev/zero of=test.loop bs=1024 count=56
sudo mkfs.fat test.loop
mkdir -p mnt-test
uid=`id -u`
gid=`id -g`
sudo mount -o loop,rw,uid=$uid,gid=$gid test.loop mnt-test
echo "reserve some space in the file system for testing" > mnt-test/reserve
df -h mnt-test
- setup logger and fill it with logs until the disk is full and Logger::log throws WriteFileException
- remove the reserved space (
rm mnt-test/reserve) - continue logging with the same logger instance, log a short message
- an exception is thrown however, it should not be...
void log(Logger &logger, int i)
{
string text =
"abcdef0123456789abcdef0123456789"
"abcdef0123456789abcdef0123456789"
"abcdef0123456789abcdef0123456789";
try {
logger.critical(text + " %d", i);
}
catch (const Exception &e) {
logger.log(e); // here WriteFileException is thrown on no space
}
}
int main(int argc, char **argv)
{
unlink("mnt-test/test.log");
Logger &logger = Logger::get("");
// prepare channels to see progress in console while writing to the test.log file
SplitterChannel *c = new SplitterChannel;
ConsoleChannel *console = new ConsoleChannel;
FileChannel *file = new FileChannel("mnt-test/test.log");
c->addChannel(console);
c->addChannel(file);
logger.setChannel(c);
for (int i = 0; ; ++i) {
try {
log(logger, i);
}
catch (...) {
break; // no more space, get out
}
}
unlink("mnt-test/reserve"); // free the reserved space
system("df -h mnt-test"); // check status of the testing file system
try {
logger.critical("unlinked reserve, try again"); // should be logged properly
}
catch (const Exception &e) {
logger.log(e); // here, we fail when not recovered
}
}
POCO version
any version
Compiler and version
g++ (GCC) 7.2.1
Operating system and version
Linux 4.9.67-1-lts
Other relevant information
I think that the LogFile_Win32 does not suffer from this issue because it does not work with the C++ streams.
The issue was discovered on a running system not with the provided testing code. It is a real problem, we have lost many hours of logs.