Are you using the latest version of samtools and HTSlib? If not, please specify.
(run samtools --version)
samtools 1.20
Using htslib 1.20
Copyright (C) 2024 Genome Research Ltd.
Samtools compilation details:
Features: build=Makefile curses=yes
CC: gcc
CPPFLAGS:
CFLAGS: -g -Wall -O2
LDFLAGS:
HTSDIR: htslib-1.20
LIBS:
CURSES_LIB: -lcurses
HTSlib compilation details:
Features: build=Makefile libcurl=yes S3=no GCS=no libdeflate=no lzma=yes bzip2=yes plugins=no htscodecs=1.6.0
CC: gcc
CPPFLAGS:
CFLAGS: -g -Wall -O2 -fvisibility=hidden
LDFLAGS: -fvisibility=hidden
HTSlib URL scheme handlers present:
built-in: preload, data, file
libcurl: imaps, pop3, gophers, http, smb, gopher, sftp, ftps, imap, smtp, smtps, rtsp, scp, ftp, telnet, mqtt, rtmp, ldap, https, ldaps, smbs, tftp, pop3s, dict
crypt4gh-needed: crypt4gh
mem: mem
Please describe your environment.
- OS (run
uname -sr on Linux/Mac OS or wmic os get Caption, Version on Windows)
$ uname -sr
Linux 5.10.199-190.747.amzn2.x86_64
- machine architecture (run
uname -m on Linux/Mac OS or wmic os get OSArchitecture on Windows)
- compiler (run
gcc --version or clang --version)
$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Please specify the steps taken to generate the issue, the command you are running and the relevant output.
This needs to be run on a filesystem that defers reporting of "no space" errors until a call to fdatasync or close (after an apparently successful write). An example of this is a weka filesystem with quotas and write caching enabled.
When quota is exhausted while samtools sort is writing temporary files (but it is reported from the filesystem during the sync or close call rather than during the write, which can happen when write caching is enabled), an error such as this one is reported:
$ ./samtools-1.20/samtools sort -@1 -o output.sorted.bam input.bam
samtools sort: error closing output file
$ ./samtools-1.20/samtools sort -@2 -o output.sorted.bam input.bam
samtools sort: error closing output file
It would be much better and much more understandable to the user if samtools sort included the reason for the failure (in this case ENOSPC (No space left on device)
An strace indicates that the ENOSPC error is being returned by the close call (in htslib):
$ strace ./samtools-1.20/samtools sort -@1 -o output.sorted.bam input.bam 2>&1 | grep -A 5 -B 5 ENOSPC
write(4, "\264ok\304\232W/\237M\217NX\313tv\311\266 w\2043l\220\376\341\25\f\356z\263\3627"..., 2097152) = 2097152
write(4, "\344\236\210\327(r\324e)/\366\301\272\255\4\5AH\266\356\264xd\267\313K\331\321\16-\333\345"..., 2097152) = 2097152
write(4, "{\342\201[\207\257z\351oUz\350\356\363o\35\376\217\332\36zp~<>\345M\201\303\343q}"..., 2097152) = 2097152
write(4, "\v\251\36b\323\273\264\342XO%\307b\253\205gn\213\351\212\326\320\342o\322L@8H\346Y;"..., 2097152) = 2097152
write(4, "\6\24\3033=\300\313\323q\367\205|p\361\17\243!\240*\37\0t\255\36\215\200\200i\0026(\213"..., 1926451) = 1926451
close(4) = -1 ENOSPC (No space left on device)
write(2, "samtools sort: ", 15samtools sort: ) = 15
write(2, "error closing output file", 25error closing output file) = 25
write(2, "\n", 1
) = 1
unlink("output.sorted.bam.tmp.0008.bam") = 0
Note that earlier versions of samtools (we originally found it with v1.17) encountered the error during an fdatasync call and reported a different incorrect error when using more than one thread (No such file or directory rather than error closing output file). The behaviour of 1.20 is slightly better, as "No such file or directory" was very confusingly wrong, whereas "error closing output file" is at least correct, but arguably incomplete since samtools is in a position to know that the issue is actually because an of the filesystem being out of space.
Are you using the latest version of samtools and HTSlib? If not, please specify.
(run
samtools --version)Please describe your environment.
uname -sron Linux/Mac OS orwmic os get Caption, Versionon Windows)uname -mon Linux/Mac OS orwmic os get OSArchitectureon Windows)gcc --versionorclang --version)Please specify the steps taken to generate the issue, the command you are running and the relevant output.
This needs to be run on a filesystem that defers reporting of "no space" errors until a call to fdatasync or close (after an apparently successful write). An example of this is a weka filesystem with quotas and write caching enabled.
When quota is exhausted while samtools sort is writing temporary files (but it is reported from the filesystem during the sync or close call rather than during the write, which can happen when write caching is enabled), an error such as this one is reported:
It would be much better and much more understandable to the user if
samtools sortincluded the reason for the failure (in this caseENOSPC(No space left on device)An strace indicates that the
ENOSPCerror is being returned by theclosecall (in htslib):Note that earlier versions of samtools (we originally found it with v1.17) encountered the error during an
fdatasynccall and reported a different incorrect error when using more than one thread (No such file or directoryrather thanerror closing output file). The behaviour of 1.20 is slightly better, as "No such file or directory" was very confusingly wrong, whereas "error closing output file" is at least correct, but arguably incomplete since samtools is in a position to know that the issue is actually because an of the filesystem being out of space.