Skip to content

samtools sort incorrectly reports ENOSPC error from fdatasync while writing temporary outputs as "error closing output file" #2049

Description

@jrandall

Are you using the latest version of samtools and HTSlib? If not, please specify.

(run samtools --version)

samtools 1.20
Using htslib 1.20
Copyright (C) 2024 Genome Research Ltd.

Samtools compilation details:
    Features:       build=Makefile curses=yes
    CC:             gcc
    CPPFLAGS:
    CFLAGS:         -g -Wall -O2
    LDFLAGS:
    HTSDIR:         htslib-1.20
    LIBS:
    CURSES_LIB:     -lcurses

HTSlib compilation details:
    Features:       build=Makefile libcurl=yes S3=no GCS=no libdeflate=no lzma=yes bzip2=yes plugins=no htscodecs=1.6.0
    CC:             gcc
    CPPFLAGS:
    CFLAGS:         -g -Wall -O2 -fvisibility=hidden
    LDFLAGS:        -fvisibility=hidden

HTSlib URL scheme handlers present:
    built-in:	 preload, data, file
    libcurl:	 imaps, pop3, gophers, http, smb, gopher, sftp, ftps, imap, smtp, smtps, rtsp, scp, ftp, telnet, mqtt, rtmp, ldap, https, ldaps, smbs, tftp, pop3s, dict
    crypt4gh-needed:	 crypt4gh
    mem:	 mem

Please describe your environment.

  • OS (run uname -sr on Linux/Mac OS or wmic os get Caption, Version on Windows)
$ uname -sr
Linux 5.10.199-190.747.amzn2.x86_64
  • machine architecture (run uname -m on Linux/Mac OS or wmic os get OSArchitecture on Windows)
$ uname -m
x86_64
  • compiler (run gcc --version or clang --version)
$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Please specify the steps taken to generate the issue, the command you are running and the relevant output.

This needs to be run on a filesystem that defers reporting of "no space" errors until a call to fdatasync or close (after an apparently successful write). An example of this is a weka filesystem with quotas and write caching enabled.

When quota is exhausted while samtools sort is writing temporary files (but it is reported from the filesystem during the sync or close call rather than during the write, which can happen when write caching is enabled), an error such as this one is reported:

$ ./samtools-1.20/samtools sort -@1 -o output.sorted.bam input.bam
samtools sort: error closing output file
$ ./samtools-1.20/samtools sort -@2 -o output.sorted.bam input.bam
samtools sort: error closing output file

It would be much better and much more understandable to the user if samtools sort included the reason for the failure (in this case ENOSPC (No space left on device)

An strace indicates that the ENOSPC error is being returned by the close call (in htslib):

$ strace ./samtools-1.20/samtools sort -@1 -o output.sorted.bam input.bam 2>&1 | grep -A 5 -B 5 ENOSPC
write(4, "\264ok\304\232W/\237M\217NX\313tv\311\266 w\2043l\220\376\341\25\f\356z\263\3627"..., 2097152) = 2097152
write(4, "\344\236\210\327(r\324e)/\366\301\272\255\4\5AH\266\356\264xd\267\313K\331\321\16-\333\345"..., 2097152) = 2097152
write(4, "{\342\201[\207\257z\351oUz\350\356\363o\35\376\217\332\36zp~<>\345M\201\303\343q}"..., 2097152) = 2097152
write(4, "\v\251\36b\323\273\264\342XO%\307b\253\205gn\213\351\212\326\320\342o\322L@8H\346Y;"..., 2097152) = 2097152
write(4, "\6\24\3033=\300\313\323q\367\205|p\361\17\243!\240*\37\0t\255\36\215\200\200i\0026(\213"..., 1926451) = 1926451
close(4)                                = -1 ENOSPC (No space left on device)
write(2, "samtools sort: ", 15samtools sort: )         = 15
write(2, "error closing output file", 25error closing output file) = 25
write(2, "\n", 1
)                       = 1
unlink("output.sorted.bam.tmp.0008.bam") = 0

Note that earlier versions of samtools (we originally found it with v1.17) encountered the error during an fdatasync call and reported a different incorrect error when using more than one thread (No such file or directory rather than error closing output file). The behaviour of 1.20 is slightly better, as "No such file or directory" was very confusingly wrong, whereas "error closing output file" is at least correct, but arguably incomplete since samtools is in a position to know that the issue is actually because an of the filesystem being out of space.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions