Are you using the latest version of samtools and HTSlib? If not, please specify.

This issue is confirmed in samtools version 1.15. Don't have time to test in 1.17 because samtools index in 1.15 resolves the problem. Thus, I leave this in your capable hands to determine if it's still a bug. :-)

Please describe your environment.

The .bam file was originally indexed on CentOS Linux release 8.4.2105 using samtools merge ... --write-index. To test and resolve, I copied .bam file to local computer (MacOS Venture 13.1). After re-indexing the .bam file with samtools --index, the issue was resolved. We then re-indexed on the cluster (CentOS) using samtools --index and it worked there, too.

Please specify the steps taken to generate the issue, the command you are running and the relevant output.

We're using a custom java software called DarkRegionFinder that's built on htsjdk (originally using htsjdk version 2.24.1 but also tested 3.0.5 with the same result). Naturally, we originally assumed it was a bug in our software, or maybe even htsjdk, but it seems that it's actually a bug in samtools merge ... --write-index. The original .csi index was NOT corrupt because we tried it multiple times.

I don't know exactly what's going on, but here's what I've learned:

1. Stack trace

The stack trace pointed to an issue in htsjdk rather than something in our code

htsjdk.samtools.util.RuntimeIOException: Invalid file pointer: 2069011727511090 for /path/to/<sample>.merged.final.bam
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:857)
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:834)
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:802)
	at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.advance(BAMFileReader.java:1058)
	at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1048)
	at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1012)
	at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:591)
	at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:570)
	at htsjdk.samtools.util.PeekableIterator.advance(PeekableIterator.java:71)
	at htsjdk.samtools.util.PeekableIterator.next(PeekableIterator.java:57)
	at htsjdk.samtools.filter.FilteringSamIterator.getNextRecord(FilteringSamIterator.java:135)
	at htsjdk.samtools.filter.FilteringSamIterator.next(FilteringSamIterator.java:108)
	at htsjdk.samtools.filter.FilteringSamIterator.next(FilteringSamIterator.java:46)
	at htsjdk.samtools.util.PeekableIterator.advance(PeekableIterator.java:71)
	at htsjdk.samtools.util.PeekableIterator.next(PeekableIterator.java:57)
	at htsjdk.samtools.util.AbstractLocusIterator.next(AbstractLocusIterator.java:312)
	at htsjdk.samtools.util.AbstractLocusIterator.hasNext(AbstractLocusIterator.java:234)
	at ebbertLab.drf.DarkRegionFinder.startWalkingByLocus(DarkRegionFinder.java:170)
	at ebbertLab.drf.DarkRegionFinderEngine.findDarkRegions(DarkRegionFinderEngine.java:308)
	at ebbertLab.drf.DarkRegionFinderEngine.main(DarkRegionFinderEngine.java:43)
Caused by: java.io.IOException: Invalid file pointer: 2069011727511090 for /path/to/<sample>.merged.final.bam
	at htsjdk.samtools.util.BlockCompressedInputStream.seek(BlockCompressedInputStream.java:385)
	at htsjdk.samtools.BAMFileReader$BAMFileIndexIterator.getNextRecord(BAMFileReader.java:1001)
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:840)
	... 19 more

2. BAMFileIterator advance() returns a read that should not be returned

I followed the code to see what was "wrong" with the bam. I found that BAMFileIterator advance() function was returning a read that should not have been returned based on the Interval I provided to SamLocusIterator in DarkRegionFinder (interval: chr4:28500001-29000001). The read that should not have been returned aligned to chr4:28358154-28474692 (i.e., its alignment ends before the beginning of the designated interval).

3. Invalid file pointer thrown by BlockCompressedInputStream.class

The Invalid file pointer is eventually thrown by the htsjdk BlockCompressedInputStream.class when uncompressedOffset == available but it's not at the end of file.

4. samtools index resolves the problem

To this point, I still assumed that the bug was in htsjdk until I came across a thread describing issues with samtools merge ... --write-index, so I tried re-indexing with samtools index and that fixed the problem.

5. Conclusion

I don't know exactly what the problem is, but since samtools index resolves the problem, we're using that. We wanted to report the bug to be good citizens.

Best wishes.