While I was trying to run the GATK HaplotypeCaller with option --alleles input.vcf.gz on a file that I had generated with BCFtools with option --write-index=tbi, I got the error:
htsjdk.samtools.util.RuntimeIOException: java.io.IOException: Invalid file pointer: 13729595107 for input.vcf.gz
I read that this is usually caused by an out-of-date or corrupt index file. I then regenerated the index file with option bcftools index --tbi and the GATK HaplotypeCaller worked without issues. Indeed the two index files generated were different with different md5sum's
I could not replicate an error with BCFtools but I did notice that --write-index=tbi and bcftools index --tbi don't always create the same index files if applied to files large enough. This is reproducible with BCFtools 1.20 (using htslib 1.20):
(echo "##fileformat=VCFv4.2"
echo "##contig=<ID=chr1>"
echo -e "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO"
seq -f "chr1\t%.0f\t.\tA\tG\t.\t.\t." 1 6914049 | sed 's/\\t/\t/g') | \
bcftools view --no-version --output-type z --write-index=tbi --output output.vcf.gz && \
cat output.vcf.gz | bcftools index --force --tbi --output output.vcf.gz.tbi2 && \
md5sum output.vcf.gz.tbi output.vcf.gz.tbi2
284fdb2a372efb91e7e692707f5ff1c6 output.vcf.gz.tbi
4cf5f01b5210ce5193c38339161027fb output.vcf.gz.tbi2
I could not understand why they are different:
diff <(xxd output.vcf.gz.tbi) <(xxd output.vcf.gz.tbi2)
2c2
< 00000010: 4311 659a 0b78 ce75 ffc7 a7c3 50e6 b41c C.e..x.u....P...
---
> 00000010: 4211 659a 0b78 ce75 ffc7 a7c3 50e6 b41c B.e..x.u....P...
240,278c240,278
< 00000ef0: f905 bbfc c1c3 f293 c042 76b9 fb84 1f0d .........Bv.....
< 00000f00: 764f d022 df52 aef1 3dea c566 4df0 f8ef vO.".R..=..fM...
< 00000f10: 1f5b 8afd ef2e 9fdd d5b3 bb78 76d7 ceee .[.........xv...
< 00000f20: d2d9 5d39 bb0b 6777 ddec 2e9b dd55 b3bb ..]9..gw.....U..
< 00000f30: 6876 d7cc ee92 d95d 31bb 0b66 77bd ec2e hv.....]1..fw...
< 00000f40: 97dd d5b2 bb58 76d7 caee 52d9 5d29 bb0b .....Xv...R.])..
< 00000f50: 6577 9dec 2e93 dd55 b2bb 4876 d7c8 ee12 ew.....U..Hv....
< 00000f60: d95d 21bb 0b64 777d ec2e 8fdd d5b1 bb38 .]!..dw}.......8
< 00000f70: 76d7 c6ee d2fc cfae 8cd9 85b1 bb2e 7697 v.............v.
< 00000f80: c5ee aad8 5d14 bb6b 6277 49ec ae88 dd05 ....]..kbwI.....
< 00000f90: b1bb 1e76 97c3 ee6a d85d 0cbb 6b61 7729 ...v...j.]..kaw)
< 00000fa0: ecae 84dd 85b0 bb0e 7697 c1ee 2ad8 5d04 ........v...*.].
< 00000fb0: bb6b 6077 09ec ae80 dd05 b077 fdf6 2edf .k`w.......w....
< 00000fc0: ded5 dbbb 787b d76e efd2 ed5d b9bd 0bb7 ....x{.n...]....
< 00000fd0: 77dd f62e dbde 55db bb68 7bd7 6cef 92ed w.....U..h{.l...
< 00000fe0: 5db1 bd0b b677 bdf6 2ed7 ded5 dabb 587b ]....w........X{
< 00000ff0: d76a ef52 ed5d a9bd 0bb5 779d f62e d3de .j.R.]....w.....
< 00001000: 55da bb48 7bd7 68ef 12ed 5da1 bd0b b477 U..H{.h...]....w
< 00001010: 7df6 2ecf ded5 d9bb 387b d766 efd2 ec5d }.......8{.f...]
< 00001020: 99bd 0bb3 775d f62e cbde 55d9 bb28 7bd7 ....w]....U..({.
< 00001030: 64ef 92ec 5d91 bd0b b277 3df6 2ec7 ded5 d...]....w=.....
< 00001040: d8bb 187b d762 ef52 ec5d 89bd 0bb1 771d ...{.b.R.]....w.
< 00001050: f62e c3de 55d8 bb08 7bd7 60ef 12ec 5d81 ....U...{.`...].
< 00001060: bd0b b0b3 7e3b cbb7 b37a 3b8b b7b3 763b ....~;...z;...v;
< 00001070: 4bb7 b372 3b0b b7b3 6e3b cbb6 b36a 3b8b K..r;...n;...j;.
< 00001080: b6b3 663b 4bb6 b362 3b0b b6b3 5e3b cbb5 ..f;K..b;...^;..
< 00001090: b35a 3b8b b5b3 563b 4bb5 b352 3b0b b5b3 .Z;...V;K..R;...
< 000010a0: 4e3b cbb4 b34a 3b8b b4b3 463b 4bb4 b342 N;...J;...F;K..B
< 000010b0: 3b0b b4b3 3e3b cbb3 b33a 3b8b b3b3 363b ;...>;...:;...6;
< 000010c0: 4bb3 b332 3b0b b3b3 2e3b cbb2 b32a 3b8b K..2;....;...*;.
< 000010d0: b2b3 263b 4bb2 b322 3b0b b2b3 1e3b cbb1 ..&;K..";....;..
< 000010e0: b31a 3b8b b1b3 163b 4bb1 b312 3b0b b1b3 ..;....;K...;...
< 000010f0: 0e3b cbb0 b30a 3b8b b0b3 063b 4bb0 b302 .;....;....;K...
< 00001100: 3b0b b067 7d7b 96b7 6775 7b16 b767 6d7b ;..g}{..gu{..gm{
< 00001110: 96b6 6765 7b16 b667 5d7b 96b5 6755 7b16 ..ge{..g]{..gU{.
< 00001120: b567 4d7b 96b4 6745 7b16 b467 3d7b 96b3 .gM{..gE{..g={..
< 00001130: 6735 7b16 fbcf 59eb d18f ff07 1f98 b6a6 g5{...Y.........
< 00001140: 4135 0000 1f8b 0804 0000 0000 00ff 0600 A5..............
< 00001150: 4243 0200 1b00 0300 0000 0000 0000 0000 BC..............
---
> 00000ef0: f905 bbdc 7de2 8906 0bd9 e5ee 13fe 3fec ....}.........?.
> 00000f00: 9ea0 45be a55c e37b d48b cd9a e0f1 df3f ..E..\.{.......?
> 00000f10: b614 fbdf 5d3e bbab 6777 f1ec ae9d dda5 ....]>..gw......
> 00000f20: b3bb 7276 17ce eeba d95d 36bb ab66 77d1 ..rv.....]6..fw.
> 00000f30: ecae 99dd 25b3 bb62 7617 ccee 7ad9 5d2e ....%..bv...z.].
> 00000f40: bbab 6577 b1ec ae95 dda5 b2bb 5276 17ca ..ew........Rv..
> 00000f50: ee3a d95d 26bb ab64 7791 ecae 91dd 25b2 .:.]&..dw.....%.
> 00000f60: bb42 7617 c8ee fad8 5d1e bbab 6377 71ec .Bv.....]...cwq.
> 00000f70: ae8d dda5 f99f 5d19 b30b 6377 5dec 2e8b ......]...cw]...
> 00000f80: dd55 b1bb 2876 d7c4 ee92 d85d 11bb 0b62 .U..(v.....]...b
> 00000f90: 773d ec2e 87dd d5b0 bb18 76d7 c2ee 52d8 w=........v...R.
> 00000fa0: 5d09 bb0b 6177 1dec 2e83 dd55 b0bb 0876 ]...aw.....U...v
> 00000fb0: d7c0 ee12 d85d 01bb 0b60 effa ed5d bebd .....]...`...]..
> 00000fc0: abb7 77f1 f6ae ddde a5db bb72 7b17 6eef ..w........r{.n.
> 00000fd0: baed 5db6 bdab b677 d1f6 aed9 de25 dbbb ..]....w.....%..
> 00000fe0: 627b 176c ef7a ed5d aebd abb5 77b1 f6ae b{.l.z.]....w...
> 00000ff0: d5de a5da bb52 7b17 6aef 3aed 5da6 bdab .....R{.j.:.]...
> 00001000: b477 91f6 aed1 de25 dabb 427b 1768 effa .w.....%..B{.h..
> 00001010: ec5d 9ebd abb3 7771 f6ae cdde a5d9 bb32 .]....wq.......2
> 00001020: 7b17 66ef baec 5d96 bdab b277 51f6 aec9 {.f...]....wQ...
> 00001030: de25 d9bb 227b 1764 ef7a ec5d 8ebd abb1 .%.."{.d.z.]....
> 00001040: 7731 f6ae c5de a5d8 bb12 7b17 62ef 3aec w1........{.b.:.
> 00001050: 5d86 bdab b077 11f6 aec1 de25 d8bb 027b ]....w.....%...{
> 00001060: 1760 67fd 7696 6f67 f576 166f 67ed 7696 .`g.v.og.v.og.v.
> 00001070: 6e67 e576 166e 67dd 7696 6d67 d576 166d ng.v.ng.v.mg.v.m
> 00001080: 67cd 7696 6c67 c576 166c 67bd 7696 6b67 g.v.lg.v.lg.v.kg
> 00001090: b576 166b 67ad 7696 6a67 a576 166a 679d .v.kg.v.jg.v.jg.
> 000010a0: 7696 6967 9576 1669 678d 7696 6867 8576 v.ig.v.ig.v.hg.v
> 000010b0: 1668 677d 7696 6767 7576 1667 676d 7696 .hg}v.gguv.ggmv.
> 000010c0: 6667 6576 1666 675d 7696 6567 5576 1665 fgev.fg]v.egUv.e
> 000010d0: 674d 7696 6467 4576 1664 673d 7696 6367 gMv.dgEv.dg=v.cg
> 000010e0: 3576 1663 672d 7696 6267 2576 1662 671d 5v.cg-v.bg%v.bg.
> 000010f0: 7696 6167 1576 1661 670d 7696 6067 0576 v.ag.v.ag.v.`g.v
> 00001100: 1660 cffa f62c 6fcf eaf6 2c6e cfda f62c .`...,o...,n...,
> 00001110: 6dcf caf6 2c6c cfba f62c 6bcf aaf6 2c6a m...,l...,k...,j
> 00001120: cf9a f62c 69cf 8af6 2c68 cf7a f62c 67cf ...,i...,h.z.,g.
> 00001130: 6af6 2cf6 9fb3 d6a3 1fff 0f09 a01d e041 j.,............A
> 00001140: 3500 001f 8b08 0400 0000 0000 ff06 0042 5..............B
> 00001150: 4302 001b 0003 0000 0000 0000 0000 00 C..............
I was not able to reproduce this discrepancy when creating .csi index files. I am not sure whether this is related to my issue with GATK HaplotypeCaller and I could not replicate an error with BCFtools when using an index generated with --write-index=tbi but hopefully understanding the source of this discrepancy might be of use
While I was trying to run the GATK HaplotypeCaller with option
--alleles input.vcf.gzon a file that I had generated with BCFtools with option--write-index=tbi, I got the error:I read that this is usually caused by an out-of-date or corrupt index file. I then regenerated the index file with option
bcftools index --tbiand the GATK HaplotypeCaller worked without issues. Indeed the two index files generated were different with different md5sum'sI could not replicate an error with BCFtools but I did notice that
--write-index=tbiandbcftools index --tbidon't always create the same index files if applied to files large enough. This is reproducible with BCFtools 1.20 (using htslib 1.20):I could not understand why they are different:
I was not able to reproduce this discrepancy when creating
.csiindex files. I am not sure whether this is related to my issue with GATK HaplotypeCaller and I could not replicate an error with BCFtools when using an index generated with--write-index=tbibut hopefully understanding the source of this discrepancy might be of use