Are you using the latest version of samtools and HTSlib? If not, please specify.
samtools 1.12
Using htslib 1.12
Same behaviour under:
samtools 1.20
Using htslib 1.20
Please describe your environment.
- Linux 4.18.0-513.24.1.el8_9.x86_64
- x86_64
- samtools downloaded via conda
Please specify the steps taken to generate the issue, the command you are running and the relevant output.
I have generated a fasta file with ~40 mio dereplicated genes. Header names are unique and are just 6 alphanumeric values [A-Z][A-Za-z0-9]*5 (e.g CaA3xL, CaA3xM, CaA3xN, CaA3xQ). Next I used BWA (0.7.17-r1188) to build an index and started aligning reads, alignments are stored in a sam file.
e.g.
bwa mem -a -t 1 index reads.fq.gz > reads_vs_index.sam
Alignments were produced quickly. Next I translated the sam into a bam file
cat reads_vs_index.sam | samtools view -bh - > reads_vs_index.bam
The BAM file remained empty for 15 hours until samtools started writing the header. samtools was constantly using 100% of a single core (so wasnt idle) and used between 1GB and 8GB of RAM.
I have tried the same exercise but renamed the genes in the fasta file (A_00000001, A_000000002,...). In that case the samtools command started writing the header within 1-2 minutes. There seems to be an internal process in samtools which performs differently depending on header names (maybe some hashing that creates too many collisions?). Any clue where this issue comes from and how to pick the shortest possible header name (needed for large catalogs where we run into the 2^31 header limit) which doesnt need hours to load with samtools?
Best and thanks,
Hans
Are you using the latest version of samtools and HTSlib? If not, please specify.
Same behaviour under:
Please describe your environment.
Please specify the steps taken to generate the issue, the command you are running and the relevant output.
I have generated a fasta file with ~40 mio dereplicated genes. Header names are unique and are just 6 alphanumeric values [A-Z][A-Za-z0-9]*5 (e.g CaA3xL, CaA3xM, CaA3xN, CaA3xQ). Next I used BWA (0.7.17-r1188) to build an index and started aligning reads, alignments are stored in a sam file.
e.g.
Alignments were produced quickly. Next I translated the sam into a bam file
The BAM file remained empty for 15 hours until samtools started writing the header. samtools was constantly using 100% of a single core (so wasnt idle) and used between 1GB and 8GB of RAM.
I have tried the same exercise but renamed the genes in the fasta file (A_00000001, A_000000002,...). In that case the samtools command started writing the header within 1-2 minutes. There seems to be an internal process in samtools which performs differently depending on header names (maybe some hashing that creates too many collisions?). Any clue where this issue comes from and how to pick the shortest possible header name (needed for large catalogs where we run into the 2^31 header limit) which doesnt need hours to load with samtools?
Best and thanks,
Hans