Skip to content

Demo / sample code update#1666

Closed
vasudeva8 wants to merge 9 commits into
samtools:developfrom
vasudeva8:demo3
Closed

Demo / sample code update#1666
vasudeva8 wants to merge 9 commits into
samtools:developfrom
vasudeva8:demo3

Conversation

@vasudeva8

Copy link
Copy Markdown
Contributor

Samples updated with fasta/fastq index using, tabix index and thread pool queueing of tasks.

Comment thread samples/write_fast.c Outdated
Comment on lines +94 to +99
//close and index it
sam_close(outfile); outfile = NULL;
if (fai_build3(outname, NULL, NULL) == -1) {
printf("Indexing failed with %d\n", errno);
goto end;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced this makes a good demonstration.

Fasta indices are good for references as they allow us to extract small portions of a large reference. However they are wholely inappropriate for NGS data, which is typically what we have in a SAM/BAM file.

Basically fasta/fastq are split personally and used for two different types of data. Messy.

We may be better having a tiny demo of a separate "samtools faidx" index builder instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This writes reference data in fasta/fastq format and showcases creation of index for later use, while reading.
It is not using NGS data, and uses sam_open/write/close and bam1_t for this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reading DEMO.md I see it explains the indexing is suitable for reference data.

As this is for education purposes, maybe put a comment just before the fai_build3 call: // NB: this is only suitable for fasta/fastq files of reference sequences, and not sequencing reads

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separated as a different samples and mentioned as indexing of reference data.

Comment thread samples/read_with_tabix.c Outdated
Comment thread samples/read_with_tabix.c Outdated
Comment thread samples/DEMO.md Outdated
Comment thread samples/read_with_tabix.c Outdated
Comment thread samples/qtask_thread.c Outdated
Comment thread samples/qtask_thread.c Outdated
Comment thread samples/qtask_thread.c Outdated
Comment thread samples/qtask_thread.c Outdated
Comment thread samples/qtask_thread.c Outdated
@jkbonfield

Copy link
Copy Markdown
Contributor

I ran a spell checker over the DEMO and README files. Highlighted in bold. Some were there before, hence not easy to add comments to lines here, but it's trivial to search and fix.

Lines in README.md:
"length equal or greather to given input"
"This application shocases the use of mulitple queues,"

DEMO.md:
"Read_multireg - This application showcases the usage of mulitple regionn "
"Qtask_thread - This application shocases the use of mulitple queues, scheduling of different tasks and getting results in orderered and unordered fashion."
"Pileup shows the transposed view of the SAM alignment data, i.e. it shows the the reference positions"

@vasudeva8

Copy link
Copy Markdown
Contributor Author

Spell check and other corrections made.

@jkbonfield jkbonfield left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments on qtask_ordered.c only.

I'm tempted to say to drop this for this release if we want the rest of the changes merging, as it needs major work to fix race conditions.

Comment thread samples/qtask_ordered.c Outdated
Comment thread samples/qtask_ordered.c Outdated
Comment thread samples/qtask_unordered.c Outdated
@jkbonfield

Copy link
Copy Markdown
Contributor

I have a rebased version here that we may want to use before doing further changes. It fixes that bizarre github induced merge commit.

I haven't squashed anything yet, so all commits are the same content, but we can do that when we're ready to merge.

@vasudeva8 vasudeva8 force-pushed the demo3 branch 3 times, most recently from 3422cdf to 20aaa40 Compare May 1, 2024 13:18
@vasudeva8

Copy link
Copy Markdown
Contributor Author

thanks James, have used this rebased version and pushed it.
also the thread pool is shared with input and output files as well, to make it a better sample as mentioned.

@jkbonfield

Copy link
Copy Markdown
Contributor

I'm still finding qtask_ordered to be considerably slower than it needs to be.

I reimplemented the qtask_ordered problem using my own logic, mostly copied from the way htslib does multi-threading of SAM by generating blocks of BAM records to operate on and separate reading and writing threads to keep things running.

My code for this is here: https://github.com/jkbonfield/htslib/blob/PR_1666d/tv2.c

$ for i in `seq 1 5`;do taskset -c 0-15 /usr/bin/time -f '%U user\t%S system\
t%e elapsed\t%P %%CPU' ./qtask_ordered ~/scratch/data/novaseq.10m.bam 16 /tmp;done
35.51 user	7.23 system	10.69 elapsed	399% %CPU
37.74 user	7.68 system	10.43 elapsed	435% %CPU
36.11 user	7.90 system	11.44 elapsed	384% %CPU
38.03 user	7.74 system	10.60 elapsed	431% %CPU
38.71 user	8.28 system	12.48 elapsed	376% %CPU

So the original is typically 10-11s and maybe around 400% CPU utilisation.

$ for i in `seq 1 5`;do taskset -c 0-15 /usr/bin/time -f '%U user\t%S system\t%e elapsed\t%P %%CPU' ../tv2 -@16 ~/scratch/data/novaseq.10m.bam /tmp/_.sam > /dev/null;done
18.33 user	3.06 system	2.33 elapsed	916% %CPU
19.42 user	3.43 system	2.67 elapsed	855% %CPU
19.27 user	3.56 system	2.64 elapsed	862% %CPU
18.40 user	3.05 system	2.40 elapsed	890% %CPU
19.43 user	3.54 system	2.70 elapsed	848% %CPU

This is 850-900% CPU utilisation and ~2.5s. So around 4x quicker. It's clear there is also a considerable difference in CPU usage, which I can't explain yet, but possibly it's the optimisations in the core counting algorithm. (I should probably remove those as it's not good at demonstrating the benefits of threading when the task we are performing is so quick.)

The output of out.sam and _.sam are identical, although I had to change my code a bit because we counted differently. I think the GC content in this PR is subtly wrong in that it is numbers of G and C over the sequence length, not over the numbers of A, C, G and T. It only differs when Ns are present (or ambiguity codes, but they didn't occur). We don't know what the N base is so it shouldn't be counted in the maths.

Anyway I digress. If we want a demonstration of threading then it needs to be performant.

I also have a different version of this (tv1) which is purely counting the frequency of each base type and reporting summary statistics. That doesn't write anything back. However it means the order in which we do our sums is irrelevant as A + B + C is the same as A + C + B. Hence this becomes a trivial change to unordered threads instead. With hindsight I think that's a simpler example, but the one here may still be relevant too and has practical purpose.

@jkbonfield

Copy link
Copy Markdown
Contributor

I tried changing qtask_ordered to use "wb" to write to "out.bam" instead. It obviously then uses more CPU up as compression of the output becomes significant, but it's still behind tv2's BAM output.

#qtask_ordered
100.16 user	4.93 system	10.18 elapsed	1031% %CPU
100.79 user	5.13 system	10.42 elapsed	1016% %CPU
100.58 user	4.86 system	10.18 elapsed	1035% %CPU
100.91 user	5.17 system	10.43 elapsed	1016% %CPU
100.39 user	5.15 system	10.44 elapsed	1010% %CPU

#tv2
84.84 user	2.06 system	6.31 elapsed	1375% %CPU
84.44 user	1.72 system	5.93 elapsed	1452% %CPU
84.55 user	1.98 system	5.93 elapsed	1458% %CPU
84.77 user	1.73 system	5.94 elapsed	1454% %CPU
84.38 user	1.81 system	6.02 elapsed	1431% %CPU

It's interesting we still see a signficant "system" time difference, which could be something to do with malloc/free and zeroing pages, or it could be something else such as pthread interfaces. I haven't profiled it to find out.

On a larger file this seems to be more pronounced too:

#qtask_ordered
1226.54 user	290.74 system	316.69 elapsed	479% %CPU

#tv2
710.26 user	17.12 system	49.46 elapsed	1470% %CPU
715.45 user	15.22 system	50.37 elapsed	1450% %CPU

System time here is vastly bigger, which is a big hint. Note it's possible to profile the main thread and main thread only, to see where it's spending the CPU time. That in turn may explain why it can't parallelise as well as it ought to. Eg:

./qtask_ordered /local/scratch01/jkb/SRR6882909.name-.bam 16 /tmp & perf record -t $!
[ Wait and then control-C ]

perf report then shows:
  50.93%  qtask_ordered  [kernel]            [k] 0xffffffff99800190                                   ▒
  15.16%  qtask_ordered  libc-2.27.so        [.] _int_malloc                                          ▒
   9.10%  qtask_ordered  libc-2.27.so        [.] calloc                                               ▒
   6.76%  qtask_ordered  libc-2.27.so        [.] __libc_malloc                                        ▒
   4.54%  qtask_ordered  qtask_ordered       [.] bam_read1                                            ▒
   2.64%  qtask_ordered  libc-2.27.so        [.] __memmove_sse2_unaligned_erms                        ▒
   1.75%  qtask_ordered  qtask_ordered       [.] bgzf_read                                            ▒
   1.67%  qtask_ordered  libc-2.27.so        [.] __lll_lock_wait_private                              ▒
   1.11%  qtask_ordered  qtask_ordered       [.] sam_realloc_bam_data                                 ▒
   1.07%  qtask_ordered  qtask_ordered       [.] sam_read1                                            ▒
   0.82%  qtask_ordered  libc-2.27.so        [.] memcpy@GLIBC_2.2.5                                   ▒
   0.74%  qtask_ordered  qtask_ordered       [.] main                                                 ▒
   0.63%  qtask_ordered  qtask_ordered       [.] bam_tag2cigar                                        ▒

It's somewhere in the kernel, but that's probably memory page clearing given the other high CPU functions. Looking at it, it seems to be spending most of its time allocating bam objects. This is one way my implementation majorly differs. I don't use bam_init1 and I don't free the bams between successive calls. I have a queue of bam arrays that I push onto and pull from, so I can reuse my bam-array from a previous job. Don't give the memory back to the system until program exit and it'll be MUCH faster!

@vasudeva8

Copy link
Copy Markdown
Contributor Author

Thanks for the sample James.

The aim of the sample was just to demonstrate the usage of queues and hts thread pool, specifically the 2 different ways of its usage, made as basic as possible for that purpose. It had no intention to show the user how best to use threading in general to get best performance.

Based on earlier reviews, I have incorporated the input and output file thread pool sharing as it gives some direction on hts thread pool usage. And taking it to next level with separate read/write threads may make the sample complex and shifts the objective of it.

But if we need one such sample, may be we should add as a separate one? Or just refer the user to a samtools functionality?

vasudeva8 and others added 5 commits July 2, 2024 16:07
update with correction

fixed issues
Use hts_tpool_next_result_wait() in threadfn_orderedwrite() so it
blocks properly when it has nothing to do.  Remove lock and
conditional that are no longer needed.  Use a sentinel job to
tell threadfn_orderedwrite() that there is no more data and it can
terminate.

Improve error handling.  Add a data::result field so that
threadfn_orderedwrite() can report back to the main thread if
it encountered a problem.  Ensure everything is cleaned up
correctly if either the main thread or threadfn_orderedwrite()
fail, and in the right order.  Set the return value to EXIT_FAILURE
if either sam_close() or threadfn_orderedwrite() report failure.

Ensure error messages are written to stderr.
@vasudeva8

Copy link
Copy Markdown
Contributor Author

Thanks James and Rob.
I have merged and updated the samples. Now they use alignments in chunks and reuses memory. Also the removed lookup / decoding of base from loop to outside the loop.
The unordered processing now just dumps the count of bases and gc ratio.

@jkbonfield

Copy link
Copy Markdown
Contributor

I've not checked the code yet, but I can verify the speed is now similar to my own implementations.

I think there are some minor niceties with this directory to fix too.

  • The default build is -O0 when it perhaps should be -O2. People can always override it with "CFLAGS=-g" on the make command line if they wish.
  • There are no dependencies, so if source is modified then it doesn't rebuild. It's just a matter of adding the source name in the build rules, e.g flags: flags.c

Comment thread samples/DEMO.md Outdated
Comment on lines +989 to +996
if (!(data = faidx_fetch_seq64(idx, faidx_iseq(idx, tid), beg, end, &len))) {
...
printf("Data: %"PRIhts_pos" %s\n", len, data);
free((void*)data);
data = NULL;
//get quality data for fastq
if (fmt == FAI_FASTQ) {
if (!(data = faidx_fetch_qual64(idx, faidx_iseq(idx, tid), beg, end, &len))) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appreciate this is just code from the demo programs that's been cutdown with "..." to get to the essence of things, but it's losing the if-else structure which is highly confusing. We could add elses clauses, but I think really the correct solution is negating the if statements.

Ie:

if ((data = faidx_fetch_seq64(idx, faidx_iseq(idx, tid), beg, end, &len)) != NULL) {
    printf("Data: %"PRIhts_pos" %s\n", len, data);
    ...

We don't need the explicit error checking here as DEMO.md isn't designed as a guide for how to write bullet proof code and instead is trying to describe the most important features. However keeping an if statement with an explicit != NULL is a hint that it can fail and it's up to the programmer to handle that (and is demonstrated in the actual sample code).

It's probably easier for me to just go through this and make a new commit on to this PR than to document each case I find here.

The "..." to indicate clipped out code is useful as it removes a lot
of the unnecessary boiler plate, especially error checking, and allows
us to drill down to the essense of what we're trying to teach.
However without indentation it's not clear what bits are being
removed.

Also there are times when the code removed is an error handler plus
a subsequent else clause, which gives the false impression with the
indentation that the if-clause has been negated.  Eg:

    ...
    //select required field alone, this is useful for CRAM alone
    if (hts_set_opt(infile, CRAM_OPT_REQUIRED_FIELDS, SAM_FLAG) < 0) {
    ...
       //read header
       in_samhdr = sam_hdr_read(infile);
    ...

Clearly the `sam_hdr_read` is not within the if-clause of
`hts_set_opt` returning < 0, but it's how it reads because "..."
hid too much.  (In the above case just indenting the ... and
un-indenting the hdr read call reverses the mental image.)

My approach here is to replace all `...` error handlers with `... // error`
and to indent them accordingly.

Also where we only have a one line thing, removing open curly braces
if we've cut the close curly brace gives a more sensible view.

None of this is really changing the code we're showing, but presenting
it in a more obvious manner to those that don't understand what is and
isn't an error case.
Comment thread samples/read_fast.c
@jkbonfield

Copy link
Copy Markdown
Contributor

Squashed, rebased and merged as f8016c0.

I didn't force push back over this as you may wish to keep the separate commit histories as there was a lot of back and forth, so I'll leave you to decide whether to delete the branch.

@jkbonfield jkbonfield closed this Jul 18, 2024
gpertea added a commit to gpertea/htslib that referenced this pull request Mar 17, 2025
Notice: this is the last SAMtools / HTSlib release where CRAM 3.0
will be the default CRAM version.  From the next we will change to
CRAM 3.1 unless the version is explicitly specified, for example
using "samtools view -O cram,version=3.0".

Updates
-------

* Extend annot-tsv with several new command line options.
    --delim permits use of other delimiters.
    --headers for selection of other header formats.
    --no-header-idx to suppress column index numbers in header.
  Also removed -h as it is now short for --headers.  Note --help
  still works. (PR samtools#1779)

* Allow annot-tsv -a to rename annotations. (PR samtools#1709)

* Extend annot-tsv --overlap to be able to specify the overlap
  fraction separately for source and target. (PR samtools#1811)

* Added new APIs to facilitate low-level CRAM container
  manipulations, used by   the new "samtools cat" region
  filtering code. Functions are:
    cram_container_get_coords()
    cram_filter_container()
    cram_index_extents()
    cram_container_num2offset()
    cram_container_offset2num()
    cram_num_containers()
    cram_num_containers_between()
  Also improved cram_index_query() to cope with HTS_IDX_NOCOOR
  regions.  (PR samtools#1771)

* Bgzip now retains file modification and access times when
  compressing and decompressing. (PR samtools#1727, fixes samtools#1718.
  Requested by Gert Hulselmans.)

* Use FNV1a for string hashing in khash.  The old algorithm was
  particularly weak with base-64 style strings and lead to a large
  number of collisions.  (PR samtools#1806.  Fixes samtools/samtools#2066,
  reported by Hans-Joachim Ruscheweyh)

* Improve the speed of the nibble2base() function on Intel (PR samtools#1667,
  PR samtools#1764, PR samtools#1786, PR samtools#1802, thanks to Ruben Vorderman) and ARM
  (PR samtools#1795, thanks to John Marshall).

* bgzf_getline() will now warn if it encounters UTF-16 data. (PR
  samtools#1487, thanks to John Marshall)

* Speed up bgzf_read().  While this does not reduce CPU
  significantly, it does increase the maximum parallelism
  available permitting 10-15% faster decoding. (PR samtools#1772, PR
  samtools#1800, Issue samtools#1798)

* Speed up faidx by use of better isgraph methods (PR samtools#1797) and
  whole-line reading (PR samtools#1799, thanks to John Marshall).

* Speed up kputll() function, speeding up BAM -> SAM conversion by
  about 5% and also samtools depth.  (PR samtools#1805)

* Added more example code, covering fasta/fastq indexing, tabix
  indexing and use of the thread pool. (PR samtools#1666)

Build Changes
-------------

* Code warning fixes for pedantic compilers (PR samtools#1777) and avoid some
  undefined behaviour (PR samtools#1810, PR samtools#1816, PR samtools#1828).

* Windows based CI has been migrated from AppVeyor to GitHub Actions.
  (PR samtools#1796, PR samtools#1803, PR samtools#1808)

* Miscellaneous minor build infrastructure and code fixes. (PR samtools#1807,
  PR samtools#1829, both thanks to John Marshall)

* Updated htscodecs submodule to version 1.6.1 (PR samtools#1828)

* Fixed an awk script in the Makefile that only worked with gawk. (PR
  samtools#1831)

Bug fixes
---------

* Fix small OSS-Fuzz reported issues with CRAM encoding and long
  CIGARS and/or illegal positions. (PR samtools#1775, PR samtools#1801, PR samtools#1817)

* Fix issues with on-the-fly indexing of VCF/BCF (bcftools
  --write-index) when not using multiple threads. (PR samtools#1837.
  Fixes samtools/bcftools#2267, reported by Giulio Genovese)

* Stricter limits on POS / MPOS / TLEN in sam_parse1().  This fixes a
  signed overflow reported by OSS-Fuzz and should help prevent other
  as-yet undetected bugs. (PR samtools#1812)

* Check that the underlying file open worked for preload: URLs.
  Fixes a NULL pointer dereference reported by OSS-Fuzz. (PR samtools#1821)

* Fix an infinite loop in hts_itr_query() when given extremely large
  positions which cause integer overflow.  Also adds hts_bin_maxpos()
  and hts_idx_maxpos() functions. (PR samtools#1774, thanks to John Marshall
  and reported by Jesus Alberto Munoz Mesa)

* Fix an out of bounds read in hts_itr_multi_next() when switching
  chromosomes.  This bug is present in releases 1.11 to 1.20. (PR
  samtools#1788. Fixes samtools/samtools#2063, reported by acorvelo)

* Work around parsing problems with colons in CHROM names. Fixes
  samtools/bcftools#2139.  (PR samtools#1781, John Marshall / James Bonfield)

* Correct the CPU detection for Mac OS X 10.7.  cpuid is used by
  htscodecs (see samtools/htscodecs#116), and the corresponding
  changes in htslib are PR samtools#1785.  Reported by Ryan Carsten Schmidt.

* Make BAM zero-length intervals work the same as CRAM; permitted
  and returning overlapping records. (PR samtools#1787.  Fixes
  samtools/samtools#2060, reported by acorvelo)

* Replace assert() with abort() in BCF synced reader.  This is not an
  ideal solution, but it gives consistent behaviour when compiling
  with or without NDEBUG.  (PR samtools#1791, thanks to Martin Pollard)

* Fixed failure to change the write block size on compressed SAM or
  VCF files due to an internal type confusion.  (PR samtools#1826)

* Fixed an out-of-bounds read in cram_codec_iter_next() (PR samtools#1832)
diekhans added a commit to diekhans/htslib that referenced this pull request May 27, 2025
htslib release 1.21:

The primary user-visible changes in this release are updates to
the annot-tsv tool and some speed improvements.  Full details of
other changes and bugs fixed are below.

Notice: this is the last SAMtools / HTSlib release where CRAM 3.0
will be the default CRAM version.  From the next we will change to
CRAM 3.1 unless the version is explicitly specified, for example
using "samtools view -O cram,version=3.0".

Updates
-------

* Extend annot-tsv with several new command line options.
    --delim permits use of other delimiters.
    --headers for selection of other header formats.
    --no-header-idx to suppress column index numbers in header.
  Also removed -h as it is now short for --headers.  Note --help
  still works. (PR samtools#1779)

* Allow annot-tsv -a to rename annotations. (PR samtools#1709)

* Extend annot-tsv --overlap to be able to specify the overlap
  fraction separately for source and target. (PR samtools#1811)

* Added new APIs to facilitate low-level CRAM container
  manipulations, used by   the new "samtools cat" region
  filtering code. Functions are:
    cram_container_get_coords()
    cram_filter_container()
    cram_index_extents()
    cram_container_num2offset()
    cram_container_offset2num()
    cram_num_containers()
    cram_num_containers_between()
  Also improved cram_index_query() to cope with HTS_IDX_NOCOOR
  regions.  (PR samtools#1771)

* Bgzip now retains file modification and access times when
  compressing and decompressing. (PR samtools#1727, fixes samtools#1718.
  Requested by Gert Hulselmans.)

* Use FNV1a for string hashing in khash.  The old algorithm was
  particularly weak with base-64 style strings and lead to a large
  number of collisions.  (PR samtools#1806.  Fixes samtools/samtools#2066,
  reported by Hans-Joachim Ruscheweyh)

* Improve the speed of the nibble2base() function on Intel (PR samtools#1667,
  PR samtools#1764, PR samtools#1786, PR samtools#1802, thanks to Ruben Vorderman) and ARM
  (PR samtools#1795, thanks to John Marshall).

* bgzf_getline() will now warn if it encounters UTF-16 data. (PR
  samtools#1487, thanks to John Marshall)

* Speed up bgzf_read().  While this does not reduce CPU
  significantly, it does increase the maximum parallelism
  available permitting 10-15% faster decoding. (PR samtools#1772, PR
  samtools#1800, Issue samtools#1798)

* Speed up faidx by use of better isgraph methods (PR samtools#1797) and
  whole-line reading (PR samtools#1799, thanks to John Marshall).

* Speed up kputll() function, speeding up BAM -> SAM conversion by
  about 5% and also samtools depth.  (PR samtools#1805)

* Added more example code, covering fasta/fastq indexing, tabix
  indexing and use of the thread pool. (PR samtools#1666)

Build Changes
-------------

* Code warning fixes for pedantic compilers (PR samtools#1777) and avoid some
  undefined behaviour (PR samtools#1810, PR samtools#1816, PR samtools#1828).

* Windows based CI has been migrated from AppVeyor to GitHub Actions.
  (PR samtools#1796, PR samtools#1803, PR samtools#1808)

* Miscellaneous minor build infrastructure and code fixes. (PR samtools#1807,
  PR samtools#1829, both thanks to John Marshall)

* Updated htscodecs submodule to version 1.6.1 (PR samtools#1828)

* Fixed an awk script in the Makefile that only worked with gawk. (PR
  samtools#1831)

Bug fixes
---------

* Fix small OSS-Fuzz reported issues with CRAM encoding and long
  CIGARS and/or illegal positions. (PR samtools#1775, PR samtools#1801, PR samtools#1817)

* Fix issues with on-the-fly indexing of VCF/BCF (bcftools
  --write-index) when not using multiple threads. (PR samtools#1837.
  Fixes samtools/bcftools#2267, reported by Giulio Genovese)

* Stricter limits on POS / MPOS / TLEN in sam_parse1().  This fixes a
  signed overflow reported by OSS-Fuzz and should help prevent other
  as-yet undetected bugs. (PR samtools#1812)

* Check that the underlying file open worked for preload: URLs.
  Fixes a NULL pointer dereference reported by OSS-Fuzz. (PR samtools#1821)

* Fix an infinite loop in hts_itr_query() when given extremely large
  positions which cause integer overflow.  Also adds hts_bin_maxpos()
  and hts_idx_maxpos() functions. (PR samtools#1774, thanks to John Marshall
  and reported by Jesus Alberto Munoz Mesa)

* Fix an out of bounds read in hts_itr_multi_next() when switching
  chromosomes.  This bug is present in releases 1.11 to 1.20. (PR
  samtools#1788. Fixes samtools/samtools#2063, reported by acorvelo)

* Work around parsing problems with colons in CHROM names. Fixes
  samtools/bcftools#2139.  (PR samtools#1781, John Marshall / James Bonfield)

* Correct the CPU detection for Mac OS X 10.7.  cpuid is used by
  htscodecs (see samtools/htscodecs#116), and the corresponding
  changes in htslib are PR samtools#1785.  Reported by Ryan Carsten Schmidt.

* Make BAM zero-length intervals work the same as CRAM; permitted
  and returning overlapping records. (PR samtools#1787.  Fixes
  samtools/samtools#2060, reported by acorvelo)

* Replace assert() with abort() in BCF synced reader.  This is not an
  ideal solution, but it gives consistent behaviour when compiling
  with or without NDEBUG.  (PR samtools#1791, thanks to Martin Pollard)

* Fixed failure to change the write block size on compressed SAM or
  VCF files due to an internal type confusion.  (PR samtools#1826)

* Fixed an out-of-bounds read in cram_codec_iter_next() (PR samtools#1832)

# -----BEGIN PGP SIGNATURE-----
#
# iQJJBAABCgAzFiEEaGn1wQ1nqxSLDC2XHsvluhXGTMwFAmbjBlMVHHJtZCtnaXRA
# c2FuZ2VyLmFjLnVrAAoJEB7L5boVxkzMgb0P/3AAxkJNLGK8Q1pBBo+in7KzwBOc
# /nRfvReki47r4+i1ev98abEF/+XZoYSS8ZkiJH8JMkLCclb1ch2fDQTIL/sjJwrU
# MfhYx6oHhzLUOjpG24m390wlvlkEsg5S69jbgImWxLQ1n4ZcSuZGumiwPx5274w5
# 3R/K9vIXtuSF9DTD/jrTgoZeBiN+B4XefAJqHLSo4/XzDo7DNMSZryNGFpHdyKcB
# oEfEXX45vJLgzA3Fxo3LqNpC8UTDO5C9u9Uup+hCLN7RTBjV5Fu0UxcwWTSPqfiC
# Jroi2FhxNPlDkRL3jraI8LrB3Cqb0G/31OU+cryasrqYnfJrl0e/MyXiRSa0sg25
# uHHWVyyC4JR7uBfUCppTVf76iajbkIVpHgHLZS1heGwM4PCL+AalvsIgGcvLXcBs
# g/AwZLoXzao+pbzg8hzN1JuLB5hHcrHp6TsQ6JbTTrWaLeAL3s/+eQ4+d+Np2iUn
# v8TMg6onu1HiNgVMQcFXSLC2Q51rB+lD6AA3eusKjcQsaxyk+U7SLCx3OhbvxGsd
# SjxQWP1W+WW//e9nm5f4CTSSf1PHoDMyR/kVEc3qwEWJslri68RyB69HrDIVzrWT
# 3gKAKJYafi5ssFHsKTsCngESpDsTg3aQV1AGOQPLN/TIjyYDTUOL+/HylRs3k9XN
# KOGO5yggx+yqeL3w
# =lHVl
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 12 Sep 2024 08:18:43 AM PDT
# gpg:                using RSA key 6869F5C10D67AB148B0C2D971ECBE5BA15C64CCC
# gpg:                issuer "rmd+git@sanger.ac.uk"
# gpg: Can't check signature: No public key
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants