Skip to content

Add region and section/part filtering to CRAM samtools cat.#2035

Merged
whitwham merged 1 commit into
samtools:developfrom
jkbonfield:cram-filter
Jun 7, 2024
Merged

Add region and section/part filtering to CRAM samtools cat.#2035
whitwham merged 1 commit into
samtools:developfrom
jkbonfield:cram-filter

Conversation

@jkbonfield

Copy link
Copy Markdown
Contributor

Region filtering permits samtools cat to produce new CRAMs that only cover a specified region.
This now acts as a potential replacement for io_lib's cram_filter tool.
Eg

samtools cat -o chr1-bit.cram -r chr1:10m-20m in.cram

This is the same as as view command, but considerably faster.

samtools view -o chr1-bit.cram in.cram chr1:10m-20m

On a similar test the view command took 22.7s while the cat took 0.4s and the files have the same records. Unlike view we cannot change format or do other filtering, however this samtools cat functionality is designed for rapid read/write based region data extraction for purposes of distributed processing pipelines.

This can be made faster still with the new -f option which doesn't precisely filter containers that overlap the start and/or end of the region, and instead copy out the whole container.

Note breaking up by region in this manner may give some alignment records in more than one adjacent CRAM file as the record spans both regions.

As elsewhere, region "*" represents the unmapped data, however we can't subdivide this easily by position. Hence we can partition a region, or the entire file, using -p A/B for part A of B total parts.

For example the first tenth of unmapped data would be

samtools cat -r "*" -p 1/10 -o unmapped_1.cram NA12878_S1.cram

Subdivisions in this manner will never share alignment records between parts, so the sum of the parts will equal the original input data.

Regions can also be container numbers, using a custom region syntax of "#:cnum-cnum" for inclusive container numbers, starting at zero. As with part numbers, records will not then be in multple output files unless the same container numbers are shared between files.

We can use samtools cat -q [-r region] to query the number of containers spanning that region and the first/last respectively. For example, to query the containers for unmapped data.

samtools cat -q -r "*" NA12878_S1.cram

A more complex usage which breaks a file up into a fixed size number of containers regardless of part numbers can be implemented using -q:

set -- $(samtools cat -q novaseq.10m.cram)
nc=$2
s=0
while [ $s -lt $nc ]
do
    e=`expr $s + 123`
    if [ $e -ge $nc ]
    then
        e=$nc
    fi
    r="#:$s-`expr $e - 1`"
    echo Region $r
    samtools cat -o part-`printf "%08d" $s`.cram novaseq.10m.cram -r $r
    s=$e
done

# Check with:
samtools cat part*cram -o full.cram
samtools view -c full.cram
samtools view -c novaseq.10m.cram

@jkbonfield

Copy link
Copy Markdown
Contributor Author

Note this requires samtools/htslib#1771 merging first due to additional htslib APIs being added.

@jkbonfield

Copy link
Copy Markdown
Contributor Author

Ping @mcshane. Please check if the syntax and functionality for this works for you as a replacement to io_lib's filtering. Thanks

@jkbonfield jkbonfield force-pushed the cram-filter branch 2 times, most recently from 3a21201 to 73453c1 Compare June 6, 2024 13:58
Region filtering permits samtools cat to produce new CRAMs that only
cover a specified region.  Eg

    samtools cat -o chr1-bit.cram -r chr1:10m-20m in.cram

This is the same as as view command, but considerably faster.

    samtools view -o chr1-bit.cram in.cram chr1:10m-20m

On a similar test the view command took 22.7s while the cat took 0.4s
and the files have the same records.  Unlike view we cannot change
format or do other filtering, however this samtools cat functionality
is designed for rapid read/write based region data extraction for
purposes of distributed processing pipelines.

This can be made faster still with the new -f option which doesn't
precisely filter containers that overlap the start and/or end of the
region, and instead copy out the whole container.

Note breaking up by region in this manner may give some alignment
records in more than one adjacent CRAM file as the record spans both
regions.

As elsewhere, region "*" represents the unmapped data, however we
can't subdivide this easily by position. Hence we can partition a
region, or the entire file, using `-p A/B` for part A of B total
parts.

For example the first tenth of unmapped data would be

    samtools cat -r "*" -p 1/10 -o unmapped_1.cram NA12878_S1.cram

Subdivisions in this manner will never share alignment records between
parts, so the sum of the parts will equal the original input data.

Regions can also be container numbers, using a custom region syntax of
"#:cnum-cnum" for inclusive container numbers, starting at zero.  As
with part numbers, records will not then be in multple output files
unless the same container numbers are shared between files.

We can use `samtools cat -q [-r region]` to query the number of
containers spanning that region and the first/last respectively.
For example, to query the containers for unmapped data.

    samtools cat -q -r "*" NA12878_S1.cram

A more complex usage which breaks a file up into a fixed size number
of containers regardless of part numbers can be implemented using `-q`:

    set -- $(samtools cat -q novaseq.10m.cram)
    nc=$2
    s=0
    while [ $s -lt $nc ]
    do
        e=`expr $s + 123`
        if [ $e -ge $nc ]
        then
            e=$nc
        fi
        r="#:$s-`expr $e - 1`"
        echo Region $r
        samtools cat -o part-`printf "%08d" $s`.cram novaseq.10m.cram -r $r
        s=$e
    done

    # Check with:
    samtools cat part*cram -o full.cram
    samtools view -c full.cram
    samtools view -c novaseq.10m.cram
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants