Add region and section/part filtering to CRAM samtools cat.#2035
Merged
Conversation
Contributor
Author
|
Note this requires samtools/htslib#1771 merging first due to additional htslib APIs being added. |
Contributor
Author
|
Ping @mcshane. Please check if the syntax and functionality for this works for you as a replacement to io_lib's filtering. Thanks |
3a21201 to
73453c1
Compare
Region filtering permits samtools cat to produce new CRAMs that only
cover a specified region. Eg
samtools cat -o chr1-bit.cram -r chr1:10m-20m in.cram
This is the same as as view command, but considerably faster.
samtools view -o chr1-bit.cram in.cram chr1:10m-20m
On a similar test the view command took 22.7s while the cat took 0.4s
and the files have the same records. Unlike view we cannot change
format or do other filtering, however this samtools cat functionality
is designed for rapid read/write based region data extraction for
purposes of distributed processing pipelines.
This can be made faster still with the new -f option which doesn't
precisely filter containers that overlap the start and/or end of the
region, and instead copy out the whole container.
Note breaking up by region in this manner may give some alignment
records in more than one adjacent CRAM file as the record spans both
regions.
As elsewhere, region "*" represents the unmapped data, however we
can't subdivide this easily by position. Hence we can partition a
region, or the entire file, using `-p A/B` for part A of B total
parts.
For example the first tenth of unmapped data would be
samtools cat -r "*" -p 1/10 -o unmapped_1.cram NA12878_S1.cram
Subdivisions in this manner will never share alignment records between
parts, so the sum of the parts will equal the original input data.
Regions can also be container numbers, using a custom region syntax of
"#:cnum-cnum" for inclusive container numbers, starting at zero. As
with part numbers, records will not then be in multple output files
unless the same container numbers are shared between files.
We can use `samtools cat -q [-r region]` to query the number of
containers spanning that region and the first/last respectively.
For example, to query the containers for unmapped data.
samtools cat -q -r "*" NA12878_S1.cram
A more complex usage which breaks a file up into a fixed size number
of containers regardless of part numbers can be implemented using `-q`:
set -- $(samtools cat -q novaseq.10m.cram)
nc=$2
s=0
while [ $s -lt $nc ]
do
e=`expr $s + 123`
if [ $e -ge $nc ]
then
e=$nc
fi
r="#:$s-`expr $e - 1`"
echo Region $r
samtools cat -o part-`printf "%08d" $s`.cram novaseq.10m.cram -r $r
s=$e
done
# Check with:
samtools cat part*cram -o full.cram
samtools view -c full.cram
samtools view -c novaseq.10m.cram
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Region filtering permits samtools cat to produce new CRAMs that only cover a specified region.
This now acts as a potential replacement for io_lib's cram_filter tool.
Eg
This is the same as as view command, but considerably faster.
On a similar test the view command took 22.7s while the cat took 0.4s and the files have the same records. Unlike view we cannot change format or do other filtering, however this samtools cat functionality is designed for rapid read/write based region data extraction for purposes of distributed processing pipelines.
This can be made faster still with the new -f option which doesn't precisely filter containers that overlap the start and/or end of the region, and instead copy out the whole container.
Note breaking up by region in this manner may give some alignment records in more than one adjacent CRAM file as the record spans both regions.
As elsewhere, region "*" represents the unmapped data, however we can't subdivide this easily by position. Hence we can partition a region, or the entire file, using
-p A/Bfor part A of B total parts.For example the first tenth of unmapped data would be
Subdivisions in this manner will never share alignment records between parts, so the sum of the parts will equal the original input data.
Regions can also be container numbers, using a custom region syntax of "#:cnum-cnum" for inclusive container numbers, starting at zero. As with part numbers, records will not then be in multple output files unless the same container numbers are shared between files.
We can use
samtools cat -q [-r region]to query the number of containers spanning that region and the first/last respectively. For example, to query the containers for unmapped data.A more complex usage which breaks a file up into a fixed size number of containers regardless of part numbers can be implemented using
-q: