Skip to content

Fix samtools merge --template-coordinate#2062

Merged
daviesrob merged 5 commits into
samtools:developfrom
nh13:bugfix/nh13-template-coordinate
Jun 5, 2024
Merged

Fix samtools merge --template-coordinate#2062
daviesrob merged 5 commits into
samtools:developfrom
nh13:bugfix/nh13-template-coordinate

Conversation

@nh13

@nh13 nh13 commented May 28, 2024

Copy link
Copy Markdown
Member

Cause: the wrong heap was being tested.

I added a test case from the original reported issue here: nf-core/fastquorum#52.

@nh13

nh13 commented May 28, 2024

Copy link
Copy Markdown
Member Author

Looks like there are detected memory leaks. I have not used LeakSanitizer before, so I'd be grateful for any help on how to run it on a Mac so I can find the line numbers and fix it.

@nh13

nh13 commented May 28, 2024

Copy link
Copy Markdown
Member Author

Looks like when I added a test case for samtools merge --template-coordinate, it found a memory leak. It looks like I've fixed it too.

@nh13

nh13 commented May 30, 2024

Copy link
Copy Markdown
Member Author

@daviesrob any chance you could review this given your past review on this feature? Also, how hard would it be to make a patch release for this bug fix?

nh13 added a commit to nh13/fastquorum that referenced this pull request May 30, 2024
There's a bug in `samtools merge --template-coordinate' that causes
the output not to be correctly sorted.  Here, we re-sort via fgbio
SortBam to ensure the output is properly sorted.

fgbio SortBam can be removed when the fix to the samtools bug is
released.

See:
samtools/samtools#2062
nh13 added a commit to nh13/fastquorum that referenced this pull request May 30, 2024
There's a bug in `samtools merge --template-coordinate' that causes
the output not to be correctly sorted.  Here, we re-sort via fgbio
SortBam to ensure the output is properly sorted.

fgbio SortBam can be removed when the fix to the samtools bug is
released.

See:
samtools/samtools#2062
nh13 added a commit to nh13/fastquorum that referenced this pull request May 30, 2024
There's a bug in `samtools merge --template-coordinate' that causes
the output not to be correctly sorted.  Here, we re-sort via fgbio
SortBam to ensure the output is properly sorted.

fgbio SortBam can be removed when the fix to the samtools bug is
released.

See:
samtools/samtools#2062
nh13 added a commit to nh13/fastquorum that referenced this pull request May 30, 2024
There's a bug in `samtools merge --template-coordinate' that causes
the output not to be correctly sorted.  Here, we re-sort via fgbio
SortBam to ensure the output is properly sorted.

fgbio SortBam can be removed when the fix to the samtools bug is
released.

See:
samtools/samtools#2062

@daviesrob daviesrob left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks OK apart from the issue noted. You could possibly shrink the test files a bit by removing the unneeded @SQ lines. Including the ones mentioned in XA and SA tags, you should only need these:

chr1
chr2
chr3
chr4
chr5
chr6
chr7
chr8
chr9
chr10
chr11
chr12
chr13
chr14
chr15
chr16
chr17
chr18
chr19
chr20
chr21
chr22
chrX
chrUn_KI270749v1
chr1_GL383519v1_alt
chr1_KI270761v1_alt
chr1_KI270765v1_alt
chr1_KI270892v1_alt
chr3_GL383526v1_alt
chr3_KI270778v1_alt
chr4_KI270790v1_alt
chr5_GL383532v1_alt
chr6_GL000250v2_alt
chr6_GL000251v2_alt
chr6_GL000252v2_alt
chr6_GL000253v2_alt
chr6_GL000254v2_alt
chr6_GL000255v2_alt
chr6_GL000256v2_alt
chr6_KI270801v1_alt
chr11_KI270829v1_alt
chr11_KI270903v1_alt
chr13_KI270843v1_alt
chr15_KI270848v1_alt
chr16_KI270728v1_random
chr16_KI270853v1_alt
chr17_KI270857v1_alt
chr19_GL949746v1_alt
chr19_GL949747v2_alt
chr19_GL949748v2_alt
chr19_GL949749v2_alt
chr19_GL949750v2_alt
chr19_GL949751v2_alt
chr19_GL949753v2_alt
chr19_KI270938v1_alt
chr22_KI270875v1_alt
chrUn_JTFH01000763v1_decoy

Removing the others would shrink the test files by over 200K each. You might be able to get away with removing some of the alignment records as well, as only the first one appears to move as a result of the bug fix...

Comment thread bam_sort.c
@daviesrob daviesrob self-assigned this May 30, 2024
@nh13 nh13 force-pushed the bugfix/nh13-template-coordinate branch from f54dd24 to cbeba40 Compare May 30, 2024 12:29
@nh13 nh13 force-pushed the bugfix/nh13-template-coordinate branch from cbeba40 to 1f4aa85 Compare May 30, 2024 12:33
@nh13

nh13 commented May 30, 2024

Copy link
Copy Markdown
Member Author

I've simplified the test SAMs, as well as added back the lines to clean up the keys.

@daviesrob

Copy link
Copy Markdown
Member

Thanks, this is looking better. One issue I've just noticed is that on template_coordinate_key() failure, it should jump to fail rather than mem_fail. It looks like all of its failure cases are due to missing or malformed tags rather than out of memory.

There are still some memory leaks on bam_merge_core2() failure, but I think they're our problem and not yours, so don't worry about them.

@nh13

nh13 commented Jun 4, 2024

Copy link
Copy Markdown
Member Author

@daviesrob thanks for the suggested improvement, done!

@daviesrob daviesrob merged commit fd7b33e into samtools:develop Jun 5, 2024
@daviesrob

Copy link
Copy Markdown
Member

Thanks for the updates, now all merged.

@nh13 nh13 deleted the bugfix/nh13-template-coordinate branch June 6, 2024 02:14
@nh13

nh13 commented Jun 6, 2024

Copy link
Copy Markdown
Member Author

Thank-you for all the suggestions and review!

@nh13

nh13 commented Jul 15, 2024

Copy link
Copy Markdown
Member Author

@daviesrob any chance for a release with this fix included? I'd like to update pipelines to use a released version :)

nh13 added a commit to nf-core/fastquorum that referenced this pull request Aug 2, 2024
There's a bug in `samtools merge --template-coordinate' that causes
the output not to be correctly sorted.  Here, we re-sort via fgbio
SortBam to ensure the output is properly sorted.

fgbio SortBam can be removed when the fix to the samtools bug is
released.

See:
samtools/samtools#2062
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants