Skip to content

bogart failed mergeOrphans canu v2.2 #1831

@wangjuan001

Description

@wangjuan001

Hi,
I ran into this bogart failure with canu 2.1, 2.1.1 and 2.2.

This is the canu.out

Found perl:
   /home/jwn2291/.conda/envs/plus/bin/perl
   This is perl 5, version 26, subversion 2 (v5.26.2) built for x86_64-linux-thread-multi

Found java:
   /home/jwn2291/.conda/envs/plus/bin/java
   openjdk version "11.0.1" 2018-10-16 LTS

Found canu:
   /projects/b1100/jwn2291/software/canu/build/bin/canu
   canu snapshot v2.2-development +69 changes (r10178 fcd7f964c882d3a346e6c0c227bab18dc5c97577)

-- canu snapshot v2.2-development +69 changes (r10178 fcd7f964c882d3a346e6c0c227bab18dc5c97577)
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
--   Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
--   Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
--   Genome Res. 2017 May;27(5):722-736.
--   http://doi.org/10.1101/gr.215087.116
-- 
-- Read and contig alignments during correction and consensus use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '11.0.1' (from '/home/jwn2291/.conda/envs/plus/bin/java') without -d64 support.
--
-- WARNING:
-- WARNING:  Failed to run gnuplot using command 'gnuplot'.
-- WARNING:  Plots will be disabled.
-- WARNING:
--
-- Detected 28 CPUs and 1007 gigabytes of memory.
-- Detected Slurm with 'sinfo' binary in /usr/bin/sinfo.
-- Detected Slurm with task IDs up to 19999 allowed.
-- 
-- Found   4 hosts with  32 cores and  187 GB memory under Slurm control.
-- Found  72 hosts with  28 cores and   92 GB memory under Slurm control.
-- Found  11 hosts with  52 cores and  187 GB memory under Slurm control.
-- Found   4 hosts with  24 cores and  125 GB memory under Slurm control.
-- Found 198 hosts with  24 cores and  124 GB memory under Slurm control.
-- Found   3 hosts with  28 cores and  502 GB memory under Slurm control.
-- Found   1 host  with  28 cores and  123 GB memory under Slurm control.
-- Found  44 hosts with  40 cores and  187 GB memory under Slurm control.
-- Found 324 hosts with  28 cores and  124 GB memory under Slurm control.
-- Found  49 hosts with  28 cores and  187 GB memory under Slurm control.
-- Found   2 hosts with  28 cores and 1006 GB memory under Slurm control.
--
--                         (tag)Threads
--                (tag)Memory         |
--        (tag)             |         |  algorithm
--        -------  ----------  --------  -----------------------------
-- Grid:  meryl     26.000 GB    7 CPUs  (k-mer counting)
-- Grid:  hap       16.000 GB   24 CPUs  (read-to-haplotype assignment)
-- Grid:  cormhap   17.000 GB    4 CPUs  (overlap detection with mhap)
-- Grid:  obtovl    24.000 GB    7 CPUs  (overlap detection)
-- Grid:  utgovl    24.000 GB    7 CPUs  (overlap detection)
-- Grid:  cor       24.000 GB    4 CPUs  (read correction)
-- Grid:  ovb        4.000 GB    1 CPU   (overlap store bucketizer)
-- Grid:  ovs       32.000 GB    1 CPU   (overlap store sorting)
-- Grid:  red       37.000 GB    8 CPUs  (read error detection)
-- Grid:  oea        8.000 GB    1 CPU   (overlap error adjustment)
-- Grid:  bat      187.000 GB   32 CPUs  (contig construction with bogart)
-- Grid:  cns        -.--- GB    8 CPUs  (consensus)
--
-- Found Nanopore reads in 'outRes.seqStore':
--   Libraries:
--     Nanopore:              1
--   Reads:
--     Raw:                   119833157302
--     Corrected:             89535458442
--     Corrected and Trimmed: 87792316061
--
--
-- Generating assembly 'outRes' in '/projects/jwn2291/WGS/':
--   genomeSize:
--     3500000000
--
--   Overlap Generation Limits:
--     corOvlErrorRate 0.3200 ( 32.00%)
--     obtOvlErrorRate 0.1200 ( 12.00%)
--     utgOvlErrorRate 0.1200 ( 12.00%)
--
--   Overlap Processing Limits:
--     corErrorRate    0.5000 ( 50.00%)
--     obtErrorRate    0.1200 ( 12.00%)
--     utgErrorRate    0.1200 ( 12.00%)
--     cnsErrorRate    0.2000 ( 20.00%)
--
--   Stages to run:
--     assemble corrected and trimmed reads.
--
--
-- Corrected reads exist in outRes.seqStore.
--
-- Trimmed reads exist in outRes.seqStore.
--
-- BEGIN ASSEMBLY
--
-- Bogart failed, tried 2 times, giving up.
--

ABORT:
ABORT: canu snapshot v2.2-development +69 changes (r10178 fcd7f964c882d3a346e6c0c227bab18dc5c97577)
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.
ABORT:
ABORT: Disk space available:  39405.75 GB
ABORT:
ABORT: Last 50 lines of the relevant log file (unitigging/4-unitigger/unitigger.err):
ABORT:
ABORT:   optimizePositions()--     changed:   1834133 reads
ABORT:   optimizePositions()--   Recomputing positions, iteration 4, with 32 threads.
ABORT:   optimizePositions()--     Reset zero.
ABORT:   optimizePositions()--     Checking convergence.
ABORT:   optimizePositions()--     converged: 1881632 reads
ABORT:   optimizePositions()--     changed:   1821626 reads
ABORT:   optimizePositions()--   Recomputing positions, iteration 5, with 32 threads.
ABORT:   optimizePositions()--     Reset zero.
ABORT:   optimizePositions()--     Checking convergence.
ABORT:   optimizePositions()--     converged: 1901495 reads
ABORT:   optimizePositions()--     changed:   1801763 reads
ABORT:   optimizePositions()--   Expanding short reads with 32 threads.
ABORT:   optimizePositions()--   Updating positions.
ABORT:   optimizePositions()--   Finished.
ABORT:   splitDiscontinuous()-- Tested 2789 tigs, split 9 tigs into 21 new tigs.
ABORT:   
ABORT:   ==> MERGE ORPHANS.
ABORT:   
ABORT:   computeErrorProfiles()-- Computing error profiles for 2819 tigs, with 32 threads.
ABORT:   computeErrorProfiles()-- Finished.
ABORT:   
ABORT:   findPotentialOrphans()-- working on 2819 tigs.
ABORT:   findPotentialOrphans()-- found 1223 potential orphans.
ABORT:   mergeOrphans()-- flagged     412        bubble tigs with 41679 reads
ABORT:   mergeOrphans()-- placed      133 unique orphan tigs with 5604 reads
ABORT:   mergeOrphans()-- shattered    56 repeat orphan tigs with 3244 reads
ABORT:   mergeOrphans()-- ignored      22               tigs with 3559 reads; failed to place
ABORT:   mergeOrphans()--
ABORT:   
ABORT:   ==> MARK SIMPLE BUBBLES.
ABORT:       using 0.010000 user-specified threshold
ABORT:   
ABORT:   
ABORT:   findPotentialOrphans()-- working on 2819 tigs.
ABORT:   read 2466988 at 1456192 1522940 olap to read 3177577 hangs 66862 66762 -> coords 1523054 1522940
ABORT:   bogart: bogart/AS_BAT_MergeOrphans.C:258: void findPotentialOrphans(TigVector&, BubTargetList&, bool): Assertion `mincoord < maxcoord' failed.
ABORT:   
ABORT:   Failed with 'Aborted'; backtrace (libbacktrace):
ABORT:   utility/src/utility/system-stackTrace.C::83 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
ABORT:   (null)::0 in (null)()
ABORT:   (null)::0 in (null)()
ABORT:   (null)::0 in (null)()
ABORT:   (null)::0 in (null)()
ABORT:   (null)::0 in (null)()
ABORT:   bogart/AS_BAT_MergeOrphans.C::258 in _Z20findPotentialOrphansR9TigVectorRSt3mapIjSt6vectorIjSaIjEESt4lessIjESaISt4pairIKjS4_EEEb()
ABORT:   bogart/AS_BAT_MergeOrphans.C::896 in _Z12mergeOrphansR9TigVectorddb()
ABORT:   bogart/bogart.C::572 in main()
ABORT:   (null)::0 in (null)()
ABORT:   (null)::0 in (null)()
ABORT:   (null)::0 in (null)()
ABORT:

I then tried to manually changed the setting of unitigger.jobSubmit-01.sh as --cpus-per-task=20 --mem-per-cpu=5000m
and unitigger.sh -threads 20 -M 50

And here is unitigger.err


==> PARAMETERS.

Resources:
  Memory                50 GB
  Compute Threads       20 (command line)

Lengths:
  Minimum read          0 bases
  Maximum read          4294967295 bases
  Minimum overlap       500 bases

Overlap Error Rates:
  Graph                 0.120 (12.000%)
  Max                   0.120 (12.000%)

Deviations:
  Graph                 12.000
  Bubble                1.000
  Repeat                1.000

Similarity Thresholds:
  Graph                 0.000
  Bubble                0.010
  Repeat                0.010

Edge Confusion:
  Absolute              2500
  Percent               15.0000

Unitig Construction:
  Minimum intersection  500 bases
  Maxiumum placements   2 positions

Debugging Enabled:
  (none)

==> LOADING AND FILTERING OVERLAPS.

ReadInfo()-- Found     2255634 reads.

OverlapCache()-- limited to 51200MB memory (user supplied).

OverlapCache()--      28MB for read data.
OverlapCache()--     113MB for best edges.
OverlapCache()--     367MB for tigs.
OverlapCache()--      98MB for tigs - read layouts.
OverlapCache()--     141MB for tigs - error profiles.
OverlapCache()--   12800MB for tigs - error profile overlaps.
OverlapCache()--       0MB for other processes.
OverlapCache()-- ---------
OverlapCache()--   13619MB for data structures (sum of above).
OverlapCache()-- ---------
OverlapCache()--      70MB for overlap store structure.
OverlapCache()--   37510MB for overlap data.
OverlapCache()-- ---------
OverlapCache()--   51200MB allowed.
OverlapCache()--
OverlapCache()-- Retain at least 50 overlaps/read, based on 25.08x coverage.
OverlapCache()-- Initial guess at 663 overlaps/read.
OverlapCache()--
OverlapCache()-- Adjusting for sparse overlaps.
OverlapCache()--
OverlapCache()--               reads loading olaps          olaps               memory
OverlapCache()--   olaps/read       all      some          loaded                 free
OverlapCache()--   ----------   -------   -------     ----------- -------     --------
OverlapCache()--          663   3427530    275727       420549242  55.54%      31092 MB
OverlapCache()--         8053   3702880       377       756931576  99.96%      25960 MB
OverlapCache()--      4520850   3703257         0       757247030 100.00%      25955 MB
OverlapCache()--
OverlapCache()-- Loading overlaps.
OverlapCache()--
OverlapCache()--          read from store           saved in cache
OverlapCache()--   ------------ ---------   ------------ ---------
OverlapCache()--       14361380 (001.90%)       14317854 (001.89%)
OverlapCache()--       28438923 (003.76%)       28353453 (003.74%)
OverlapCache()--       42483008 (005.61%)       42352424 (005.59%)
OverlapCache()--       82696369 (010.92%)       82482188 (010.89%)
OverlapCache()--      105186365 (013.89%)      104901726 (013.85%)
OverlapCache()--      126035633 (016.64%)      125684348 (016.60%)
OverlapCache()--      154741070 (020.43%)      154329339 (020.38%)
OverlapCache()--      169727901 (022.41%)      169274904 (022.35%)
OverlapCache()--      182688399 (024.13%)      182199416 (024.06%)
OverlapCache()--      197035705 (026.02%)      196501103 (025.95%)
OverlapCache()--      219480679 (028.98%)      218884684 (028.91%)
OverlapCache()--      240180077 (031.72%)      239533812 (031.63%)
OverlapCache()--      262085903 (034.61%)      261375907 (034.52%)
OverlapCache()--      282764695 (037.34%)      281994298 (037.24%)
OverlapCache()--      303043723 (040.02%)      302209678 (039.91%)
OverlapCache()--      323045538 (042.66%)      322154019 (042.54%)
OverlapCache()--      343988533 (045.43%)      343040420 (045.30%)
OverlapCache()--      365895219 (048.32%)      364867615 (048.18%)
OverlapCache()--      386396447 (051.03%)      385299953 (050.88%)
OverlapCache()--      407072282 (053.76%)      405904062 (053.60%)
OverlapCache()--      427654987 (056.47%)      426431384 (056.31%)
OverlapCache()--      447017954 (059.03%)      445729731 (058.86%)
OverlapCache()--      467291293 (061.71%)      465944296 (061.53%)
OverlapCache()--      487903484 (064.43%)      486498873 (064.25%)
OverlapCache()--      509134188 (067.23%)      507664094 (067.04%)
OverlapCache()--      529792366 (069.96%)      528267534 (069.76%)
OverlapCache()--      549822616 (072.61%)      548234784 (072.40%)
OverlapCache()--      571028533 (075.41%)      569362466 (075.19%)
OverlapCache()--      591717466 (078.14%)      589990417 (077.91%)
OverlapCache()--      612761514 (080.92%)      610980694 (080.68%)
OverlapCache()--      633950053 (083.72%)      632109565 (083.47%)
OverlapCache()--      655697485 (086.59%)      653795504 (086.34%)
OverlapCache()--      674893327 (089.12%)      672931803 (088.87%)
OverlapCache()--      694320297 (091.69%)      692296243 (091.42%)
OverlapCache()--      714728615 (094.39%)      712639352 (094.11%)
OverlapCache()--      735946769 (097.19%)      733787946 (096.90%)
OverlapCache()--      756588565 (099.91%)      754369338 (099.62%)
OverlapCache()--   ------------ ---------   ------------ ---------
OverlapCache()--      757247030 (100.00%)      755025624 (099.71%)
OverlapCache()--
OverlapCache()-- Ignored 1791242 duplicate overlaps.
OverlapCache()--
OverlapCache()-- Symmetrizing overlaps.
OverlapCache()--   Finding missing twins.
OverlapCache()--   In 755025624 overlaps:
OverlapCache()--     Found 116121601 overlaps with non-symmetric error rates.
OverlapCache()--     Found 108438 overlaps with missing twins.
OverlapCache()--     Dropped 0 weak missing-twin overlaps.
OverlapCache()--   Shifting overlaps.
OverlapCache()--   Adding missing twins.
OverlapCache()--   Sorting overlaps.
OverlapCache()--   Checking overlap symmetry.
OverlapCache()--   Finished.

BestOverlapGraph()-- Computing Best Overlap Graph.
BestOverlapGraph()-- Allocating best edges (113MB).
BestOverlapGraph()-- Filtering high error edges.
BestOverlapGraph()--   Ignore overlaps with more than 2.881350% error.
BestOverlapGraph()-- Filtering reads with a gap in overlap coverage.
BestOverlapGraph()--   89720 reads removed.
BestOverlapGraph()-- Filtering reads with lopsided best edges (more than 25.00% different).
BestOverlapGraph()--   1815 reads have lopsided edges.
BestOverlapGraph()-- Filtering spur reads.
BestOverlapGraph()--   Initial          51565 terminal  spur reads -    53357/52880    5'/3' spur path reads.
BestOverlapGraph()--   Iteration 1 -    51565 terminal  spur reads -    53230/52759    5'/3' spur path reads -      449/449      edges changed to avoid spur path.
BestOverlapGraph()--   Iteration 2 -    51565 terminal  spur reads -    53201/52723    5'/3' spur path reads -       28/29       edges changed to avoid spur path.
BestOverlapGraph()--   Iteration 3 -    51565 terminal  spur reads -    53195/52720    5'/3' spur path reads -        3/2        edges changed to avoid spur path.
BestOverlapGraph()--   Iteration 4 -    51565 terminal  spur reads -    53195/52720    5'/3' spur path reads -        0/0        edges changed to avoid spur path.
BestOverlapGraph()--   Final            57416 confirmed spur reads -    53195/52720    5'/3' spur path reads.

==> BUILDING GREEDY TIGS.

breakSingletonTigs()-- Removed 20 singleton tigs; reads are now unplaced.
optimizePositions()-- Optimizing read positions for 3703258 reads in 2798 tigs, with 20 threads.
optimizePositions()--   Allocating scratch space for 3703258 reads (115726 KB).
optimizePositions()--   Initializing positions with 20 threads.
optimizePositions()--   Recomputing positions, iteration 1, with 20 threads.
optimizePositions()--     Reset zero.
optimizePositions()--     Checking convergence.
optimizePositions()--     converged: 2245351 reads
optimizePositions()--     changed:   1457907 reads
optimizePositions()--   Recomputing positions, iteration 2, with 20 threads.
optimizePositions()--     Reset zero.
optimizePositions()--     Checking convergence.
optimizePositions()--     converged: 2247763 reads
optimizePositions()--     changed:   1455495 reads
optimizePositions()--   Recomputing positions, iteration 3, with 20 threads.
optimizePositions()--     Reset zero.
optimizePositions()--     Checking convergence.
optimizePositions()--     converged: 2255102 reads
optimizePositions()--     changed:   1448156 reads
optimizePositions()--   Recomputing positions, iteration 4, with 20 threads.
optimizePositions()--     Reset zero.
optimizePositions()--     Checking convergence.
optimizePositions()--     converged: 2255407 reads
optimizePositions()--     changed:   1447851 reads
optimizePositions()--   Recomputing positions, iteration 5, with 20 threads.
optimizePositions()--     Reset zero.
optimizePositions()--     Checking convergence.
optimizePositions()--     converged: 2255487 reads
optimizePositions()--     changed:   1447771 reads
optimizePositions()--   Expanding short reads with 20 threads.
optimizePositions()--   Updating positions.
optimizePositions()--   Finished.
splitDiscontinuous()-- Tested 2777 tigs, split none.
detectSpur() done.
tested         1312
nEdges      5'  433   3'  438
nPotential       55        64
nVerified         0         0

==> PLACE CONTAINED READS.

computeErrorProfiles()-- Computing error profiles for 2798 tigs, with 20 threads.
computeErrorProfiles()-- Finished.

placeContains()-- placing 1934373 contained and 1590376 unplaced reads, with 20 threads.
placeContains()-- Placed 1920072 contained reads and 84672 unplaced reads.
placeContains()-- Failed to place 14301 contained reads (too high error suspected) and 1505704 unplaced reads (lack of overlaps suspected).
optimizePositions()-- Optimizing read positions for 3703258 reads in 2798 tigs, with 20 threads.
optimizePositions()--   Allocating scratch space for 3703258 reads (115726 KB).
optimizePositions()--   Initializing positions with 20 threads.
optimizePositions()--   Recomputing positions, iteration 1, with 20 threads.
optimizePositions()--     Reset zero.
optimizePositions()--     Checking convergence.
optimizePositions()--     converged: 1232244 reads
optimizePositions()--     changed:   2471014 reads
optimizePositions()--   Recomputing positions, iteration 2, with 20 threads.
optimizePositions()--     Reset zero.
optimizePositions()--     Checking convergence.
optimizePositions()--     converged: 1848999 reads
optimizePositions()--     changed:   1854259 reads
optimizePositions()--   Recomputing positions, iteration 3, with 20 threads.
optimizePositions()--     Reset zero.
optimizePositions()--     Checking convergence.
optimizePositions()--     converged: 1869125 reads
optimizePositions()--     changed:   1834133 reads
optimizePositions()--   Recomputing positions, iteration 4, with 20 threads.
optimizePositions()--     Reset zero.
optimizePositions()--     Checking convergence.
optimizePositions()--     converged: 1881632 reads
optimizePositions()--     changed:   1821626 reads
optimizePositions()--   Recomputing positions, iteration 5, with 20 threads.
optimizePositions()--     Reset zero.
optimizePositions()--     Checking convergence.
optimizePositions()--     converged: 1901495 reads
optimizePositions()--     changed:   1801763 reads
optimizePositions()--   Expanding short reads with 20 threads.
optimizePositions()--   Updating positions.
optimizePositions()--   Finished.
splitDiscontinuous()-- Tested 2789 tigs, split 9 tigs into 21 new tigs.

==> MERGE ORPHANS.

computeErrorProfiles()-- Computing error profiles for 2819 tigs, with 20 threads.
computeErrorProfiles()-- Finished.

findPotentialOrphans()-- working on 2819 tigs.
findPotentialOrphans()-- found 1223 potential orphans.
mergeOrphans()-- flagged     412        bubble tigs with 41679 reads
mergeOrphans()-- placed      133 unique orphan tigs with 5604 reads
mergeOrphans()-- shattered    56 repeat orphan tigs with 3244 reads
mergeOrphans()-- ignored      22               tigs with 3559 reads; failed to place
mergeOrphans()--

==> MARK SIMPLE BUBBLES.
    using 0.010000 user-specified threshold


findPotentialOrphans()-- working on 2819 tigs.
read 2466988 at 1456192 1522940 olap to read 3177577 hangs 66862 66762 -> coords 1523054 1522940
bogart: bogart/AS_BAT_MergeOrphans.C:258: void findPotentialOrphans(TigVector&, BubTargetList&, bool): Assertion `mincoord < maxcoord' failed.

Failed with 'Aborted'; backtrace (libbacktrace):
utility/src/utility/system-stackTrace.C::83 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
bogart/AS_BAT_MergeOrphans.C::258 in _Z20findPotentialOrphansR9TigVectorRSt3mapIjSt6vectorIjSaIjEESt4lessIjESaISt4pairIKjS4_EEEb()
bogart/AS_BAT_MergeOrphans.C::896 in _Z12mergeOrphansR9TigVectorddb()
bogart/bogart.C::572 in main()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions