Hi,
I ran into this bogart failure with canu 2.1, 2.1.1 and 2.2.
This is the canu.out
Found perl:
/home/jwn2291/.conda/envs/plus/bin/perl
This is perl 5, version 26, subversion 2 (v5.26.2) built for x86_64-linux-thread-multi
Found java:
/home/jwn2291/.conda/envs/plus/bin/java
openjdk version "11.0.1" 2018-10-16 LTS
Found canu:
/projects/b1100/jwn2291/software/canu/build/bin/canu
canu snapshot v2.2-development +69 changes (r10178 fcd7f964c882d3a346e6c0c227bab18dc5c97577)
-- canu snapshot v2.2-development +69 changes (r10178 fcd7f964c882d3a346e6c0c227bab18dc5c97577)
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
-- Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
-- Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
-- Genome Res. 2017 May;27(5):722-736.
-- http://doi.org/10.1101/gr.215087.116
--
-- Read and contig alignments during correction and consensus use:
-- Šošic M, Šikic M.
-- Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
-- Bioinformatics. 2017 May 1;33(9):1394-1395.
-- http://doi.org/10.1093/bioinformatics/btw753
--
-- Overlaps are generated using:
-- Berlin K, et al.
-- Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
-- Nat Biotechnol. 2015 Jun;33(6):623-30.
-- http://doi.org/10.1038/nbt.3238
--
-- Myers EW, et al.
-- A Whole-Genome Assembly of Drosophila.
-- Science. 2000 Mar 24;287(5461):2196-204.
-- http://doi.org/10.1126/science.287.5461.2196
--
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
-- Chin CS, et al.
-- Phased diploid genome assembly with single-molecule real-time sequencing.
-- Nat Methods. 2016 Dec;13(12):1050-1054.
-- http://doi.org/10.1038/nmeth.4035
--
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
-- Chin CS, et al.
-- Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
-- Nat Methods. 2013 Jun;10(6):563-9
-- http://doi.org/10.1038/nmeth.2474
--
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '11.0.1' (from '/home/jwn2291/.conda/envs/plus/bin/java') without -d64 support.
--
-- WARNING:
-- WARNING: Failed to run gnuplot using command 'gnuplot'.
-- WARNING: Plots will be disabled.
-- WARNING:
--
-- Detected 28 CPUs and 1007 gigabytes of memory.
-- Detected Slurm with 'sinfo' binary in /usr/bin/sinfo.
-- Detected Slurm with task IDs up to 19999 allowed.
--
-- Found 4 hosts with 32 cores and 187 GB memory under Slurm control.
-- Found 72 hosts with 28 cores and 92 GB memory under Slurm control.
-- Found 11 hosts with 52 cores and 187 GB memory under Slurm control.
-- Found 4 hosts with 24 cores and 125 GB memory under Slurm control.
-- Found 198 hosts with 24 cores and 124 GB memory under Slurm control.
-- Found 3 hosts with 28 cores and 502 GB memory under Slurm control.
-- Found 1 host with 28 cores and 123 GB memory under Slurm control.
-- Found 44 hosts with 40 cores and 187 GB memory under Slurm control.
-- Found 324 hosts with 28 cores and 124 GB memory under Slurm control.
-- Found 49 hosts with 28 cores and 187 GB memory under Slurm control.
-- Found 2 hosts with 28 cores and 1006 GB memory under Slurm control.
--
-- (tag)Threads
-- (tag)Memory |
-- (tag) | | algorithm
-- ------- ---------- -------- -----------------------------
-- Grid: meryl 26.000 GB 7 CPUs (k-mer counting)
-- Grid: hap 16.000 GB 24 CPUs (read-to-haplotype assignment)
-- Grid: cormhap 17.000 GB 4 CPUs (overlap detection with mhap)
-- Grid: obtovl 24.000 GB 7 CPUs (overlap detection)
-- Grid: utgovl 24.000 GB 7 CPUs (overlap detection)
-- Grid: cor 24.000 GB 4 CPUs (read correction)
-- Grid: ovb 4.000 GB 1 CPU (overlap store bucketizer)
-- Grid: ovs 32.000 GB 1 CPU (overlap store sorting)
-- Grid: red 37.000 GB 8 CPUs (read error detection)
-- Grid: oea 8.000 GB 1 CPU (overlap error adjustment)
-- Grid: bat 187.000 GB 32 CPUs (contig construction with bogart)
-- Grid: cns -.--- GB 8 CPUs (consensus)
--
-- Found Nanopore reads in 'outRes.seqStore':
-- Libraries:
-- Nanopore: 1
-- Reads:
-- Raw: 119833157302
-- Corrected: 89535458442
-- Corrected and Trimmed: 87792316061
--
--
-- Generating assembly 'outRes' in '/projects/jwn2291/WGS/':
-- genomeSize:
-- 3500000000
--
-- Overlap Generation Limits:
-- corOvlErrorRate 0.3200 ( 32.00%)
-- obtOvlErrorRate 0.1200 ( 12.00%)
-- utgOvlErrorRate 0.1200 ( 12.00%)
--
-- Overlap Processing Limits:
-- corErrorRate 0.5000 ( 50.00%)
-- obtErrorRate 0.1200 ( 12.00%)
-- utgErrorRate 0.1200 ( 12.00%)
-- cnsErrorRate 0.2000 ( 20.00%)
--
-- Stages to run:
-- assemble corrected and trimmed reads.
--
--
-- Corrected reads exist in outRes.seqStore.
--
-- Trimmed reads exist in outRes.seqStore.
--
-- BEGIN ASSEMBLY
--
-- Bogart failed, tried 2 times, giving up.
--
ABORT:
ABORT: canu snapshot v2.2-development +69 changes (r10178 fcd7f964c882d3a346e6c0c227bab18dc5c97577)
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting. If that doesn't work, ask for help.
ABORT:
ABORT: Disk space available: 39405.75 GB
ABORT:
ABORT: Last 50 lines of the relevant log file (unitigging/4-unitigger/unitigger.err):
ABORT:
ABORT: optimizePositions()-- changed: 1834133 reads
ABORT: optimizePositions()-- Recomputing positions, iteration 4, with 32 threads.
ABORT: optimizePositions()-- Reset zero.
ABORT: optimizePositions()-- Checking convergence.
ABORT: optimizePositions()-- converged: 1881632 reads
ABORT: optimizePositions()-- changed: 1821626 reads
ABORT: optimizePositions()-- Recomputing positions, iteration 5, with 32 threads.
ABORT: optimizePositions()-- Reset zero.
ABORT: optimizePositions()-- Checking convergence.
ABORT: optimizePositions()-- converged: 1901495 reads
ABORT: optimizePositions()-- changed: 1801763 reads
ABORT: optimizePositions()-- Expanding short reads with 32 threads.
ABORT: optimizePositions()-- Updating positions.
ABORT: optimizePositions()-- Finished.
ABORT: splitDiscontinuous()-- Tested 2789 tigs, split 9 tigs into 21 new tigs.
ABORT:
ABORT: ==> MERGE ORPHANS.
ABORT:
ABORT: computeErrorProfiles()-- Computing error profiles for 2819 tigs, with 32 threads.
ABORT: computeErrorProfiles()-- Finished.
ABORT:
ABORT: findPotentialOrphans()-- working on 2819 tigs.
ABORT: findPotentialOrphans()-- found 1223 potential orphans.
ABORT: mergeOrphans()-- flagged 412 bubble tigs with 41679 reads
ABORT: mergeOrphans()-- placed 133 unique orphan tigs with 5604 reads
ABORT: mergeOrphans()-- shattered 56 repeat orphan tigs with 3244 reads
ABORT: mergeOrphans()-- ignored 22 tigs with 3559 reads; failed to place
ABORT: mergeOrphans()--
ABORT:
ABORT: ==> MARK SIMPLE BUBBLES.
ABORT: using 0.010000 user-specified threshold
ABORT:
ABORT:
ABORT: findPotentialOrphans()-- working on 2819 tigs.
ABORT: read 2466988 at 1456192 1522940 olap to read 3177577 hangs 66862 66762 -> coords 1523054 1522940
ABORT: bogart: bogart/AS_BAT_MergeOrphans.C:258: void findPotentialOrphans(TigVector&, BubTargetList&, bool): Assertion `mincoord < maxcoord' failed.
ABORT:
ABORT: Failed with 'Aborted'; backtrace (libbacktrace):
ABORT: utility/src/utility/system-stackTrace.C::83 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
ABORT: (null)::0 in (null)()
ABORT: (null)::0 in (null)()
ABORT: (null)::0 in (null)()
ABORT: (null)::0 in (null)()
ABORT: (null)::0 in (null)()
ABORT: bogart/AS_BAT_MergeOrphans.C::258 in _Z20findPotentialOrphansR9TigVectorRSt3mapIjSt6vectorIjSaIjEESt4lessIjESaISt4pairIKjS4_EEEb()
ABORT: bogart/AS_BAT_MergeOrphans.C::896 in _Z12mergeOrphansR9TigVectorddb()
ABORT: bogart/bogart.C::572 in main()
ABORT: (null)::0 in (null)()
ABORT: (null)::0 in (null)()
ABORT: (null)::0 in (null)()
ABORT:
I then tried to manually changed the setting of unitigger.jobSubmit-01.sh as --cpus-per-task=20 --mem-per-cpu=5000m
and unitigger.sh -threads 20 -M 50
And here is unitigger.err
==> PARAMETERS.
Resources:
Memory 50 GB
Compute Threads 20 (command line)
Lengths:
Minimum read 0 bases
Maximum read 4294967295 bases
Minimum overlap 500 bases
Overlap Error Rates:
Graph 0.120 (12.000%)
Max 0.120 (12.000%)
Deviations:
Graph 12.000
Bubble 1.000
Repeat 1.000
Similarity Thresholds:
Graph 0.000
Bubble 0.010
Repeat 0.010
Edge Confusion:
Absolute 2500
Percent 15.0000
Unitig Construction:
Minimum intersection 500 bases
Maxiumum placements 2 positions
Debugging Enabled:
(none)
==> LOADING AND FILTERING OVERLAPS.
ReadInfo()-- Found 2255634 reads.
OverlapCache()-- limited to 51200MB memory (user supplied).
OverlapCache()-- 28MB for read data.
OverlapCache()-- 113MB for best edges.
OverlapCache()-- 367MB for tigs.
OverlapCache()-- 98MB for tigs - read layouts.
OverlapCache()-- 141MB for tigs - error profiles.
OverlapCache()-- 12800MB for tigs - error profile overlaps.
OverlapCache()-- 0MB for other processes.
OverlapCache()-- ---------
OverlapCache()-- 13619MB for data structures (sum of above).
OverlapCache()-- ---------
OverlapCache()-- 70MB for overlap store structure.
OverlapCache()-- 37510MB for overlap data.
OverlapCache()-- ---------
OverlapCache()-- 51200MB allowed.
OverlapCache()--
OverlapCache()-- Retain at least 50 overlaps/read, based on 25.08x coverage.
OverlapCache()-- Initial guess at 663 overlaps/read.
OverlapCache()--
OverlapCache()-- Adjusting for sparse overlaps.
OverlapCache()--
OverlapCache()-- reads loading olaps olaps memory
OverlapCache()-- olaps/read all some loaded free
OverlapCache()-- ---------- ------- ------- ----------- ------- --------
OverlapCache()-- 663 3427530 275727 420549242 55.54% 31092 MB
OverlapCache()-- 8053 3702880 377 756931576 99.96% 25960 MB
OverlapCache()-- 4520850 3703257 0 757247030 100.00% 25955 MB
OverlapCache()--
OverlapCache()-- Loading overlaps.
OverlapCache()--
OverlapCache()-- read from store saved in cache
OverlapCache()-- ------------ --------- ------------ ---------
OverlapCache()-- 14361380 (001.90%) 14317854 (001.89%)
OverlapCache()-- 28438923 (003.76%) 28353453 (003.74%)
OverlapCache()-- 42483008 (005.61%) 42352424 (005.59%)
OverlapCache()-- 82696369 (010.92%) 82482188 (010.89%)
OverlapCache()-- 105186365 (013.89%) 104901726 (013.85%)
OverlapCache()-- 126035633 (016.64%) 125684348 (016.60%)
OverlapCache()-- 154741070 (020.43%) 154329339 (020.38%)
OverlapCache()-- 169727901 (022.41%) 169274904 (022.35%)
OverlapCache()-- 182688399 (024.13%) 182199416 (024.06%)
OverlapCache()-- 197035705 (026.02%) 196501103 (025.95%)
OverlapCache()-- 219480679 (028.98%) 218884684 (028.91%)
OverlapCache()-- 240180077 (031.72%) 239533812 (031.63%)
OverlapCache()-- 262085903 (034.61%) 261375907 (034.52%)
OverlapCache()-- 282764695 (037.34%) 281994298 (037.24%)
OverlapCache()-- 303043723 (040.02%) 302209678 (039.91%)
OverlapCache()-- 323045538 (042.66%) 322154019 (042.54%)
OverlapCache()-- 343988533 (045.43%) 343040420 (045.30%)
OverlapCache()-- 365895219 (048.32%) 364867615 (048.18%)
OverlapCache()-- 386396447 (051.03%) 385299953 (050.88%)
OverlapCache()-- 407072282 (053.76%) 405904062 (053.60%)
OverlapCache()-- 427654987 (056.47%) 426431384 (056.31%)
OverlapCache()-- 447017954 (059.03%) 445729731 (058.86%)
OverlapCache()-- 467291293 (061.71%) 465944296 (061.53%)
OverlapCache()-- 487903484 (064.43%) 486498873 (064.25%)
OverlapCache()-- 509134188 (067.23%) 507664094 (067.04%)
OverlapCache()-- 529792366 (069.96%) 528267534 (069.76%)
OverlapCache()-- 549822616 (072.61%) 548234784 (072.40%)
OverlapCache()-- 571028533 (075.41%) 569362466 (075.19%)
OverlapCache()-- 591717466 (078.14%) 589990417 (077.91%)
OverlapCache()-- 612761514 (080.92%) 610980694 (080.68%)
OverlapCache()-- 633950053 (083.72%) 632109565 (083.47%)
OverlapCache()-- 655697485 (086.59%) 653795504 (086.34%)
OverlapCache()-- 674893327 (089.12%) 672931803 (088.87%)
OverlapCache()-- 694320297 (091.69%) 692296243 (091.42%)
OverlapCache()-- 714728615 (094.39%) 712639352 (094.11%)
OverlapCache()-- 735946769 (097.19%) 733787946 (096.90%)
OverlapCache()-- 756588565 (099.91%) 754369338 (099.62%)
OverlapCache()-- ------------ --------- ------------ ---------
OverlapCache()-- 757247030 (100.00%) 755025624 (099.71%)
OverlapCache()--
OverlapCache()-- Ignored 1791242 duplicate overlaps.
OverlapCache()--
OverlapCache()-- Symmetrizing overlaps.
OverlapCache()-- Finding missing twins.
OverlapCache()-- In 755025624 overlaps:
OverlapCache()-- Found 116121601 overlaps with non-symmetric error rates.
OverlapCache()-- Found 108438 overlaps with missing twins.
OverlapCache()-- Dropped 0 weak missing-twin overlaps.
OverlapCache()-- Shifting overlaps.
OverlapCache()-- Adding missing twins.
OverlapCache()-- Sorting overlaps.
OverlapCache()-- Checking overlap symmetry.
OverlapCache()-- Finished.
BestOverlapGraph()-- Computing Best Overlap Graph.
BestOverlapGraph()-- Allocating best edges (113MB).
BestOverlapGraph()-- Filtering high error edges.
BestOverlapGraph()-- Ignore overlaps with more than 2.881350% error.
BestOverlapGraph()-- Filtering reads with a gap in overlap coverage.
BestOverlapGraph()-- 89720 reads removed.
BestOverlapGraph()-- Filtering reads with lopsided best edges (more than 25.00% different).
BestOverlapGraph()-- 1815 reads have lopsided edges.
BestOverlapGraph()-- Filtering spur reads.
BestOverlapGraph()-- Initial 51565 terminal spur reads - 53357/52880 5'/3' spur path reads.
BestOverlapGraph()-- Iteration 1 - 51565 terminal spur reads - 53230/52759 5'/3' spur path reads - 449/449 edges changed to avoid spur path.
BestOverlapGraph()-- Iteration 2 - 51565 terminal spur reads - 53201/52723 5'/3' spur path reads - 28/29 edges changed to avoid spur path.
BestOverlapGraph()-- Iteration 3 - 51565 terminal spur reads - 53195/52720 5'/3' spur path reads - 3/2 edges changed to avoid spur path.
BestOverlapGraph()-- Iteration 4 - 51565 terminal spur reads - 53195/52720 5'/3' spur path reads - 0/0 edges changed to avoid spur path.
BestOverlapGraph()-- Final 57416 confirmed spur reads - 53195/52720 5'/3' spur path reads.
==> BUILDING GREEDY TIGS.
breakSingletonTigs()-- Removed 20 singleton tigs; reads are now unplaced.
optimizePositions()-- Optimizing read positions for 3703258 reads in 2798 tigs, with 20 threads.
optimizePositions()-- Allocating scratch space for 3703258 reads (115726 KB).
optimizePositions()-- Initializing positions with 20 threads.
optimizePositions()-- Recomputing positions, iteration 1, with 20 threads.
optimizePositions()-- Reset zero.
optimizePositions()-- Checking convergence.
optimizePositions()-- converged: 2245351 reads
optimizePositions()-- changed: 1457907 reads
optimizePositions()-- Recomputing positions, iteration 2, with 20 threads.
optimizePositions()-- Reset zero.
optimizePositions()-- Checking convergence.
optimizePositions()-- converged: 2247763 reads
optimizePositions()-- changed: 1455495 reads
optimizePositions()-- Recomputing positions, iteration 3, with 20 threads.
optimizePositions()-- Reset zero.
optimizePositions()-- Checking convergence.
optimizePositions()-- converged: 2255102 reads
optimizePositions()-- changed: 1448156 reads
optimizePositions()-- Recomputing positions, iteration 4, with 20 threads.
optimizePositions()-- Reset zero.
optimizePositions()-- Checking convergence.
optimizePositions()-- converged: 2255407 reads
optimizePositions()-- changed: 1447851 reads
optimizePositions()-- Recomputing positions, iteration 5, with 20 threads.
optimizePositions()-- Reset zero.
optimizePositions()-- Checking convergence.
optimizePositions()-- converged: 2255487 reads
optimizePositions()-- changed: 1447771 reads
optimizePositions()-- Expanding short reads with 20 threads.
optimizePositions()-- Updating positions.
optimizePositions()-- Finished.
splitDiscontinuous()-- Tested 2777 tigs, split none.
detectSpur() done.
tested 1312
nEdges 5' 433 3' 438
nPotential 55 64
nVerified 0 0
==> PLACE CONTAINED READS.
computeErrorProfiles()-- Computing error profiles for 2798 tigs, with 20 threads.
computeErrorProfiles()-- Finished.
placeContains()-- placing 1934373 contained and 1590376 unplaced reads, with 20 threads.
placeContains()-- Placed 1920072 contained reads and 84672 unplaced reads.
placeContains()-- Failed to place 14301 contained reads (too high error suspected) and 1505704 unplaced reads (lack of overlaps suspected).
optimizePositions()-- Optimizing read positions for 3703258 reads in 2798 tigs, with 20 threads.
optimizePositions()-- Allocating scratch space for 3703258 reads (115726 KB).
optimizePositions()-- Initializing positions with 20 threads.
optimizePositions()-- Recomputing positions, iteration 1, with 20 threads.
optimizePositions()-- Reset zero.
optimizePositions()-- Checking convergence.
optimizePositions()-- converged: 1232244 reads
optimizePositions()-- changed: 2471014 reads
optimizePositions()-- Recomputing positions, iteration 2, with 20 threads.
optimizePositions()-- Reset zero.
optimizePositions()-- Checking convergence.
optimizePositions()-- converged: 1848999 reads
optimizePositions()-- changed: 1854259 reads
optimizePositions()-- Recomputing positions, iteration 3, with 20 threads.
optimizePositions()-- Reset zero.
optimizePositions()-- Checking convergence.
optimizePositions()-- converged: 1869125 reads
optimizePositions()-- changed: 1834133 reads
optimizePositions()-- Recomputing positions, iteration 4, with 20 threads.
optimizePositions()-- Reset zero.
optimizePositions()-- Checking convergence.
optimizePositions()-- converged: 1881632 reads
optimizePositions()-- changed: 1821626 reads
optimizePositions()-- Recomputing positions, iteration 5, with 20 threads.
optimizePositions()-- Reset zero.
optimizePositions()-- Checking convergence.
optimizePositions()-- converged: 1901495 reads
optimizePositions()-- changed: 1801763 reads
optimizePositions()-- Expanding short reads with 20 threads.
optimizePositions()-- Updating positions.
optimizePositions()-- Finished.
splitDiscontinuous()-- Tested 2789 tigs, split 9 tigs into 21 new tigs.
==> MERGE ORPHANS.
computeErrorProfiles()-- Computing error profiles for 2819 tigs, with 20 threads.
computeErrorProfiles()-- Finished.
findPotentialOrphans()-- working on 2819 tigs.
findPotentialOrphans()-- found 1223 potential orphans.
mergeOrphans()-- flagged 412 bubble tigs with 41679 reads
mergeOrphans()-- placed 133 unique orphan tigs with 5604 reads
mergeOrphans()-- shattered 56 repeat orphan tigs with 3244 reads
mergeOrphans()-- ignored 22 tigs with 3559 reads; failed to place
mergeOrphans()--
==> MARK SIMPLE BUBBLES.
using 0.010000 user-specified threshold
findPotentialOrphans()-- working on 2819 tigs.
read 2466988 at 1456192 1522940 olap to read 3177577 hangs 66862 66762 -> coords 1523054 1522940
bogart: bogart/AS_BAT_MergeOrphans.C:258: void findPotentialOrphans(TigVector&, BubTargetList&, bool): Assertion `mincoord < maxcoord' failed.
Failed with 'Aborted'; backtrace (libbacktrace):
utility/src/utility/system-stackTrace.C::83 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
bogart/AS_BAT_MergeOrphans.C::258 in _Z20findPotentialOrphansR9TigVectorRSt3mapIjSt6vectorIjSaIjEESt4lessIjESaISt4pairIKjS4_EEEb()
bogart/AS_BAT_MergeOrphans.C::896 in _Z12mergeOrphansR9TigVectorddb()
bogart/bogart.C::572 in main()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
Thanks!
Hi,
I ran into this bogart failure with canu 2.1, 2.1.1 and 2.2.
This is the canu.out
I then tried to manually changed the setting of unitigger.jobSubmit-01.sh as --cpus-per-task=20 --mem-per-cpu=5000m
and unitigger.sh -threads 20 -M 50
And here is unitigger.err
Thanks!