Skip to content

canu/2.0 failed to find the number of jobs in 'correction/0-mercounts/meryl-count.sh' #1740

@einzigsue

Description

@einzigsue

I installed canu/2.0 under CentOS 8 on our cluster and we tested the installation with two cases and both of them failed with the following error in the file canu.out.

Loading python3/3.7.4
  Loading requirement: intel-mkl/2019.3.199

Found perl:
   /bin/perl
   This is perl 5, version 26, subversion 3 (v5.26.3) built for x86_64-linux-thread-multi

Found java:
   /bin/java
   openjdk version "1.8.0_242"

Found canu:
   /apps/canu/2.0/bin/canu
   Canu 2.0

-- Canu 2.0
--
-- Detected Java(TM) Runtime Environment '1.8.0_242' (from 'java') with -d64 support.
-- Detected gnuplot version '5.2 patchlevel 4   ' (from 'gnuplot') and image format 'png'.
-- Detected 96 CPUs and 189 gigabytes of memory.
-- Detecting PBSPro resources.
--
-- Found 2875 hosts with  48 cores and  192 GB memory under PBSPro control.
-- Found 153 hosts with  48 cores and  203 GB memory under PBSPro control.
-- Found  49 hosts with  48 cores and 1536 GB memory under PBSPro control.
-- Found 160 hosts with  48 cores and  383 GB memory under PBSPro control.
-- Found   1 host  with  48 cores and 1503 GB memory under PBSPro control.
--
--                         (tag)Threads
--                (tag)Memory         |
--        (tag)             |         |  algorithm
--        -------  ----------  --------  -----------------------------
-- Grid:  meryl     12.000 GB    4 CPUs  (k-mer counting)
-- Grid:  hap        8.000 GB    4 CPUs  (read-to-haplotype assignment)
-- Grid:  cormhap    6.000 GB   16 CPUs  (overlap detection with mhap)
-- Grid:  obtmhap    6.000 GB   16 CPUs  (overlap detection with mhap)
-- Grid:  utgmhap    6.000 GB   16 CPUs  (overlap detection with mhap)
-- Grid:  cor        8.000 GB    4 CPUs  (read correction)
-- Grid:  ovb        4.000 GB    1 CPU   (overlap store bucketizer)
-- Grid:  ovs        8.000 GB    1 CPU   (overlap store sorting)
-- Grid:  red       16.000 GB    4 CPUs  (read error detection)
-- Grid:  oea        8.000 GB    1 CPU   (overlap error adjustment)
-- Grid:  bat       16.000 GB    4 CPUs  (contig construction with bogart)
-- Grid:  cns        -.--- GB    4 CPUs  (consensus)
-- Grid:  gfa       16.000 GB    4 CPUs  (GFA alignment and processing)
...
----------------------------------------
-- Starting command on Tue Jun  9 09:44:08 2020 with 4913847.217 GB free disk space

    cd correction/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Tue Jun  9 09:44:09 2020 (one second) with 4913847.217 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--
--  For 1572 reads with 16809400 bases, limit to 1 batch.
--  Will count kmers using  jobs, each using  GB and 4 threads.
--
-- Report changed.
-- Finished stage 'merylConfigure', reset canuIteration.

ABORT:
ABORT: Canu 2.0
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.
ABORT:
ABORT:   failed to find the number of jobs in 'correction/0-mercounts/meryl-count.sh'.
ABORT:

Is it because the file correction/0-mercounts/meryl-count.sh is not correctly generated in the following lines?

if [ $jobid -gt  ]; then
  echo Error: Only  jobs, you asked for $jobid.
  exit 1
fi

Does canu use some sort of template to generate the meryl-count script? Is there any recent changes since version 1.9? We don't have any issues using the earlier version 1.9 yet.
Here is the command we used to start canu and the jobwrapper.sh passed to gridEngineSubmitCommnad, together with other grid engine related options, is the way we adapt canu to our cluster which should have very little impact on how the file correction/0-mercounts/meryl-count.sh is generated.

canu \
    -d canu_out4 \
    -p rdna \
    overlapper=mhap \
    utgReAlign=true \
    genomeSize=84k \
    useGrid=true \
    gridEngineSubmitCommand="${CANU_BASE}/Linux-amd64/bin/jobwrapper.sh -j oe" \
    gridEngine=pbspro \
    gridEngineResourceOption="-lncpus=THREADS,mem=MEMORY" \
    gridEngineNameToJobIDCommand="qstat -f |grep -F -B 1 WAIT_TAG | grep Id: | cut -c 9-" \
    stageDirectory=\$PBS_JOBFS \
    gridEngineStageOption="-ljobfs=10GB" \
    gridEngineArrayMaxJobs=500 \
    gridOptionsExecutive="-lwalltime=4:00:00" \
    gridOptions="-q normal -lwd -P a00 -lstorage=tmp/a00" \
    -nanopore-raw 20samples.fastq 2> canu.log

Let us know your diagnosis
Yue

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions