I installed canu/2.0 under CentOS 8 on our cluster and we tested the installation with two cases and both of them failed with the following error in the file canu.out.
Loading python3/3.7.4
Loading requirement: intel-mkl/2019.3.199
Found perl:
/bin/perl
This is perl 5, version 26, subversion 3 (v5.26.3) built for x86_64-linux-thread-multi
Found java:
/bin/java
openjdk version "1.8.0_242"
Found canu:
/apps/canu/2.0/bin/canu
Canu 2.0
-- Canu 2.0
--
-- Detected Java(TM) Runtime Environment '1.8.0_242' (from 'java') with -d64 support.
-- Detected gnuplot version '5.2 patchlevel 4 ' (from 'gnuplot') and image format 'png'.
-- Detected 96 CPUs and 189 gigabytes of memory.
-- Detecting PBSPro resources.
--
-- Found 2875 hosts with 48 cores and 192 GB memory under PBSPro control.
-- Found 153 hosts with 48 cores and 203 GB memory under PBSPro control.
-- Found 49 hosts with 48 cores and 1536 GB memory under PBSPro control.
-- Found 160 hosts with 48 cores and 383 GB memory under PBSPro control.
-- Found 1 host with 48 cores and 1503 GB memory under PBSPro control.
--
-- (tag)Threads
-- (tag)Memory |
-- (tag) | | algorithm
-- ------- ---------- -------- -----------------------------
-- Grid: meryl 12.000 GB 4 CPUs (k-mer counting)
-- Grid: hap 8.000 GB 4 CPUs (read-to-haplotype assignment)
-- Grid: cormhap 6.000 GB 16 CPUs (overlap detection with mhap)
-- Grid: obtmhap 6.000 GB 16 CPUs (overlap detection with mhap)
-- Grid: utgmhap 6.000 GB 16 CPUs (overlap detection with mhap)
-- Grid: cor 8.000 GB 4 CPUs (read correction)
-- Grid: ovb 4.000 GB 1 CPU (overlap store bucketizer)
-- Grid: ovs 8.000 GB 1 CPU (overlap store sorting)
-- Grid: red 16.000 GB 4 CPUs (read error detection)
-- Grid: oea 8.000 GB 1 CPU (overlap error adjustment)
-- Grid: bat 16.000 GB 4 CPUs (contig construction with bogart)
-- Grid: cns -.--- GB 4 CPUs (consensus)
-- Grid: gfa 16.000 GB 4 CPUs (GFA alignment and processing)
...
----------------------------------------
-- Starting command on Tue Jun 9 09:44:08 2020 with 4913847.217 GB free disk space
cd correction/0-mercounts
./meryl-configure.sh \
> ./meryl-configure.err 2>&1
-- Finished on Tue Jun 9 09:44:09 2020 (one second) with 4913847.217 GB free disk space
----------------------------------------
-- segments memory batches
-- -------- -------- -------
--
-- For 1572 reads with 16809400 bases, limit to 1 batch.
-- Will count kmers using jobs, each using GB and 4 threads.
--
-- Report changed.
-- Finished stage 'merylConfigure', reset canuIteration.
ABORT:
ABORT: Canu 2.0
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting. If that doesn't work, ask for help.
ABORT:
ABORT: failed to find the number of jobs in 'correction/0-mercounts/meryl-count.sh'.
ABORT:
Is it because the file correction/0-mercounts/meryl-count.sh is not correctly generated in the following lines?
if [ $jobid -gt ]; then
echo Error: Only jobs, you asked for $jobid.
exit 1
fi
Does canu use some sort of template to generate the meryl-count script? Is there any recent changes since version 1.9? We don't have any issues using the earlier version 1.9 yet.
Here is the command we used to start canu and the jobwrapper.sh passed to gridEngineSubmitCommnad, together with other grid engine related options, is the way we adapt canu to our cluster which should have very little impact on how the file correction/0-mercounts/meryl-count.sh is generated.
canu \
-d canu_out4 \
-p rdna \
overlapper=mhap \
utgReAlign=true \
genomeSize=84k \
useGrid=true \
gridEngineSubmitCommand="${CANU_BASE}/Linux-amd64/bin/jobwrapper.sh -j oe" \
gridEngine=pbspro \
gridEngineResourceOption="-lncpus=THREADS,mem=MEMORY" \
gridEngineNameToJobIDCommand="qstat -f |grep -F -B 1 WAIT_TAG | grep Id: | cut -c 9-" \
stageDirectory=\$PBS_JOBFS \
gridEngineStageOption="-ljobfs=10GB" \
gridEngineArrayMaxJobs=500 \
gridOptionsExecutive="-lwalltime=4:00:00" \
gridOptions="-q normal -lwd -P a00 -lstorage=tmp/a00" \
-nanopore-raw 20samples.fastq 2> canu.log
Let us know your diagnosis
Yue
I installed canu/2.0 under CentOS 8 on our cluster and we tested the installation with two cases and both of them failed with the following error in the file canu.out.
Is it because the file correction/0-mercounts/meryl-count.sh is not correctly generated in the following lines?
Does canu use some sort of template to generate the meryl-count script? Is there any recent changes since version 1.9? We don't have any issues using the earlier version 1.9 yet.
Here is the command we used to start canu and the jobwrapper.sh passed to gridEngineSubmitCommnad, together with other grid engine related options, is the way we adapt canu to our cluster which should have very little impact on how the file correction/0-mercounts/meryl-count.sh is generated.
Let us know your diagnosis
Yue