Skip to content

default maxpendtime set for 1 day#1595

Merged
shahzebsiddiqui merged 3 commits intodevelfrom
typeerror_bug_maxpendtime
Aug 21, 2023
Merged

default maxpendtime set for 1 day#1595
shahzebsiddiqui merged 3 commits intodevelfrom
typeerror_bug_maxpendtime

Conversation

@shahzebsiddiqui
Copy link
Member

@shahzebsiddiqui shahzebsiddiqui commented Aug 21, 2023

This PR will enforce we have a default maxpendtime set for 1 day. This address bug in #1593 reported by @karcaw. I can confirm this error is present in codebase. I was able to reproduce this error when submitting a job. We can test this feature by holding the job permanently so add -H.

Here is an example on Perlmutter

buildspecs:
  hostname_perlmutter:
    description: run hostname on perlmutter or muller
    type: script
    executor: '(perlmutter|muller).slurm.(regular|debug|preempt)'
    tags: ["queues","jobs"]
    sbatch: ["-t 5", "-n 1", "-N 1", "-C gpu", "-A m3503", '-H']
    run: hostname

When building this test it was polling every 30sec and will continue to poll till it reaches 1 day. Shown below is output of the run phase

Polling Jobs in 30 seconds
                                Pending and Suspended Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                      ┃ executor                 ┃ jobid    ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hostname_perlmutter/34101ea5 │ perlmutter.slurm.regular │ 14179008 │ PENDING  │ 34.416  │
│ hostname_perlmutter/4f8f3be3 │ perlmutter.slurm.debug   │ 14179007 │ PENDING  │ 34.457  │
│ hostname_perlmutter/53570379 │ perlmutter.slurm.preempt │ 14179006 │ PENDING  │ 34.499  │
└──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘
Polling Jobs in 30 seconds
                                Pending and Suspended Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                      ┃ executor                 ┃ jobid    ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hostname_perlmutter/34101ea5 │ perlmutter.slurm.regular │ 14179008 │ PENDING  │ 64.57   │
│ hostname_perlmutter/4f8f3be3 │ perlmutter.slurm.debug   │ 14179007 │ PENDING  │ 64.611  │
│ hostname_perlmutter/53570379 │ perlmutter.slurm.preempt │ 14179006 │ PENDING  │ 64.653  │
└──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘
Polling Jobs in 30 seconds
                                Pending and Suspended Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                      ┃ executor                 ┃ jobid    ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hostname_perlmutter/34101ea5 │ perlmutter.slurm.regular │ 14179008 │ PENDING  │ 94.724  │
│ hostname_perlmutter/4f8f3be3 │ perlmutter.slurm.debug   │ 14179007 │ PENDING  │ 94.763  │
│ hostname_perlmutter/53570379 │ perlmutter.slurm.preempt │ 14179006 │ PENDING  │ 94.803  │
└──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘
Polling Jobs in 30 seconds
                                Pending and Suspended Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                      ┃ executor                 ┃ jobid    ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hostname_perlmutter/34101ea5 │ perlmutter.slurm.regular │ 14179008 │ PENDING  │ 124.874 │
│ hostname_perlmutter/4f8f3be3 │ perlmutter.slurm.debug   │ 14179007 │ PENDING  │ 124.913 │
│ hostname_perlmutter/53570379 │ perlmutter.slurm.preempt │ 14179006 │ PENDING  │ 124.953 │
└──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘
Polling Jobs in 30 seconds
                                Pending and Suspended Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                      ┃ executor                 ┃ jobid    ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hostname_perlmutter/34101ea5 │ perlmutter.slurm.regular │ 14179008 │ PENDING  │ 155.024 │
│ hostname_perlmutter/4f8f3be3 │ perlmutter.slurm.debug   │ 14179007 │ PENDING  │ 155.065 │
│ hostname_perlmutter/53570379 │ perlmutter.slurm.preempt │ 14179006 │ PENDING  │ 155.119 │
└──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘

When one specifies --maxpendtime on the command line this value is honored and has the highest precendence

(buildtest) siddiq90@login02> buildtest bd -b perlmutter_hostname.yml --maxpendtime=15
╭───────────────────────────────────────────────────── buildtest summary ──────────────────────────────────────────────────────╮
│                                                                                                                              │
│ User:               siddiq90                                                                                                 │
│ Hostname:           login02                                                                                                  │
│ Platform:           Linux                                                                                                    │
│ Current Time:       2023/08/21 07:37:07                                                                                      │
│ buildtest path:     /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest                                                │
│ buildtest version:  1.5                                                                                                      │
│ python path:        /global/u1/s/siddiq90/.local/share/virtualenvs/buildtest-WqshQcL1/bin/python3                            │
│ python version:     3.9.7                                                                                                    │
│ Configuration File: /global/u1/s/siddiq90/gitrepos/buildtest-nersc/config.yml                                                │
│ Test Directory:     /global/u1/s/siddiq90/gitrepos/buildtest/var/tests                                                       │
│ Report File:        /global/u1/s/siddiq90/gitrepos/buildtest/var/report.json                                                 │
│ Command:            /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest bd -b perlmutter_hostname.yml --maxpendtime=15 │
│                                                                                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
─────────────────────────────────────────────────────────────────────────────────────────────────  Discovering Buildspecs ─────────────────────────────────────────────────────────────────────────────────────────────────
                                   Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                                ║
╟──────────────────────────────────────────────────────────────────────────────────────────╢
║ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
─────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml: VALID
Total builder objects created: 3
                                                                                                  Builders by type=script
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder                      ┃ type   ┃ executor                 ┃ compiler ┃ nodes ┃ procs ┃ description                          ┃ buildspecs                                                                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hostname_perlmutter/6ea344a2 │ script │ perlmutter.slurm.regular │ None     │ None  │ None  │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │
├──────────────────────────────┼────────┼──────────────────────────┼──────────┼───────┼───────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ hostname_perlmutter/ced0716d │ script │ perlmutter.slurm.debug   │ None     │ None  │ None  │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │
├──────────────────────────────┼────────┼──────────────────────────┼──────────┼───────┼───────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ hostname_perlmutter/2f840152 │ script │ perlmutter.slurm.preempt │ None     │ None  │ None  │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │
└──────────────────────────────┴────────┴──────────────────────────┴──────────┴───────┴───────┴──────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘
                                                                  Batch Job Builders
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder                      ┃ executor                 ┃ buildspecs                                                                               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hostname_perlmutter/6ea344a2 │ perlmutter.slurm.regular │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │
├──────────────────────────────┼──────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
│ hostname_perlmutter/ced0716d │ perlmutter.slurm.debug   │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │
├──────────────────────────────┼──────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
│ hostname_perlmutter/2f840152 │ perlmutter.slurm.preempt │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │
└──────────────────────────────┴──────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ──────────────────────────────────────────────────────────────────────────────────────────────────────
hostname_perlmutter/6ea344a2: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/6ea344a2
hostname_perlmutter/6ea344a2: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/6ea344a2/stage
hostname_perlmutter/6ea344a2: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/6ea344a2/hostname_perlmutter_build.sh
hostname_perlmutter/ced0716d: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/ced0716d
hostname_perlmutter/ced0716d: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/ced0716d/stage
hostname_perlmutter/ced0716d: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/ced0716d/hostname_perlmutter_build.sh
hostname_perlmutter/2f840152: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/2f840152
hostname_perlmutter/2f840152: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/2f840152/stage
hostname_perlmutter/2f840152: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/2f840152/hostname_perlmutter_build.sh
────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ──────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 8 processes for processing builders
─────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ───────────────────────────────────────────────────────────────────────────────────────────────────────
hostname_perlmutter/2f840152 does not have any dependencies adding test to queue
hostname_perlmutter/6ea344a2 does not have any dependencies adding test to queue
hostname_perlmutter/ced0716d does not have any dependencies adding test to queue
In this iteration we are going to run the following tests: [hostname_perlmutter/2f840152, hostname_perlmutter/6ea344a2, hostname_perlmutter/ced0716d]
hostname_perlmutter/ced0716d: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/ced0716d/stage
hostname_perlmutter/2f840152: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/2f840152/stage
hostname_perlmutter/6ea344a2: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/6ea344a2/stage
hostname_perlmutter/2f840152: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh
hostname_perlmutter/ced0716d: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh
hostname_perlmutter/6ea344a2: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh
hostname_perlmutter/6ea344a2: JobID 14179165 dispatched to scheduler
hostname_perlmutter/ced0716d: JobID 14179166 dispatched to scheduler
hostname_perlmutter/2f840152: JobID 14179167 dispatched to scheduler
Polling Jobs in 30 seconds
hostname_perlmutter/ced0716d: Cancelling Job 14179166 because job exceeds max pend time of 15 sec with current pend time of 30.156 sec
hostname_perlmutter/6ea344a2: Cancelling Job 14179165 because job exceeds max pend time of 15 sec with current pend time of 30.268 sec
hostname_perlmutter/2f840152: Cancelling Job 14179167 because job exceeds max pend time of 15 sec with current pend time of 30.377 sec
Unable to run any tests

Now i added maxpendtime in the configuration file this will be under defaults keyword shown be is a sample configuration on Perlmutter

  perlmutter:
    description: Cray Shasta system with AMD CPU and NVIDIA A100 GPUs
    hostnames:
    - login[01-40]
    moduletool: lmod
    poolsize: 8
    buildspecs:
      rebuild: false
      count: 15
      format: name,description
      terse: false
    report:
      count: 25
      terse: false
      format: name,id,state,runtime,returncode
      latest: true
      oldest: false
    executors:
      defaults:
        pollinterval: 30
        maxpendtime: 35

Now if i run this without specifying --maxpendtime it will take the value of the configuration file. Note we should have this test cancel at 60s since we poll every 30s and maxpendtime hasnt reached

(buildtest) siddiq90@login02> buildtest bd -b perlmutter_hostname.yml
╭───────────────────────────────────────────── buildtest summary ─────────────────────────────────────────────╮
│                                                                                                             │
│ User:               siddiq90                                                                                │
│ Hostname:           login02                                                                                 │
│ Platform:           Linux                                                                                   │
│ Current Time:       2023/08/21 07:39:35                                                                     │
│ buildtest path:     /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest                               │
│ buildtest version:  1.5                                                                                     │
│ python path:        /global/u1/s/siddiq90/.local/share/virtualenvs/buildtest-WqshQcL1/bin/python3           │
│ python version:     3.9.7                                                                                   │
│ Configuration File: /global/u1/s/siddiq90/gitrepos/buildtest-nersc/config.yml                               │
│ Test Directory:     /global/u1/s/siddiq90/gitrepos/buildtest/var/tests                                      │
│ Report File:        /global/u1/s/siddiq90/gitrepos/buildtest/var/report.json                                │
│ Command:            /global/homes/s/siddiq90/gitrepos/buildtest/bin/buildtest bd -b perlmutter_hostname.yml │
│                                                                                                             │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
─────────────────────────────────────────────────────────────────────────────────────────────────  Discovering Buildspecs ─────────────────────────────────────────────────────────────────────────────────────────────────
                                   Discovered buildspecs
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ buildspec                                                                                ║
╟──────────────────────────────────────────────────────────────────────────────────────────╢
║ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
─────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml: VALID
Total builder objects created: 3
                                                                                                  Builders by type=script
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder                      ┃ type   ┃ executor                 ┃ compiler ┃ nodes ┃ procs ┃ description                          ┃ buildspecs                                                                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hostname_perlmutter/9eb614a3 │ script │ perlmutter.slurm.regular │ None     │ None  │ None  │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │
├──────────────────────────────┼────────┼──────────────────────────┼──────────┼───────┼───────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ hostname_perlmutter/b1b8ecaf │ script │ perlmutter.slurm.debug   │ None     │ None  │ None  │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │
├──────────────────────────────┼────────┼──────────────────────────┼──────────┼───────┼───────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ hostname_perlmutter/f4547c2f │ script │ perlmutter.slurm.preempt │ None     │ None  │ None  │ run hostname on perlmutter or muller │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostn… │
└──────────────────────────────┴────────┴──────────────────────────┴──────────┴───────┴───────┴──────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘
                                                                  Batch Job Builders
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder                      ┃ executor                 ┃ buildspecs                                                                               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ hostname_perlmutter/9eb614a3 │ perlmutter.slurm.regular │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │
├──────────────────────────────┼──────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
│ hostname_perlmutter/b1b8ecaf │ perlmutter.slurm.debug   │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │
├──────────────────────────────┼──────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────┤
│ hostname_perlmutter/f4547c2f │ perlmutter.slurm.preempt │ /global/u1/s/siddiq90/gitrepos/buildtest-nersc/buildspecs/queues/perlmutter_hostname.yml │
└──────────────────────────────┴──────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ──────────────────────────────────────────────────────────────────────────────────────────────────────
hostname_perlmutter/9eb614a3: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/9eb614a3
hostname_perlmutter/9eb614a3: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/9eb614a3/stage
hostname_perlmutter/9eb614a3: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/9eb614a3/hostname_perlmutter_build.sh
hostname_perlmutter/b1b8ecaf: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/b1b8ecaf
hostname_perlmutter/b1b8ecaf: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/b1b8ecaf/stage
hostname_perlmutter/b1b8ecaf: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/b1b8ecaf/hostname_perlmutter_build.sh
hostname_perlmutter/f4547c2f: Creating test directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/f4547c2f
hostname_perlmutter/f4547c2f: Creating the stage directory: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/f4547c2f/stage
hostname_perlmutter/f4547c2f: Writing build script: /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/f4547c2f/hostname_perlmutter_build.sh
────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ──────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 8 processes for processing builders
─────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ───────────────────────────────────────────────────────────────────────────────────────────────────────
hostname_perlmutter/9eb614a3 does not have any dependencies adding test to queue
hostname_perlmutter/f4547c2f does not have any dependencies adding test to queue
hostname_perlmutter/b1b8ecaf does not have any dependencies adding test to queue
In this iteration we are going to run the following tests: [hostname_perlmutter/9eb614a3, hostname_perlmutter/f4547c2f, hostname_perlmutter/b1b8ecaf]
hostname_perlmutter/b1b8ecaf: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.debug/perlmutter_hostname/hostname_perlmutter/b1b8ecaf/stage
hostname_perlmutter/9eb614a3: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.regular/perlmutter_hostname/hostname_perlmutter/9eb614a3/stage
hostname_perlmutter/f4547c2f: Current Working Directory : /global/u1/s/siddiq90/gitrepos/buildtest/var/tests/perlmutter.slurm.preempt/perlmutter_hostname/hostname_perlmutter/f4547c2f/stage
hostname_perlmutter/f4547c2f: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh
hostname_perlmutter/b1b8ecaf: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh
hostname_perlmutter/9eb614a3: Running Test via command: bash --norc --noprofile -eo pipefail hostname_perlmutter_build.sh
hostname_perlmutter/f4547c2f: JobID 14179217 dispatched to scheduler
hostname_perlmutter/9eb614a3: JobID 14179218 dispatched to scheduler
hostname_perlmutter/b1b8ecaf: JobID 14179219 dispatched to scheduler
Polling Jobs in 30 seconds
                                Pending and Suspended Jobs
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┓
┃ builder                      ┃ executor                 ┃ jobid    ┃ jobstate ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━┩
│ hostname_perlmutter/f4547c2f │ perlmutter.slurm.preempt │ 14179217 │ PENDING  │ 30.155  │
│ hostname_perlmutter/b1b8ecaf │ perlmutter.slurm.debug   │ 14179219 │ PENDING  │ 30.192  │
│ hostname_perlmutter/9eb614a3 │ perlmutter.slurm.regular │ 14179218 │ PENDING  │ 30.231  │
└──────────────────────────────┴──────────────────────────┴──────────┴──────────┴─────────┘
Polling Jobs in 30 seconds
hostname_perlmutter/f4547c2f: Cancelling Job 14179217 because job exceeds max pend time of 35 sec with current pend time of 60.307 sec
hostname_perlmutter/b1b8ecaf: Cancelling Job 14179219 because job exceeds max pend time of 35 sec with current pend time of 61.653 sec
hostname_perlmutter/9eb614a3: Cancelling Job 14179218 because job exceeds max pend time of 35 sec with current pend time of 63.043 sec
Unable to run any tests

@shahzebsiddiqui
refactor codebase to ensure we have a maxpendtime of 1 day (86400sec)…
2b390fb 
… which will address bug where maxpendtime is None when submitting job

to batch scheduler and no --maxpendtime is specified or defined in configuration file
@shahzebsiddiqui
applying black
bec2d83
@shahzebsiddiqui
fix a bug that was intrudced when load() method was not invoked at st…
5c7ad8e 
…artup. This method

needs to be invoked. This fix an issue in regression test
@shahzebsiddiqui shahzebsiddiqui merged commit 7ca12b4 into devel Aug 21, 2023
@shahzebsiddiqui shahzebsiddiqui deleted the typeerror_bug_maxpendtime branch August 21, 2023 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

size/XS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@shahzebsiddiqui