Last login: Fri Sep 26 09:22:51 UTC 2025 on pts/0 Running Spack setup, this may take a moment on first login. [root@slurmh4d-controller ~]# cat /var/log/slurm/setup.log 2025-09-26 09:17:01,218 INFO: Starting setup, fetching config 2025-09-26 09:17:01,291 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:06,340 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:11,382 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:16,422 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:21,477 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:26,521 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:31,568 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:36,626 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:41,666 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:46,714 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:51,754 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:17:56,794 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:01,838 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:06,877 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:11,912 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:16,962 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:22,000 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:27,058 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:32,094 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:37,131 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:42,172 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:47,208 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:52,245 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:18:57,284 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:02,330 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:07,368 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:12,407 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:17,444 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:22,490 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:27,529 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:32,570 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:37,606 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:42,657 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:47,701 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:52,738 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:19:57,777 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:02,820 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:07,864 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:12,908 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:17,960 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:23,004 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:28,057 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:33,099 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:38,143 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:43,196 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:48,243 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:53,280 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:20:58,317 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:21:03,359 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:21:08,400 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:21:13,445 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:21:18,479 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:21:23,520 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:21:28,562 WARNING: config is not ready yet: slurmh4d-files/config.yaml not found in bucket, sleeping for 5s 2025-09-26 09:21:33,715 WARNING: config is not ready yet: nodeset h4dnodeset not defined in config, sleeping for 5s 2025-09-26 09:21:39,126 INFO: Config fetched 2025-09-26 09:21:39,132 ERROR: Command '['systemctl', 'is-active', '--quiet', 'google-cloud-ops-agent.service']' returned exit status 3. 2025-09-26 09:21:39,133 INFO: Setting up controller 2025-09-26 09:21:39,515 INFO: installing custom script: controller.d/ghpc_startup.sh from slurmh4d-files/slurm-controller-script-ghpc_startup_sh 2025-09-26 09:22:04,172 INFO: Set up network storage 2025-09-26 09:22:04,174 INFO: Setting up mount (nfs) 10.18.99.42:/homeshare to /home 2025-09-26 09:22:04,174 INFO: Setting up mount (nfs) 10.217.27.194:/appsshare to /opt/apps 2025-09-26 09:22:04,176 INFO: Waiting for '/home' to be mounted... 2025-09-26 09:22:04,177 INFO: Waiting for '/opt/apps' to be mounted... 2025-09-26 09:22:07,372 INFO: Mount point '/home' was mounted. 2025-09-26 09:22:07,374 INFO: Mount point '/opt/apps' was mounted. 2025-09-26 09:22:07,376 INFO: running script ghpc_startup.sh with timeout=300 2025-09-26 09:24:43,872 INFO: ghpc_startup.sh returncode=0 stdout=Last metadata expiration check: 0:05:01 ago on Fri 26 Sep 2025 09:17:11 AM UTC. Package python3.12-3.12.11-2.el8_10.x86_64 is already installed. Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: python3.12-pip noarch 23.2.1-4.el8 appstream 3.2 M Installing weak dependencies: python3.12-setuptools noarch 68.2.2-5.el8_10 appstream 1.7 M Transaction Summary ================================================================================ Install 2 Packages Total download size: 4.9 M Installed size: 21 M Downloading Packages: (1/2): python3.12-pip-23.2.1-4.el8.noarch.rpm 56 MB/s | 3.2 MB 00:00 (2/2): python3.12-setuptools-68.2.2-5.el8_10.no 25 MB/s | 1.7 MB 00:00 -------------------------------------------------------------------------------- Total 24 MB/s | 4.9 MB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : python3.12-setuptools-68.2.2-5.el8_10.noarch 1/2 Installing : python3.12-pip-23.2.1-4.el8.noarch 2/2 Running scriptlet: python3.12-pip-23.2.1-4.el8.noarch 2/2 Verifying : python3.12-pip-23.2.1-4.el8.noarch 1/2 Verifying : python3.12-setuptools-68.2.2-5.el8_10.noarch 2/2 Installed: python3.12-pip-23.2.1-4.el8.noarch python3.12-setuptools-68.2.2-5.el8_10.noarch Complete! pip 23.2.1 from /usr/lib/python3.12/site-packages/pip (python 3.12) Requirement already satisfied: pip in /usr/local/ghpc-venv/lib64/python3.12/site-packages (23.2.1) Collecting pip Obtaining dependency information for pip from https://files.pythonhosted.org/packages/b7/3f/945ef7ab14dc4f9d7f40288d2df998d1837ee0888ec3659c813487572faa/pip-25.2-py3-none-any.whl.metadata Downloading pip-25.2-py3-none-any.whl.metadata (4.7 kB) Downloading pip-25.2-py3-none-any.whl (1.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 51.8 MB/s eta 0:00:00 Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 23.2.1 Uninstalling pip-23.2.1: Successfully uninstalled pip-23.2.1 Successfully installed pip-25.2 Collecting wheel==0.45.1 Using cached wheel-0.45.1-py3-none-any.whl.metadata (2.3 kB) Using cached wheel-0.45.1-py3-none-any.whl (72 kB) Installing collected packages: wheel Successfully installed wheel-0.45.1 Collecting setuptools==80.8.0 Downloading setuptools-80.8.0-py3-none-any.whl.metadata (6.6 kB) Downloading setuptools-80.8.0-py3-none-any.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 65.4 MB/s 0:00:00 Installing collected packages: setuptools Successfully installed setuptools-80.8.0 Collecting ansible==8.7.0 Downloading ansible-8.7.0-py3-none-any.whl.metadata (7.9 kB) Collecting ansible-core~=2.15.7 (from ansible==8.7.0) Downloading ansible_core-2.15.13-py3-none-any.whl.metadata (7.0 kB) Collecting jinja2>=3.0.0 (from ansible-core~=2.15.7->ansible==8.7.0) Downloading jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB) Collecting PyYAML>=5.1 (from ansible-core~=2.15.7->ansible==8.7.0) Downloading pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.4 kB) Collecting cryptography (from ansible-core~=2.15.7->ansible==8.7.0) Downloading cryptography-46.0.1-cp311-abi3-manylinux_2_28_x86_64.whl.metadata (5.7 kB) Collecting packaging (from ansible-core~=2.15.7->ansible==8.7.0) Downloading packaging-25.0-py3-none-any.whl.metadata (3.3 kB) Collecting resolvelib<1.1.0,>=0.5.3 (from ansible-core~=2.15.7->ansible==8.7.0) Downloading resolvelib-1.0.1-py2.py3-none-any.whl.metadata (4.0 kB) Collecting MarkupSafe>=2.0 (from jinja2>=3.0.0->ansible-core~=2.15.7->ansible==8.7.0) Downloading MarkupSafe-3.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB) Collecting cffi>=2.0.0 (from cryptography->ansible-core~=2.15.7->ansible==8.7.0) Downloading cffi-2.0.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.6 kB) Collecting pycparser (from cffi>=2.0.0->cryptography->ansible-core~=2.15.7->ansible==8.7.0) Downloading pycparser-2.23-py3-none-any.whl.metadata (993 bytes) Downloading ansible-8.7.0-py3-none-any.whl (48.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 MB 163.7 MB/s 0:00:00 Downloading ansible_core-2.15.13-py3-none-any.whl (2.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 171.4 MB/s 0:00:00 Downloading resolvelib-1.0.1-py2.py3-none-any.whl (17 kB) Downloading jinja2-3.1.6-py3-none-any.whl (134 kB) Downloading MarkupSafe-3.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23 kB) Downloading pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (807 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 807.9/807.9 kB 91.1 MB/s 0:00:00 Downloading cryptography-46.0.1-cp311-abi3-manylinux_2_28_x86_64.whl (4.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 193.3 MB/s 0:00:00 Downloading cffi-2.0.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (219 kB) Downloading packaging-25.0-py3-none-any.whl (66 kB) Downloading pycparser-2.23-py3-none-any.whl (118 kB) Installing collected packages: resolvelib, PyYAML, pycparser, packaging, MarkupSafe, jinja2, cffi, cryptography, ansible-core, ansible Successfully installed MarkupSafe-3.0.2 PyYAML-6.0.3 ansible-8.7.0 ansible-core-2.15.13 cffi-2.0.0 cryptography-46.0.1 jinja2-3.1.6 packaging-25.0 pycparser-2.23 resolvelib-1.0.1 Last metadata expiration check: 0:05:40 ago on Fri 26 Sep 2025 09:17:11 AM UTC. Package python3.12-3.12.11-2.el8_10.x86_64 is already installed. Package python3.12-pip-23.2.1-4.el8.noarch is already installed. Dependencies resolved. Nothing to do. Complete! pip 23.2.1 from /usr/lib/python3.12/site-packages/pip (python 3.12) PLAY [Create python virtual env for a tool] ************************************ TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Install dependencies through system package manager] ********************* ok: [localhost] TASK [Create virtualenv for tool] ********************************************** changed: [localhost] TASK [Add google-cloud-storage to virtualenv] ********************************** changed: [localhost] PLAY RECAP ********************************************************************* localhost : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 PLAY [Install Software] ******************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Print software name] ***************************************************** ok: [localhost] => {} MSG: Running installation for software: spack TASK [Add profile script for software] ***************************************** changed: [localhost] TASK [Check if user already exists] ******************************************** fatal: [localhost]: FAILED! => { "changed": false } MSG: One or more supplied key could not be found in the database. TASK [User did not exist, create group for system user] ************************ changed: [localhost] TASK [Create system user] ****************************************************** changed: [localhost] TASK [Create parent of install directory] ************************************** changed: [localhost] TASK [Set lock dir] ************************************************************ ok: [localhost] TASK [Acquire lock] ************************************************************ changed: [localhost] TASK [Add hostname to lock_dir] ************************************************ changed: [localhost] TASK [Clone branch or tag into installation directory] ************************* changed: [localhost] TASK [Clone commit hash into installation directory] *************************** skipping: [localhost] => (item=git clone https://github.com/spack/spack /app/spack) skipping: [localhost] => (item=git -C /app/spack checkout v0.20.0) skipping: [localhost] TASK [Transfer ownership to system user] *************************************** changed: [localhost] TASK [Finalize setup] ********************************************************** changed: [localhost] TASK [Apply chmod] ************************************************************* changed: [localhost] TASK [Release lock] ************************************************************ changed: [localhost] TASK [Wait for lock] *********************************************************** skipping: [localhost] PLAY RECAP ********************************************************************* localhost : ok=14 changed=11 unreachable=0 failed=0 skipped=2 rescued=1 ignored=0 PLAY [Execute Commands] ******************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Print commands to be executed] ******************************************* ok: [localhost] => {} MSG: - echo 'no spack commands provided' TASK [Streaming log info] ****************************************************** ok: [localhost] => {} MSG: Logs from commands will not be printed here until success (or failure) Streaming logs can be found at /var/log/spack.log TASK [Ensure user can write to log file] *************************************** changed: [localhost] TASK [Execute commands] ******************************************************** changed: [localhost] TASK [Print commands output] *************************************************** ok: [localhost] => { "output.stdout_lines": [ " === Starting commands ===", "no spack commands provided", " === Finished commands ===" ] } PLAY RECAP ********************************************************************* localhost : ok=6 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 stderr=Fri Sep 26 09:22:09 +0000 2025 Info [6040]: === start executing runner: 99-running-script-warning.sh-54ee === Fri Sep 26 09:22:09 +0000 2025 Info [6040]: === 99-running-script-warning.sh-54ee finished with exit_code=0 === Fri Sep 26 09:22:10 +0000 2025 Info [6040]: === start executing runner: early_run_hotfixes.sh-b383 === Fri Sep 26 09:22:10 +0000 2025 Info [6040]: === early_run_hotfixes.sh-b383 finished with exit_code=0 === Fri Sep 26 09:22:12 +0000 2025 Info [6040]: === start executing runner: install_ansible_automatic.sh-ccf1 === + REQ_ANSIBLE_VERSION=2.15 + REQ_ANSIBLE_PIP_VERSION=8.7.0 + REQ_PIP_WHEEL_VERSION=0.45.1 + REQ_PIP_SETUPTOOLS_VERSION=80.8.0 + REQ_PIP_MAJOR_VERSION=25 + REQ_PYTHON3_VERSION=9 + main /usr/local/ghpc-venv + '[' 1 -gt 1 ']' + venv_path=/usr/local/ghpc-venv + get_python_path + python_path= + command -v python3 ++ command -v python3 + python_path=/usr/bin/python3 + get_python_major_version /usr/bin/python3 + python_path=/usr/bin/python3 ++ /usr/bin/python3 -c 'import sys; print(sys.version_info.major)' + python_major_version=3 + get_python_minor_version /usr/bin/python3 + python_path=/usr/bin/python3 ++ /usr/bin/python3 -c 'import sys; print(sys.version_info.minor)' + python_minor_version=8 + '[' /usr/bin/python3 = '' ']' + '[' 3 = 2 ']' + '[' 8 -lt 9 ']' + install_python3 + '[' -f /etc/redhat-release ']' + install_python3_dnf ++ rpm -E '%{rhel}' + major_version=8 + set -- '--disablerepo=*' --enablerepo=baseos,appstream + grep -qi 'ID="rhel"' /etc/os-release + '[' 8 -lt 10 ']' + dnf install '--disablerepo=*' --enablerepo=baseos,appstream -y python3.12 python3.12-pip WARNING: kmod-idpf-irdma is installed, but there are no kmod-idpf-irdma packages available in the configured repositories. WARNING: Future kernel updates may be blocked until kmod-idpf-irdma is removed from this system. ++ command -v python3.12 + python_path=/usr/bin/python3.12 + get_python_major_version /usr/bin/python3.12 + python_path=/usr/bin/python3.12 ++ /usr/bin/python3.12 -c 'import sys; print(sys.version_info.major)' + python_major_version=3 + get_python_minor_version /usr/bin/python3.12 + python_path=/usr/bin/python3.12 ++ /usr/bin/python3.12 -c 'import sys; print(sys.version_info.minor)' + python_minor_version=12 + /usr/bin/python3.12 -m pip --version + /usr/bin/python3.12 -m venv /usr/local/ghpc-venv --copies + venv_python_path=/usr/local/ghpc-venv/bin/python3 ++ /usr/local/ghpc-venv/bin/python3 -m pip --version ++ sed -nr 's/^pip ([0-9]+\.[0-9]+).*$/\1/p' + pip_version=23.2 ++ echo 23.2 ++ cut -d . -f 1 + pip_major_version=23 + '[' 23 -lt 25 ']' + /usr/local/ghpc-venv/bin/python3 -m pip install --upgrade pip ++ /usr/local/ghpc-venv/bin/python3 -m pip list --format=freeze ++ grep '^wheel' ++ true + wheel_pkg= + '[' '' '!=' wheel==0.45.1 ']' + /usr/local/ghpc-venv/bin/python3 -m pip install -U wheel==0.45.1 ++ /usr/local/ghpc-venv/bin/python3 -m pip list --format=freeze ++ grep '^setuptools' ++ true + setuptools_pkg= + '[' '' '!=' setuptools==80.8.0 ']' + /usr/local/ghpc-venv/bin/python3 -m pip install -U setuptools==80.8.0 + '[' '!' -f /etc/ansible/ansible.cfg ']' + mkdir /etc/ansible + cat + ansible_version= + command -v ansible-playbook + '[' -z '' ']' + /usr/local/ghpc-venv/bin/python3 -m pip install ansible==8.7.0 + read -r cmd + '[' -L /usr/bin/ansible ']' + ln -s /usr/local/ghpc-venv/bin/ansible /usr/bin/ansible + read -r cmd + '[' -L /usr/bin/ansible-config ']' + ln -s /usr/local/ghpc-venv/bin/ansible-config /usr/bin/ansible-config + read -r cmd + '[' -L /usr/bin/ansible-connection ']' + ln -s /usr/local/ghpc-venv/bin/ansible-connection /usr/bin/ansible-connection + read -r cmd + '[' -L /usr/bin/ansible-console ']' + ln -s /usr/local/ghpc-venv/bin/ansible-console /usr/bin/ansible-console + read -r cmd + '[' -L /usr/bin/ansible-doc ']' + ln -s /usr/local/ghpc-venv/bin/ansible-doc /usr/bin/ansible-doc + read -r cmd + '[' -L /usr/bin/ansible-galaxy ']' + ln -s /usr/local/ghpc-venv/bin/ansible-galaxy /usr/bin/ansible-galaxy + read -r cmd + '[' -L /usr/bin/ansible-inventory ']' + ln -s /usr/local/ghpc-venv/bin/ansible-inventory /usr/bin/ansible-inventory + read -r cmd + '[' -L /usr/bin/ansible-playbook ']' + ln -s /usr/local/ghpc-venv/bin/ansible-playbook /usr/bin/ansible-playbook + read -r cmd + '[' -L /usr/bin/ansible-pull ']' + ln -s /usr/local/ghpc-venv/bin/ansible-pull /usr/bin/ansible-pull + read -r cmd + '[' -L /usr/bin/ansible-test ']' + ln -s /usr/local/ghpc-venv/bin/ansible-test /usr/bin/ansible-test + read -r cmd + '[' -L /usr/bin/ansible-vault ']' + ln -s /usr/local/ghpc-venv/bin/ansible-vault /usr/bin/ansible-vault + read -r cmd Fri Sep 26 09:22:44 +0000 2025 Info [6040]: === install_ansible_automatic.sh-ccf1 finished with exit_code=0 === Fri Sep 26 09:22:46 +0000 2025 Info [6040]: === start executing runner: spack-install-and-setup.sh-58f9 === Fri Sep 26 09:22:48 +0000 2025 Info [7034]: === start executing runner: 99-running-script-warning.sh-54ee === Fri Sep 26 09:22:48 +0000 2025 Info [7034]: === 99-running-script-warning.sh-54ee finished with exit_code=0 === Fri Sep 26 09:22:49 +0000 2025 Info [7034]: === start executing runner: early_run_hotfixes.sh-b383 === Fri Sep 26 09:22:49 +0000 2025 Info [7034]: === early_run_hotfixes.sh-b383 finished with exit_code=0 === Fri Sep 26 09:22:51 +0000 2025 Info [7034]: === start executing runner: install_ansible_automatic.sh-ccf1 === + REQ_ANSIBLE_VERSION=2.15 + REQ_ANSIBLE_PIP_VERSION=8.7.0 + REQ_PIP_WHEEL_VERSION=0.45.1 + REQ_PIP_SETUPTOOLS_VERSION=80.8.0 + REQ_PIP_MAJOR_VERSION=25 + REQ_PYTHON3_VERSION=9 + main /usr/local/ghpc-venv + '[' 1 -gt 1 ']' + venv_path=/usr/local/ghpc-venv + get_python_path + python_path= + command -v python3 ++ command -v python3 + python_path=/usr/bin/python3 + get_python_major_version /usr/bin/python3 + python_path=/usr/bin/python3 ++ /usr/bin/python3 -c 'import sys; print(sys.version_info.major)' + python_major_version=3 + get_python_minor_version /usr/bin/python3 + python_path=/usr/bin/python3 ++ /usr/bin/python3 -c 'import sys; print(sys.version_info.minor)' + python_minor_version=8 + '[' /usr/bin/python3 = '' ']' + '[' 3 = 2 ']' + '[' 8 -lt 9 ']' + install_python3 + '[' -f /etc/redhat-release ']' + install_python3_dnf ++ rpm -E '%{rhel}' + major_version=8 + set -- '--disablerepo=*' --enablerepo=baseos,appstream + grep -qi 'ID="rhel"' /etc/os-release + '[' 8 -lt 10 ']' + dnf install '--disablerepo=*' --enablerepo=baseos,appstream -y python3.12 python3.12-pip WARNING: kmod-idpf-irdma is installed, but there are no kmod-idpf-irdma packages available in the configured repositories. WARNING: Future kernel updates may be blocked until kmod-idpf-irdma is removed from this system. ++ command -v python3.12 + python_path=/usr/bin/python3.12 + get_python_major_version /usr/bin/python3.12 + python_path=/usr/bin/python3.12 ++ /usr/bin/python3.12 -c 'import sys; print(sys.version_info.major)' + python_major_version=3 + get_python_minor_version /usr/bin/python3.12 + python_path=/usr/bin/python3.12 ++ /usr/bin/python3.12 -c 'import sys; print(sys.version_info.minor)' + python_minor_version=12 + /usr/bin/python3.12 -m pip --version + /usr/bin/python3.12 -m venv /usr/local/ghpc-venv --copies + venv_python_path=/usr/local/ghpc-venv/bin/python3 ++ /usr/local/ghpc-venv/bin/python3 -m pip --version ++ sed -nr 's/^pip ([0-9]+\.[0-9]+).*$/\1/p' + pip_version=25.2 ++ echo 25.2 ++ cut -d . -f 1 + pip_major_version=25 + '[' 25 -lt 25 ']' ++ /usr/local/ghpc-venv/bin/python3 -m pip list --format=freeze ++ grep '^wheel' + wheel_pkg=wheel==0.45.1 + '[' wheel==0.45.1 '!=' wheel==0.45.1 ']' ++ /usr/local/ghpc-venv/bin/python3 -m pip list --format=freeze ++ grep '^setuptools' + setuptools_pkg=setuptools==80.8.0 + '[' setuptools==80.8.0 '!=' setuptools==80.8.0 ']' + '[' '!' -f /etc/ansible/ansible.cfg ']' + ansible_version= + command -v ansible-playbook ++ ansible-playbook --version ++ sed -nr 's/^ansible-playbook.*([0-9]+\.[0-9]+\.[0-9]+).*/\1/p' + ansible_version=2.15.13 ++ echo 2.15.13 ++ cut -d . -f 1 + ansible_major_vers=2 ++ echo 2.15.13 ++ cut -d . -f 2 + ansible_minor_vers=15 ++ echo 2.15 ++ cut -d . -f 1 + ansible_req_major_vers=2 ++ echo 2.15 ++ cut -d . -f 2 + ansible_req_minor_vers=15 + '[' -z 2.15.13 ']' + '[' 2 -ne 2 ']' + '[' 15 -lt 15 ']' + read -r cmd + '[' -L /usr/bin/ansible ']' + read -r cmd + '[' -L /usr/bin/ansible-config ']' + read -r cmd + '[' -L /usr/bin/ansible-connection ']' + read -r cmd + '[' -L /usr/bin/ansible-console ']' + read -r cmd + '[' -L /usr/bin/ansible-doc ']' + read -r cmd + '[' -L /usr/bin/ansible-galaxy ']' + read -r cmd + '[' -L /usr/bin/ansible-inventory ']' + read -r cmd + '[' -L /usr/bin/ansible-playbook ']' + read -r cmd + '[' -L /usr/bin/ansible-pull ']' + read -r cmd + '[' -L /usr/bin/ansible-test ']' + read -r cmd + '[' -L /usr/bin/ansible-vault ']' + read -r cmd Fri Sep 26 09:22:55 +0000 2025 Info [7034]: === install_ansible_automatic.sh-ccf1 finished with exit_code=0 === Fri Sep 26 09:22:56 +0000 2025 Info [7034]: === start executing runner: install_spack_deps.yml-532d === Fri Sep 26 09:23:08 +0000 2025 Info [7034]: === install_spack_deps.yml-532d finished with exit_code=0 === Fri Sep 26 09:23:09 +0000 2025 Info [7034]: === start executing runner: install_spack.yml-c947 === [WARNING]: Module remote_tmp /home/spack/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually Fri Sep 26 09:24:40 +0000 2025 Info [7034]: === install_spack.yml-c947 finished with exit_code=0 === Fri Sep 26 09:24:40 +0000 2025 Info [6040]: === spack-install-and-setup.sh-58f9 finished with exit_code=0 === Fri Sep 26 09:24:41 +0000 2025 Info [6040]: === start executing runner: spack_execute_1af3.yml-1af3 === Fri Sep 26 09:24:43 +0000 2025 Info [6040]: === spack_execute_1af3.yml-1af3 finished with exit_code=0 === 2025-09-26 09:24:53,229 INFO: Check status of cluster services 2025-09-26 09:24:53,261 INFO: 3 instances to resume (slurmh4d-h4dnodeset-[0-2]) 2025-09-26 09:24:53,357 INFO: Done setting up controller