Skip to content

Staging dirs disappear for failed builds. #4977

@hartzell

Description

@hartzell

I build a long list of packages regularly via a Jenkins job.

When a package fails to build, its staging directory is often removed, much to my frustration.

Here's what I see in Jenkins log file (the last line from spack is the ...'make'... line, the it goes on with other output from the driver script):

[...]
==> Installing mono
==> Fetching file:///tmp/apps-build-mirror.DiDlFd/mirror/mono/mono-5.0.1.1.tar.bz2
==> Staging archive: /scratch/hartzelg/spack-cime-sc1-working-dir/workspace/sc1-daily-build/spack/var/spack/stage/mono-5.0.1.1-hx7nxp5x4irgecrvw6mwl354ehdejztd/mono-5.0.1.1.tar.bz2
==> Created stage in /scratch/hartzelg/spack-cime-sc1-working-dir/workspace/sc1-daily-build/spack/var/spack/stage/mono-5.0.1.1-hx7nxp5x4irgecrvw6mwl354ehdejztd
==> Ran patch() for mono
==> Building mono [AutotoolsPackage]
==> Error: ProcessError: Command exited with status 2:
    'make' '-j28'
OTHER SCRIPT OUTPUT

Here's my etc/spack/config.yaml file, I believe this says to stage things in /tmp and link them into the Spack tree:

# -------------------------------------------------------------------------
# This is the project specific spack configuration file.
# -------------------------------------------------------------------------
config:
  # Temporary locations Spack can try to use for builds.
  #
  # A value of $spack/var/spack/stage indicates that Spack should run
  # builds directly inside its install directory without staging them in
  # temporary space.
  #
  # The build stage can be purged with `spack purge --stage`.
  build_stage:
    - $tempdir
    - $spack/var/spack/stage

The link in the Spack tree exists but it points to a directory within /tmp that does not. Other Spack related staging dirs do exist there, they're all empty. /tmp is part of the 698GB / partition, which on this machine is currently 2% full.

[hartzelg@lb097hmdev spack]$ ls -l /scratch/hartzelg/spack-cime-sc1-working-dir/workspace/sc1-daily-build/spack/var/spack/stage/mono-5.0.1.1-hx7nxp5x4irgecrvw6mwl354ehdejztd
lrwxrwxrwx 1 hartzelg users 44 Aug  4 11:31 /scratch/hartzelg/spack-cime-sc1-working-dir/workspace/sc1-daily-build/spack/var/spack/stage/mono-5.0.1.1-hx7nxp5x4irgecrvw6mwl354ehdejztd -> /tmp/hartzelg/spack-stage/spack-stage-rUCQxf
[hartzelg@lb097hmdev spack]$ ls -l /tmp/hartzelg/spack-stage/spack-stage-rUCQxf
ls: cannot access /tmp/hartzelg/spack-stage/spack-stage-rUCQxf: No such file or directory
[hartzelg@lb097hmdev spack]$ ls -l /tmp/hartzelg/spack-stage/spack-stage-* | head
/tmp/hartzelg/spack-stage/spack-stage-0ATOqP:
total 0

/tmp/hartzelg/spack-stage/spack-stage-1foqeC:
total 0

/tmp/hartzelg/spack-stage/spack-stage-3OlQZn:
total 0
[...]

Have I mis-understood something or am I tripping over a bug?

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions