The containerize feature should allow more customization of the container spec file.

Rationale

Consider the use case of an application container to be run multi-node over InfiniBand. The MPI library needs to be built with fabric support. E.g.,

spack:
  specs:
  - gromacs@2019.4 build_type=Release
  - openmpi@3.1.4 fabrics=verbs
  - fftw precision=float
  packages:
    all:
      target: [broadwell]

  container:
    format: docker

    base:
      image: "ubuntu:16.04"
      spack: develop

    strip: true

    os_packages:
    - libgomp1

This container build will fail because the IB verbs packages are not present in the container base image.

Description

The set of base images is highly restricted, so the option of using a custom base image with the IB libraries already included is not allowed.

The os_packages section applies only to the runtime stage of the spec, not the build stage. Regardless, the packages are installed after spack, not before as would be necessary in this case.

The extra_instructions build section is executed after spack, so it's also not an option.

Based on the failed approaches, there are 3, not mutually exclusive, potential containerize enhancements.

1. Remove the base image restriction. Perhaps display a warning if a custom base image is selected, but don't disallow it outright.

2. Extend the os_packages section to include build and runtime sections. E.g.,

  os_packages:
    build: libibverbs-dev
    runtime: libibverbs

The behavior should also be changed to install the packages before spack, not after.

3. Add a new extra_instructions section that is inserted prior to spack, e.g., prebuild or setup.

  extra_instructions:
    prebuild: RUN apt-get update -y && apt-get install libibverbs-dev 

My suggestion would be 2 to address this particular use case, but 3 would be a more general, if less user friendly, solution.