Skip to content

FileSourceHandler is slow for large sources #1175

@enriqueesanchz

Description

@enriqueesanchz

Bug Description

When doing snapcraft pull in a riscv64 emulated machine (Ubuntu 24.04.2 6.14.0-15-generic) it downloads the part in ~30 mins from the source. We can see that the process is still writing the file after that and it takes ~6 hours to complete it.

tcpdump produces no output when we finish the download. We can see the file size increasing during that ~6 hours:

$ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz
-rw-r--r-- 1 root root 3304247296 Jul  7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz
$ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz
-rw-r--r-- 1 root root 3306123264 Jul  7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz
$ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz
-rw-r--r-- 1 root root 3307675648 Jul  7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz
$ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz
-rw-r--r-- 1 root root 3310784512 Jul  7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz

As a workaround I changed snapcraft.yaml to use the nil plugin with source: . and wget & tar the file in the override-build step. This takes ~1.5h.

What is snapcraft pull doing, shouldn't it be equivalent?

To Reproduce

To reproduce the behavior you can use a launchpad riscv64 builder from production or staging and trigger a build using the snapcraft.yaml below.

Environment

Snapcraft is running in an emulated riscv64 machine using Ubuntu 24.04.2 6.14.0-15-generic. It runs inside an LXC container. This is the setup from launchpad riscv64 builders.

snapcraft.yaml

name: llvm-chromium
version: "21"
summary: Clang used by Google's build of the Chromium web browser
source-code: https://code.launchpad.net/~chromium-team/+git/llvm-chromium/+ref/master
description: |
 Chromium and Chromium-ffmpeg snaps build with LLVM, but upstream relies on
 bleeding edge LLVM, in particular Clang. LLVM from the Ubuntu archive often
 cannot keep pace, therefore this snap.

 This snap exposes no commands. To use it, invoke the binaries directly,
 e.g. /snap/llvm-chromium/current/bin/clang or use the environment, i.e.

 * CC=/snap/llvm-chromium/current/bin/clang
 * CXX=/snap/llvm-chromium/current/bin/clang++
 * AR=/snap/llvm-chromium/current/bin/llvm-ar
 * ...
confinement: strict
base: core24

parts:
  llvm:
    plugin: nil
    # Roll when CLANG_REVISION changes.
    source: https://commondatastorage.googleapis.com/chromium-browser-official/chromium-135.0.7023.0.tar.xz
    build-packages:
      - chrpath
      - cmake
      - elfutils
      - g++
      - gperf
      - ninja-build
      - pkg-config
      - python3-pkg-resources
      - quilt
      - wget
      - yasm
    override-build: |
      # Find all patches that don't apply anymore before running Quilt, as it
      # stops at the 1st unappliable and doesn't tells us how the file looks.
      d=$CRAFT_PROJECT_DIR/patches
      while IFS= read -r p; do
        < "$d/$p" patch --dry-run -p1 -o "$d/$p.out" || cat "$d/$p.out"
      done < "$d/series"
      QUILT_PATCHES=$d quilt push -a

      # --pic for building shared library
      python3 tools/clang/scripts/build.py --skip-checkout --bootstrap \
              --disable-asserts --pgo --without-android --without-fuchsia \
              --use-system-cmake --with-ml-inliner-model= --pic --no-tools
      #Restore this when LP:2061075 is fixed
      #version="$(python3 tools/clang/scripts/update.py --print-revision|head -c31)"
      ## Trailing 'e' avoids "invalid snap version: must end with an ASCII
      ## alphanumeric or one of '+' or '~' (and not '-')"
      #craftctl set version="$version"e
      #craftctl set version=llvmorg-19-smth-smth
      arch=$(uname -m)
      # Save space by removing libraries from other architectures
      # XXX: bashism
      shopt -s extglob
      rm -rf third_party/llvm-build/Release+Asserts/lib/clang/*/lib/!(${arch%?}*gnu*)
      mkdir -p $CRAFT_PART_INSTALL
      mv third_party/llvm-build/Release+Asserts/bin $CRAFT_PART_INSTALL
      mv third_party/llvm-build/Release+Asserts/lib $CRAFT_PART_INSTALL

Relevant log output

2025-07-07 09:58:33.506 verify plugin environment for part 'llvm'
2025-07-07 09:58:33.546 Pulling llvm
2025-07-07 09:58:33.565 execute action llvm:Action(part_name='llvm', step=Step.PULL, action_type=ActionType.RUN, reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None))
2025-07-07 09:58:35.165 Downloading '/build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz'

Additional context

No response

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions