Skip to content

CI (Buildkite): use a TMPDIR that is backed by real storage#42995

Merged
DilumAluthge merged 1 commit intomasterfrom
dpa/bk-tmpdir
Nov 10, 2021
Merged

CI (Buildkite): use a TMPDIR that is backed by real storage#42995
DilumAluthge merged 1 commit intomasterfrom
dpa/bk-tmpdir

Conversation

@DilumAluthge
Copy link
Copy Markdown
Member

@DilumAluthge DilumAluthge commented Nov 8, 2021

No description provided.

@DilumAluthge DilumAluthge added ci Continuous integration backport 1.6 Change should be backported to release-1.6 backport 1.7 labels Nov 8, 2021
@DilumAluthge DilumAluthge requested a review from a team as a code owner November 8, 2021 00:23
@DilumAluthge DilumAluthge removed the request for review from a team November 8, 2021 00:24
@DilumAluthge
Copy link
Copy Markdown
Member Author

DilumAluthge commented Nov 8, 2021 •
edited
Loading

@staticfloat I am seeing the following test failure on several Buildkite jobs.

file                               (13) |         failed at 2021-11-08T00:42:22.921
Test Failed at /cache/build/amdci4-5/julialang/julia-master/julia-e6d4b33a81/share/julia/test/file.jl:113
  Expression: dirname(t) == d
   Evaluated: "/cache/build/amdci4-5/julialang/julia-master/tmp" == "/cache/build/amdci4-5/julialang/julia-master/tmp/jl_PnXa94"

Keno
Keno reviewed Nov 8, 2021
View reviewed changes
Comment thread .buildkite/pipelines/main/platforms/tester_linux.yml
unset JULIA_DEPOT_PATH
unset JULIA_PKG_SERVER

# Make sure that temp files and temp directories are created in a location that is
Copy link
Copy Markdown
Member

@Keno Keno Nov 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this required?

Copy link
Copy Markdown
Member Author

@DilumAluthge DilumAluthge Nov 8, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inside the Buildkite sandbox, the /tmp directory lives on a tempfs that is backed by 1GB of RAM.

On certain Buildkite jobs (e.g. when we run the Pkg test suite with the Pkg server disabled), we end up cloning a lot of Git repos, and as a result we end up running out of space in the tempfs.

While /tmp lives in the tempfs, the build directory (the current working directory to which the Julia repo has been cloned) is actually backed by a real filesystem. So we want to tell all programs to create temp files and temp directories inside our current working directory.

Copy link
Copy Markdown
Member

@Keno Keno Nov 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Increase the size of the tempfs?

Copy link
Copy Markdown
Member Author

@DilumAluthge DilumAluthge Nov 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could do that. I know that @staticfloat and @maleadt have expressed concerns about that in the past. We have multiple Buildkite agents on each machine, and we don't want to use up too much RAM.

This was referenced Nov 8, 2021
Merged
Merged
@DilumAluthge DilumAluthge mentioned this pull request Nov 9, 2021
Merged
@DilumAluthge DilumAluthge added the merge me PR is reviewed. Merge when all tests are passing label Nov 9, 2021
@DilumAluthge DilumAluthge merged commit f317d57 into master Nov 10, 2021
@DilumAluthge DilumAluthge deleted the dpa/bk-tmpdir branch November 10, 2021 03:45
@DilumAluthge DilumAluthge removed the merge me PR is reviewed. Merge when all tests are passing label Nov 10, 2021
KristofferC pushed a commit that referenced this pull request Nov 10, 2021
@DilumAluthge @KristofferC
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
9e11179 
(cherry picked from commit f317d57)
KristofferC pushed a commit that referenced this pull request Nov 10, 2021
@DilumAluthge @KristofferC
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
7000a62 
(cherry picked from commit f317d57)
KristofferC pushed a commit that referenced this pull request Nov 10, 2021
@DilumAluthge @KristofferC
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
0012a6a 
(cherry picked from commit f317d57)
KristofferC pushed a commit that referenced this pull request Nov 11, 2021
@DilumAluthge @KristofferC
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
c0dfb80 
(cherry picked from commit f317d57)
KristofferC pushed a commit that referenced this pull request Nov 12, 2021
@DilumAluthge @KristofferC
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
593922a 
(cherry picked from commit f317d57)
@KristofferC KristofferC removed backport 1.6 Change should be backported to release-1.6 backport 1.7 labels Nov 13, 2021
daviehh pushed a commit to daviehh/julia that referenced this pull request Nov 16, 2021
@DilumAluthge @daviehh
CI (Buildkite): use a TMPDIR that is backed by real storage (JuliaL…
fc55ebf 
…ang#42995)

(cherry picked from commit f317d57)
LilithHafner pushed a commit to LilithHafner/julia that referenced this pull request Feb 22, 2022
@DilumAluthge @LilithHafner
CI (Buildkite): use a TMPDIR that is backed by real storage (JuliaL…
064ff76 
…ang#42995)
LilithHafner pushed a commit to LilithHafner/julia that referenced this pull request Mar 8, 2022
@DilumAluthge @LilithHafner
CI (Buildkite): use a TMPDIR that is backed by real storage (JuliaL…
692669a 
…ang#42995)
staticfloat pushed a commit that referenced this pull request Dec 23, 2022
@DilumAluthge @staticfloat
CI (Buildkite): use a TMPDIR that is backed by real storage (#42995)
079b1f3 
(cherry picked from commit f317d57)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Keno Keno Keno left review comments

@staticfloat staticfloat Awaiting requested review from staticfloat

Assignees

No one assigned

Labels

ci Continuous integration

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@DilumAluthge @Keno @KristofferC