make use of tempfile when writing stdout content for 'buildtest build' by shahzebsiddiqui · Pull Request #1605 · buildtesters/buildtest

shahzebsiddiqui · 2023-08-29T15:21:37Z

This PR will address an issue when running buildtest build command in parallel typically 2 background process will result in some error as shown below.

FileNotFoundError: [Errno 2] No such file or directory: '/Users/siddiq90/Documents/github/buildtest/var/output.txt'

[2]  + 27972 exit 1     buildtest bd -b tutorials/sleep.yml > /dev/null

With this PR it will address this bug, see example below

(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] buildtest bd -b tutorials/sleep.yml 1>/dev/null & 
[1] 29565
(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] buildtest bd -b tutorials/sleep.yml 1>/dev/null & 
[2] 29724
(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] 
[1]  - 29565 done       buildtest bd -b tutorials/sleep.yml > /dev/null
(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] 
[2]  + 29724 done       buildtest bd -b tutorials/sleep.yml > /dev/null

In the history we will see both runs captured

(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] buildtest hy list                                 
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ id ┃ hostname                     ┃ user     ┃ system  ┃ date                ┃ pass tests ┃ fail tests ┃ total tests ┃ command                                           ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0  │ DOE-7086392.vpn-dhcp.lbl.gov │ siddiq90 │ generic │ 2023/08/29 11:18:41 │ 1          │ 0          │ 1           │ /Users/siddiq90/Documents/github/buildtest/bin/bu │
│    │                              │          │         │                     │            │            │             │ ildtest bd -b tutorials/sleep.yml                 │
├────┼──────────────────────────────┼──────────┼─────────┼─────────────────────┼────────────┼────────────┼─────────────┼───────────────────────────────────────────────────┤
│ 1  │ DOE-7086392.vpn-dhcp.lbl.gov │ siddiq90 │ generic │ 2023/08/29 11:18:43 │ 1          │ 0          │ 1           │ /Users/siddiq90/Documents/github/buildtest/bin/bu │
│    │                              │          │         │                     │            │            │             │ ildtest bd -b tutorials/sleep.yml                 │
└────┴──────────────────────────────┴──────────┴─────────┴─────────────────────┴────────────┴────────────┴─────────────┴───────────────────────────────────────────────────┘

With this change there will be no such file $BUILDTEST_ROOT/var/output.txt instead this will be a temporary file

(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] ls $BUILDTEST_ROOT/var/output.txt
ls: /Users/siddiq90/Documents/github/buildtest/var/output.txt: No such file or directory

The output.txt file is copied over and still present

(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] ls -l $BUILDTEST_ROOT/var/.history/*/output.txt
-rw-r--r--@ 1 siddiq90  92503  10839 Aug 29 11:18 /Users/siddiq90/Documents/github/buildtest/var/.history/0/output.txt
-rw-r--r--@ 1 siddiq90  92503  10839 Aug 29 11:18 /Users/siddiq90/Documents/github/buildtest/var/.history/1/output.txt

We can still query the content so long as file exists

(buildtest)  ~/Documents/github/buildtest/ [write_buildtest_build_stdout_to_temporary_file] buildtest hy query 0 --output
╭────────────────────────────────────────── buildtest summary ───────────────────────────────────────────╮                                                                  
│                                                                                                        │                                                                  
│ User:               siddiq90                                                                           │                                                                  
│ Hostname:           DOE-7086392.vpn-dhcp.lbl.gov                                                       │                                                                  
│ Platform:           Darwin                                                                             │                                                                  
│ Current Time:       2023/08/29 11:18:38                                                                │                                                                  
│ buildtest path:     /Users/siddiq90/Documents/github/buildtest/bin/buildtest                           │                                                                  
│ buildtest version:  1.5                                                                                │                                                                  
│ python path:        /Users/siddiq90/.local/share/virtualenvs/buildtest-Ir4AdrfC/bin/python3            │                                                                  
│ python version:     3.10.12                                                                            │                                                                  
│ Configuration File: /Users/siddiq90/Documents/github/buildtest/buildtest/settings/config.yml           │                                                                  
│ Test Directory:     /Users/siddiq90/Documents/github/buildtest/var/tests                               │                                                                  
│ Report File:        /Users/siddiq90/Documents/github/buildtest/var/report.json                         │                                                                  
│ Command:            /Users/siddiq90/Documents/github/buildtest/bin/buildtest bd -b tutorials/sleep.yml │                                                                  
│                                                                                                        │                                                                  
╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                  
─────────────────────────────────────────────────────────────────────────  Discovering Buildspecs ──────────────────────────────────────────────────────────────────────────
                      Discovered buildspecs                       
╔════════════════════════════════════════════════════════════════╗
║ buildspec                                                      ║
╟────────────────────────────────────────────────────────────────╢
║ /Users/siddiq90/Documents/github/buildtest/tutorials/sleep.yml ║
╚════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  1
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  1
──────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 1
Invalid Buildspecs: 0
/Users/siddiq90/Documents/github/buildtest/tutorials/sleep.yml: VALID
Total builder objects created: 1
                                                                   Builders by type=script                                                                    
┏━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder        ┃ type   ┃ executor           ┃ compiler ┃ nodes ┃ procs ┃ description     ┃ buildspecs                                                     ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ sleep/d798dda7 │ script │ generic.local.bash │ None     │ None  │ None  │ sleep 2 seconds │ /Users/siddiq90/Documents/github/buildtest/tutorials/sleep.yml │
└────────────────┴────────┴────────────────────┴──────────┴───────┴───────┴─────────────────┴────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────── Building Test ───────────────────────────────────────────────────────────────────────────────
sleep/d798dda7: Creating test directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/sleep/sleep/d798dda7
sleep/d798dda7: Creating the stage directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/sleep/sleep/d798dda7/stage
sleep/d798dda7: Writing build script: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/sleep/sleep/d798dda7/sleep_build.sh
────────────────────────────────────────────────────────────────────────────── Running Tests ───────────────────────────────────────────────────────────────────────────────
Spawning 1 processes for processing builders
─────────────────────────────────────────────────────────────────────────────── Iteration 1 ────────────────────────────────────────────────────────────────────────────────
sleep/d798dda7 does not have any dependencies adding test to queue
In this iteration we are going to run the following tests: [sleep/d798dda7]
                                                 Test Summary                                                 
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder        ┃ executor           ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime  ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ sleep/d798dda7 │ generic.local.bash │ PASS   │ None None None                      │ 0          │ 2.495698 │
└────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /Users/siddiq90/Documents/github/buildtest/var/report.json
Writing Logfile to: /Users/siddiq90/Documents/github/buildtest/var/logs/buildtest_zwhm1zem.log

…t build'. This will avoid issue where multiple buildtest build commands are running in parallel since previously it was writing to same file

jscook2345 · 2023-08-29T18:45:00Z

buildtest/main.py

            build_history_dir = cmd.get_build_history_dir()

-            shutil.move(fname, os.path.join(build_history_dir, "output.txt"))
+            shutil.copyfile(


It might be a good idea to close the file before you copy it in case any write buffering or something else random happens.

make use of tempfile when writing stdout file temporary for 'buildtes…

5d07953

…t build'. This will avoid issue where multiple buildtest build commands are running in parallel since previously it was writing to same file

pull-request-size bot added the size/S label Aug 29, 2023

applying black

6063bba

shahzebsiddiqui merged commit 20ad993 into devel Aug 29, 2023

shahzebsiddiqui deleted the write_buildtest_build_stdout_to_temporary_file branch August 29, 2023 17:53

jscook2345 reviewed Aug 29, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

make use of tempfile when writing stdout content for 'buildtest build'#1605

make use of tempfile when writing stdout content for 'buildtest build'#1605
shahzebsiddiqui merged 2 commits intodevelfrom
write_buildtest_build_stdout_to_temporary_file

shahzebsiddiqui commented Aug 29, 2023

Uh oh!

jscook2345 Aug 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shahzebsiddiqui commented Aug 29, 2023

Uh oh!

jscook2345 Aug 29, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants