Skip to content

Fail the work item when the azure pipelines upload fails #7310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alexperovich merged 2 commits into from
Apr 29, 2021
Merged

Fail the work item when the azure pipelines upload fails #7310

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ff43194
Fail the work item when the azure pipelines upload fails
alexperovich Apr 28, 2021
dac3520
global
alexperovich Apr 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/AzurePipelines.MonoQueue.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
<PropertyGroup>
<HelixPostCommands Condition="$(IsPosixShell)">
$(HelixPostCommands);
/bin/sh $HELIX_CORRELATION_PAYLOAD/reporter/run.sh $(SYSTEM_TEAMFOUNDATIONCOLLECTIONURI) $(SYSTEM_TEAMPROJECT) $(TestRunId) $(SYSTEM_ACCESSTOKEN)
/bin/sh $HELIX_CORRELATION_PAYLOAD/reporter/run.sh $(SYSTEM_TEAMFOUNDATIONCOLLECTIONURI) $(SYSTEM_TEAMPROJECT) $(TestRunId) $(SYSTEM_ACCESSTOKEN) || exit $?
</HelixPostCommands>
<HelixPostCommands Condition="!$(IsPosixShell)">
$(HelixPostCommands);
call %HELIX_CORRELATION_PAYLOAD%\reporter\run.bat $(SYSTEM_TEAMFOUNDATIONCOLLECTIONURI) $(SYSTEM_TEAMPROJECT) $(TestRunId) $(SYSTEM_ACCESSTOKEN)
call %HELIX_CORRELATION_PAYLOAD%\reporter\run.bat $(SYSTEM_TEAMFOUNDATIONCOLLECTIONURI) $(SYSTEM_TEAMPROJECT) $(TestRunId) $(SYSTEM_ACCESSTOKEN) || exit /b
</HelixPostCommands>
</PropertyGroup>

Expand Down
2 changes: 2 additions & 0 deletions src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/run.bat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ GOTO :retryloop

echo %date%-%time%
%ENV_PATH%\Scripts\python.exe -B %~dp0run.py %*
set _uploaderExitCode=%ERRORLEVEL%
echo %date%-%time%

set PYTHONPATH=%_OLD_PYTHONPATH%
exit /b %_uploaderExitCode%
13 changes: 12 additions & 1 deletion src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,16 @@
import traceback
import logging
from queue import Queue
from threading import Thread
from threading import Thread, Lock
from typing import Tuple, Optional

from test_results_reader import read_results
from helpers import batch, get_env
from azure_devops_result_publisher import AzureDevOpsTestResultPublisher

workerFailedLock = Lock()
workerFailed = False

class UploadWorker(Thread):
def __init__(self, queue, idx, collection_uri, team_project, test_run_id, access_token):
super(UploadWorker, self).__init__()
Expand All @@ -33,13 +36,16 @@ def __process(self, batch):
self.__print('uploaded {} results'.format(self.total_uploaded))

def run(self):
global workerFailed, workerFailedLock
self.__print("starting...")
while True:
try:
item = self.queue.get()
self.__process(item)
except:
self.__print("got error: {}".format(traceback.format_exc()))
with workerFailedLock:
workerFailed = True
finally:
self.queue.task_done()

Expand All @@ -65,6 +71,7 @@ def process_args() -> Tuple[str, str, str, Optional[str]]:


def main():
global workerFailed, workerFailedLock
logging.basicConfig(
format='%(asctime)s: %(levelname)s: %(thread)d: %(module)s(%(lineno)d): %(funcName)s: %(message)s',
level=logging.INFO,
Expand Down Expand Up @@ -105,6 +112,10 @@ def main():
q.join()

log.info("Main thread exiting")

with workerFailedLock:
if workerFailed:
sys.exit(1337)

if __name__ == '__main__':
main()
Expand Down
4 changes: 3 additions & 1 deletion src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ fi

date -u +"%FT%TZ"
$ENV_PATH/bin/python -B $script_path/run.py "$@"
export _uploaderExitCode=$?
date -u +"%FT%TZ"

export PYTHONPATH=$_OLD_PYTHONPATH
export PYTHONPATH=$_OLD_PYTHONPATH
exit $_uploaderExitCode