Skip to content

Simplify appveyor.yml#12540

Merged
seanbudd merged 2 commits into
masterfrom
appveyor-split-scripts
Jul 1, 2021
Merged

Simplify appveyor.yml#12540
seanbudd merged 2 commits into
masterfrom
appveyor-split-scripts

Conversation

@seanbudd

@seanbudd seanbudd commented Jun 12, 2021

Copy link
Copy Markdown
Member

Link to issue number:

None

Summary of the issue:

If a developer wishes to use multiple appveyor.yml files, there would be a lot of code duplication. In order to make changes to the AppVeyor build process(es), inline powershell scripts must be edited. Inline powershell scripts are harder to lint and parse than individual files within IDEs.

Description of how this pull request fixes the issue:

Simplifies the appveyor.yml file by putting the powershell scripts into separate files.

Note to reviewers: I would recommend reading through these changes, and then compare them to the individual commits and README. The git bash script is a tool you can use to review the final changes or specific commits. I've noted some proposed changes of behaviour, but otherwise original behaviour is intended.

Changes:

  • Moved comments to appveyor\README.md and created an outline of AppVeyor proccess in the appveyor\README.md
  • Move inline ps into script files
    • adds $ErrorActionPreference = "Stop"; to the start of each except our test scripts which are allowed to fail
    • confirm with the following (git) bash script, run from the project root. This generates diff files optimised for review between masters appveyor.yml and our powershell scripts on HEAD. To see the diff at a specific commit, check it out, as the files specified by ls must be checked out locally.
    # the git diff argument -W ensures whole functions are preserved in the diff, to aid reviewing and give context
    # the git diff arguments -wb ignore all trailing and leading whitespace changes. this is ignored due to
    # indentation changes.
    #
    # perl (as it supports advanced regex - PCRE) replaces our large blocks of removed code with a snippet
    # the perl preserves the first, second, second last, and last lines of blocks of removed code longer than
    # 4 lines. This makes it easier to diff between the relevant functions and preserves context (such as the
    # appveyor.yml build phase we are in).
    export subLine='\n-[^\n]*';  # subtracted line regex
    export perlSed="s/($subLine)($subLine)(?:$subLine)+($subLine)($subLine)/\1\2\n-...\3\4/g";
    for file in $(ls appveyor/scripts/*.ps1 appveyor/scripts/tests/*.ps1)
    do
    git diff -Wwb master:appveyor.yml HEAD:$file | perl -0777 -pe $perlSed > $file-appveyor.yml-hide-removed.diff
    done
  • as the repo isn't cloned until after the init phase, setBuildVersionVars.ps1 was moved until after that, into the install build phase
    • this means we don't update the appveyor build number until after a successful clone
  • moved the translation comments check to the test_script build phase
  • moved some inline cmd to ps and cleaned up appveyor.yml
  • removed unnecessary exit commands
  • removed markdown from the build link message as AppVeyor and other applications don't support it - only GitHub

Testing strategy:

  • Ensure each part and type of build occurs successfully.
  • See review tools to ensure no unintended changes of behaviour: especially decryptFilesForSigning.ps1 as this acts upon secure files when on non PR builds.
  • Confirm intended changes of behaviour
  • Check build artifacts are there and the same as other PR builds (and eventual master builds).

Known issues with pull request:

None needed

Change log entries:

None needed

Code Review Checklist:

  • Pull Request description is up to date.
  • Unit tests.
  • System (end to end) tests.
  • Manual testing.
  • User Documentation.
  • Change log entry.
  • Context sensitive help for GUI changes.
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers

@AppVeyorBot

This comment has been minimized.

@seanbudd seanbudd force-pushed the appveyor-split-scripts branch from 2c06944 to 260bd1f Compare June 12, 2021 02:23
@AppVeyorBot

This comment has been minimized.

@AppVeyorBot

This comment has been minimized.

@seanbudd seanbudd force-pushed the appveyor-split-scripts branch from 4378ec0 to 6da20f9 Compare June 12, 2021 05:29
@AppVeyorBot

This comment has been minimized.

@seanbudd seanbudd force-pushed the appveyor-split-scripts branch 4 times, most recently from 77370e1 to 72cdb74 Compare June 12, 2021 08:02
@AppVeyorBot

This comment has been minimized.

@seanbudd seanbudd force-pushed the appveyor-split-scripts branch 3 times, most recently from d78931c to 5ab5ea5 Compare June 13, 2021 04:21
@seanbudd seanbudd marked this pull request as ready for review June 13, 2021 04:36
@seanbudd seanbudd requested a review from a team as a code owner June 13, 2021 04:36

@feerrenrut feerrenrut left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you plan for this to be merged (to preserve the movement history)? If so, it would be nice to add some reasoning to each of the commit messages.

In the description you say:

as the repo isn't cloned until after the init phase, setBuildVersionVars.ps1 was moved until after that, into the install build phase

  • this means we don't update the appveyor build number until after a successful clone

Can you elaborate on why the timing of build number increment should change?

The first commit would be easier to review if it was a straight move of code (no modifications). It's ok if this commit won't build, just make it clear in the commit message that it doesn't build and is part of a larger change. Could you move the modifications to a subsequent commit please?

I actually can review this with various incantations of --color-moved and --color-moved-whitespace but it is slow and likely error prone.

Comment thread appveyor.yml
Comment on lines -267 to -268
# The server side deploy code ('nvdaAppveyorHook') relies on artifacts, they must be uploaded first.
# For ordering of appveyor yml phases, see: https://www.appveyor.com/docs/build-configuration/#build-pipeline

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seem obvious, but mistakes have been made here before. I think it is a shame to lose the "why" provided in some of these removed comments. I think I'd prefer them in the yml file (for locality of reference) but if you're concerned with duplication and want them removed then at least make sure they are captured in the readme file.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preserving these inline comments here shouldn't be an issue but all old comments should be captured in the README. To get a real picture of the process, before making any changes to this file, one would have to refer (and reflect changes) to the README anyway.

If there are multiple appveyor-*.yml files though, I imagine the README could get confusing and information (or references to sections) might be better deferred to inline comments.

@seanbudd seanbudd marked this pull request as draft June 23, 2021 00:20
seanbudd added a commit that referenced this pull request Jun 23, 2021
* lint and reference powershell code moved to files from appveyor.yml
@seanbudd seanbudd force-pushed the appveyor-split-scripts branch from df9c5ea to 170fce6 Compare June 23, 2021 02:27
@seanbudd

Copy link
Copy Markdown
Member Author

Do you plan for this to be merged (to preserve the movement history)? If so, it would be nice to add some reasoning to each of the commit messages.

I've refactored this PR to be better as a merge commit. I'm planning on squashing all subsequent commits after the code move into "Simplify appveyor.yml (#12540)" - with summary/reasoning like the first bullet point.

Can you elaborate on why the timing of build number increment should change?

It's more a matter of necessity. Before our repo is cloned (which happens after init) we cannot be calling script files from our repo. Any code in init must be inline. I'm not aware of any need for the increment to happen before the clone, but I flagged this change just in case. No (new) actions on our end have happened as part of the build before we increment the build number.

The first commit would be easier to review if it was a straight move of code (no modifications). It's ok if this commit won't build, just make it clear in the commit message that it doesn't build and is part of a larger change. Could you move the modifications to a subsequent commit please?

The first commit is now generated as the result of the following script:

sed -n '25,60 p' appveyor.yml > appveyor/scripts/setBuildVersionVars.ps1
sed -n '66,80 p' appveyor.yml > appveyor/scripts/decryptFilesForSigning.ps1
sed -n '87,106 p' appveyor.yml > appveyor/scripts/setSconsArgs.ps1
sed -n '112,118 p' appveyor.yml > appveyor/scripts/tests/translationCheck.ps1
sed -n '125,134 p' appveyor.yml > appveyor/scripts/buildSymbolStore.ps1
sed -n '146,172 p' appveyor.yml > appveyor/scripts/installNVDA.ps1
sed -n '177,188 p' appveyor.yml > appveyor/scripts/tests/unitTests.ps1
sed -n '192,223 p' appveyor.yml > appveyor/scripts/tests/lintCheck.ps1
sed -n '227,239 p' appveyor.yml > appveyor/scripts/tests/systemTests.ps1
sed -n '244,244 p' appveyor.yml > appveyor/scripts/tests/checkTestFailure.ps1
sed -n '256,265 p' appveyor.yml > appveyor/scripts/uploadArtifacts.ps1
sed -n '271,306 p' appveyor.yml > appveyor/scripts/deployScript.ps1
sed -n '311,319 p' appveyor.yml > appveyor/scripts/pushPackagingInfo.ps1

sed -i'' -e '311,319 d' appveyor.yml
sed -i'' -e '271,306 d' appveyor.yml
sed -i'' -e '256,265 d' appveyor.yml
sed -i'' -e '244,244 d' appveyor.yml
sed -i'' -e '227,239 d' appveyor.yml
sed -i'' -e '192,223 d' appveyor.yml
sed -i'' -e '177,188 d' appveyor.yml
sed -i'' -e '146,172 d' appveyor.yml
sed -i'' -e '125,134 d' appveyor.yml
sed -i'' -e '112,118 d' appveyor.yml
sed -i'' -e '87,106 d' appveyor.yml
sed -i'' -e '66,80 d' appveyor.yml
sed -i'' -e '25,60 d' appveyor.yml

@seanbudd seanbudd marked this pull request as ready for review June 23, 2021 02:52
@seanbudd seanbudd requested a review from feerrenrut June 30, 2021 09:21
@seanbudd seanbudd marked this pull request as draft June 30, 2021 09:21
@seanbudd seanbudd marked this pull request as ready for review July 1, 2021 03:06

@feerrenrut feerrenrut left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good @seanbudd

- lint and reference powershell code moved to files from appveyor.yml
- setBuildVersion vars after init and repo clone
- move translation check to tests
- move some inline cmd into ps scripts
- remove redundant exit checks
- add appveyor readme
- remove markdown from build url message as it is not compatible with slack and appveyor
Comment thread appveyor.yml
- if ERRORLEVEL 1 exit %ERRORLEVEL%
- cd ..
- ps: appveyor\scripts\buildSymbolStore.ps1
- 7z a -tzip -r output\symbols.zip symbols\*.dl_ symbols\*.ex_ symbols\*.pd_

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seanbudd As this command now manually mentions the symbols directory in each file to be included, rather than changing directory to symbols first, the zip file now contains a subdirectory called symbols with everything inside.
This is not what our server expects, so when the zip file is extracted, we end up with a symbols directory within our symbol server.
We should change this line back to

cd symbols
7z a -tzip -r ..\output\symbols.zip *.dl_ *.ex_ *.pd_

Though I don't know the full reasoning as to why changing directories was avoided here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was just to simplify code. I will revert this.

seanbudd added a commit that referenced this pull request Jun 14, 2022
Summary of the issue:
The server expects the symbols archive to be structured as ./*.ex_ not ./symbols/*.ex_.
Due to changes in #12540, the symbols archive has been restructured to ./symbols/*.ex_.
This is because 7zip structures the archive based on relative paths.

Description of how this pull request fixes the issue:
Reverts the 7zip symbol build to the old behaviour
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants