Skip to content

Conversation

@Piinks
Copy link
Contributor

@Piinks Piinks commented Feb 12, 2020

This change will add a Gold status check to all framework PRs that reflects the state of the current tryjobs run through Flutter Gold for the pr and commit.

This was originally attempted in flutter/flutter#49370 as a Cirrus check. That implementation was subject to timeouts from Cirrus, so it will be implemented here instead.

Adding this check will do a couple of things for improved workflow and productivity.

  • This will streamline framework changes.
    • Currently, framework tests fail when a golden file change is detected. After triaging on the Flutter Gold dashboard, the affected test shards need to be re-executed, which in some cases could take over an hour. A follow-up to this change in the framework would remove the failure case on individual test shards, making this check the gatekeeper. It will notify when a golden file change is detected, and once triaged, will update in 3 minutes or less.
  • This will help the engine roll go more smoothly when golden files are affected.
    • Currently, the engine auto-roller will repeatedly close attempts to roll into the framework since golden file shards fail. Without a notification or delay for images to be triaged and shards to be re-triggered, this often results in the need for a manual engine roll to check in golden file changes. This will stop the auto-roller from closing the pull request, leaving this check in a running state and notifying the engine sheriff to triage images on the dashboard. Once the images have been triaged, the auto-roller will proceed. Related issue: Golden file changes originating from flutter/engine require a manual roll flutter#48744

@Piinks Piinks changed the title WIP Adding gold status check to framework PRs Adding gold status check to framework PRs Feb 14, 2020
@Piinks Piinks marked this pull request as ready for review February 14, 2020 01:05
@Piinks
Copy link
Contributor Author

Piinks commented Feb 14, 2020

fyi @dnfield and @kjlubick to address flutter/flutter#48744

@Piinks Piinks requested review from dnfield and keyonghan February 28, 2020 00:13
Copy link
Contributor

@keyonghan keyonghan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it is easy to simplify the main function body by replacing with some more sub-functions? It might be easier to follow.

Comment on lines 63 to 68
const List<String> cirrusInProgressStates = <String>[
'EXECUTING',
'CREATED',
'TRIGGERED',
'NEEDS_APPROVAL'
];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! Thank you. I've made those a constant now, if we are going to use them in multiple places I am sure we don't want to have to track down all of them for updates. :)

'Found Cirrus build status for pull request ${pr.number}, commit '
'${pr.head.sha}: $taskName ($status)');

if (taskName.contains('framework')) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this is liable to get broken in some way down the road if, for example, web is using goldens and doesn't use framework in the task name.

Not sure how to fix that - maybe adopting a convention that tasks using gold have gold in the task name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed this. I thought some folks might be bothered by an irrelevant check on their PR since we try to keep the checks lean, but if they aren't running a tryjob then it will just go green on the first run.

log.debug('Last known Gold status for ${pr.number} was with sha '
'${lastUpdate?.head ?? 'initial'}, status: ${lastUpdate?.status ?? 'initial'}');

if (lastUpdate.head == null || lastUpdate.head != pr.head.sha) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This whole if/else block would read more clearly if it was broken up into a few functions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have refactored, I think it is much simpler now. :)

Copy link
Contributor

@dnfield dnfield left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants