-
Notifications
You must be signed in to change notification settings - Fork 100
Append old master data to release branch for /api/public/get-benchmarks #753
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Append old master data to release branch for /api/public/get-benchmarks #753
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How did we get this number?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This number inherits from existing logic (from GO backend more earlier). Up to change if necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be a module or class level constant?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is only used in function level.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like a constant, how is this different from the one you defined in cocoon config?
Having the same value defined in several places makes an update of constant error prone.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we use a better name? from the name we don't know what it refers to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Replace with defaultBranch, and move it to cocoon config.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use config.maxRecords everywhere, you have three variables pointing to the same value.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rm and use the config.maxRecords.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Replaced maxRecords with config.maxRecords. But we need to keep variable masterLimit, which is used to track the number of master commits.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This variable name is really confusing I'd recommend using a different name for it. In this case is not clear if you want 'master' instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed to numberOfMasterCommits to make it clear.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please add docs for the parameters.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1 hour is too much, how did we get to this value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Like already mentioned in the comment doc: values are inserted into datastore later than commit. The workflow inserts commit first into datastore, suppose with timestamp1, and then inserts values of different tests results later once tests finish, suppose with timestampX. timestampX varies for different tests/commits, depending on test execution time and environment. Based on our infra metrics collected/analyzed, executing time of a commit row is less than 1 hour, that's why I put the 1 hour timeframe. If too small, we may miss some values.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding, subtracting time can cause race conditions(we don't control the time) should we fix the query to return only commits that have all the data instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess the current logic is designed like this to reduce datastore traffic. If we query timeseriesValue based on commit, then we need to make config.maxRecords requests each time the web refreshes.
With current logic, it needs to query datastore only once.
Also, I don't think race condition is an issue here. This API reads data all the time, rather than writes anything. If data is not there yet, it simply show empty value on the web.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My suggestion was to update the query to return only commits with complete results up to the passed timestamp. That should not affect the current performance and will eliminate the need to guess if a commit is complete based on estimates of the execution time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unluckily that is not possible due to datastore query limitation.
To achieve what you suggested, we need to add a commit list which contains all commits to filter out our interested records of timeseriesValue. However, dart db supports only simple filter options like = (https://github.com/dart-lang/gcloud/blob/master/lib/src/db/db.dart#L128).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a TODO: flutter/flutter#56731
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we place this class in models?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class is used to hold temporary results, without a general-wide usage purpose across the whole app. We may just keep it here loally.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case please document its purpose.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a constant for this value it will make easier to change in a single place.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to add tests that are adding records from master and one release branch?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
reports statuses with input branch is testing this case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case can we refactor the tests to cover more atomic cases? Having tests that are testing too much logic require to understand the entire workflow to fix them when they break because we don't know exactly what is failing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you please elaborate on what you mean by covering atomic cases. Do you mean test their sub functions individually? In that case, we may need to move those private functions public.
BTW: Each API in cocoon performs a specific purpose. It seems that all APIs' tests test their whole logic no matter the API is as complex as https://github.com/flutter/cocoon/blob/master/app_dart/lib/src/request_handlers/check_for_waiting_pull_requests.dart, or as simply as https://github.com/flutter/cocoon/blob/master/app_dart/lib/src/request_handlers/get_branches.dart.
To make the test more easily readable, I have added some documents.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sent the link to tests best practices by email, please read it and refactor accordingly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the link! Have added more atomic tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here and below, let's try to keep test names short. If we need to add full descriptions let's add it as docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use shorter test names, and put extra words to doc.
godofredoc
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some nits with very long test names, LGTM after making them short.
60358f3 to
f559ba1
Compare
format
2d290f5 to
8203159
Compare
Benchmark dashboards shows only commits of specific release branches right now. It would be easier to view performance trending/regression if we can have historical
masterdata prior to this release branch.This pr adds the above feature.