I think I follow what is going on with this minimum_variance and the associated close_to_maximum_score criteria, but I'm not clear on what the benefit for this "closeness" check is. It seems to be adding a fair bit of complexity to this logic and was a bit hard to follow. Can you help me understand what we get in return for this additional complexity?

Lots of the health score stuff is relative to other features in the codebase. Because of this, it becomes hard to get a "perfect" score as features start to converge. So if a given feature is "close" in a given metric to the top scoring feature, it should be awarded full points. For example, if there are four features with test coverages of [97, 98, 98, 99], then then 97% test coverage is technically in the bottom 25% of all features -- but it's still really good so it should be awarded full points. That's the general idea behind this implementation, which was more or less copied over from the metrics.js file.

That example is helpful to understand the real world scenario in which this logic provides more valuable feedback/insight to our users. I wonder if there is a place in the code that this or a similar example could be shared to highlight our intention behind this logic. It seems like something that would help future engineers by keeping in mind as they modify this logic.

If an engineer runs the docs CLI command without having first run the additional_metrics CLI command, it doesn't look like we raise any sort of error but in this case it appears that this key will receive a value of null rather than a fully initialized payload of the sort shown here. It seems like this would likely cause the documentation site to raise an error.

Do we need to support this key being assigned a value of null? If so, should we make that the value of this feature to ensure engineers are prompted to test and consider this scenario when working on the documentation site in isolation? If not, should we raise an error in this case when the docs CLI command is triggered without additional test metrics having been generated?

I'm seeing the documentation site fail to load when it is generated without the additional_metrics CLI command being run:

It seems like we either need to make this content optional, aligning with the behavior of the test_pyramid and test_coverage commands, or execute these additional metrics calculations automatically as part of the docs command.

Let's keep additional_metrics as a separate cli command because it seems useful for devs to be able to run that locally. It's a good idea to add the creation of additional_metrics to the docs command. @hstrowd updated here: a5b50ea

Is there a reason this operation is being performed as an independent CLI command rather than as part of the existing docs command?

I believe we were just trying to follow the same implementation pattern as other commands like: validate, test_pyramid, and test_coverage.

My understanding was that the test_pyramid and test_coverage commands were isolated because they have external dependencies (i.e. requiring test summary files and regression test details or requiring CodeCov credentials/configurations to be provided). These are optional commands that can be run to inject additional insight into the documentation site but are not provided.

This use case seems a bit different, since we're just repackaging existing information that was generated/collected as part of previous commands. For that reason, it seems more reasonable to consider performing these operations automatically as part of the docs operation. However, I'm not tied to that approach. If we want to keep it separate we should probably just allow the documentation site to be successfully generated when it has not been executed.