Design Doc: Changelog Triggered Only Runs

Solves: #298 (comment)

Test full sweep: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20081821674
Test with changelog: https://github.com/InferenceMAX/InferenceMAX/actions/runs/20218553330

Introduction and Motivation

InferenceMAX prides itself on running regularly to keep up with the fast pace of software changes. However, running software release builds & release candidate builds don’t typically happen on a _nightly _basis and as such, it is unnecessary to perform a full-sweep every single night. Instead, it is sufficient to run only the configs (from the master configs) whose performance is affected as a result of any particular change.

Implementation

After the major refactors (#251 and #145), all run information is defined in source of truth master configuration files. These files are then parsed via the utils/matrix_logic/generate_sweep_configs.py script, whose JSON output is subsequently passed to a benchmark-*-tmpl.yml workflow. This sets up this proposal nicely, as each configuration has a “key” string associated with it. We propose a high-level “performance changelog” at the root of the repo (perf-changelog.yaml) that developers will use to specify config keys whose performance is affected in a particular PR.

The main change here can be summarized as follows: instead of running each config every night, configs will only be run on an “as needed” basis (i.e., when a change is made that affects their performance). This will more efficiently use compute resources while leading to a faster feedback cycle.

Example Developer Workflow

As an example, suppose a developer updates some code that will affect the performance of the dsr1-fp4-b200-sglang config. Perhaps they change the version of the SGLang serving image. Then, the developer should append to the end of the perf-changelog.yaml and add the following:

Then, upon merge to main, the run-sweep.yml workflow will be triggered (if and only if a change to perf-changelog.yaml has been detected). This workflow will subsequently run a script called process-changelog.py. This script takes in the following as arguments:

• Path to the perf changelog file
• A base Git ref (i.e., main/HEAD~1)
• A head Git ref (i.e., main/HEAD)

The script then gets the diff between the base perf changelog and the head perf changelog. If there is any negative diff, the script fails and so does the workflow (there should never be any negative diff as this changelog is meant to be a running changelog of all perf changes, and it runs from the bottom). The positive diff is then pre-processed to remove the line diff indicators, and then loaded to a list of dicts representing the changes. Then, a function retrieves the list of all config keys to run and passes them to generate_sweep_configs.py with the new subcommand test-config. This subcommand simply takes in a list of config keys (from the master config YAMLs) and retrieves the cases to run for each config. Once all results are retrieved, we organize them by sequence length and single/multi node and then dump them to stdout and direct them to an output of the invoking job in the workflow.

Workflow Mechanisms

The run-sweep.yml workflow will be the main workflow used for running “official” sweeps. As described above, this runs on any merge to main where there is an _addition _to perf-changelog.yml. This workflow begins with this job that retrieves which jobs to run based on diff: \

The output of process-changelog.py will be split by single/multi node and sequence length. Recall this is due to the 256 matrix generation limit that GitHub Actions imposes. If this limit didn’t exist, we would only split by single and multi node. We note that this implementation is not very scalable in the event we add more sequence lengths and/or more models (if we add more sequence lengths, we will continue having to hard-code new jobs sweep-single/multi-node-XkYk for each one, if we add new models, then we will have to further split by model as adding new models will increase the number of configs for each scenario described above). We hope to have a more permanent solution to this in the near future (that will likely involve introducing a sub-workflow). However, this is not as big of an issue, since we don’t anticipate needing to run “full” sweeps often, if at all.

We also note that the GitHub Action engineers are close to implementing lazy loading for workflows with many jobs, so the 10s loading timeout will hopefully no longer be an issue.

Testing

In order to ensure that sweeps can be tested and validated before merging and running the “official” sweep, developers can enable sweeps on their pull request by ensuring three things:

1. The PR is marked as “ready for review”
2. The PR includes valid changes to perf-changelog.yaml
3. The PR includes the label sweep-enabled

This will trigger the run-sweep.yml workflow on the PR branch.

Ensuing Frontend Changes

These changes would have many frontend implications. First we will describe the high-level considerations (such as what the UI displays) and then we will discuss the low-level considerations (how data is scraped from the artifacts).

In order to not delay main InferenceMAX development, the plan is to “freeze” the InferenceMAX UI on the December 9th run and implement the frontend changes over the course of the following week.

High-Level Considerations

Currently, the InferenceMAX frontend displays runs for each date, with an option to view historical runs via a date selector. This method assumes that a full sweep of runs is being run each day. However with this proposal, we will be running configurations relatively infrequently (only when necessary). Therefore, it doesn’t make sense to display runs for each date in the date dropdown selector. We propose the following: for the “Select a run date” dropdown calendar, only make clickable the dates on which a run was triggered. When this date is selected, there are two scenarios:

1. For the configs that were _unchanged _as a result of the triggered run (i.e., the configs that were not specified in the perf-changelog.yaml change associated with this run), carry-forward the data points from that config’s most recent run. One thing to note here is that we ought to make sure that all runs are completely successful (i.e., all jobs completed successfully) so as not to carry-forward incomplete run data.
2. For the configs that were _updated _as a result of the triggered run, display the new data points associated with the run.

With this strategy, we should differentiate the data points/frontier line in a way such that it is obvious to the user, on any given date, what data is _new _and what data is carry-forward. Implementation details will be left up to Akira.

We also should get rid of reliability data (at least in its current form). If we want to keep it, we can simply display “success rate per GPU sku” and filter over “last X runs,” or something like that. But even so, it appears reliability sort of loses its meaning now that we aren’t running every night. Now when running via a trigger, we will want to test rigorously _before _merging such that the “official” run is successful.

Low-Level Considerations

Recall, we are currently running three separate workflows each night (separated by sequence length, 1k1k, 1k8k, and 8k1k). This is shown here. With this proposal, we will be running _all _desired configs in a _single _run. The resulting artifact will be called results_all (as opposed to being split up by model and sequence length e.g., results_dsr1_1k1k). This shouldn’t be too difficult to implement, in fact it should make things easier since we won’t have to aggregate all results into a single file.

Further, we need to ensure backwards compatibility such that the old workflows can still be processed correctly.

Additional Considerations

We plan on back-filling the perf-changelog.yaml to reflect perf changes since the launch of InferenceMAX v1. We will not run CI on these back-filled changes – it is just so that the changes will be reflected. In order to accomplish this, we need to add a mechanism to the run-sweep.yml workflow in order to skip running the actual CI pipeline upon merge to main. We add the following condition to the setup job:

Now, if the merge commit contains the “[skip-sweep]” string, CI will not be run. This will allow developers to “force” make alterations to perf-changelog.yaml without actually running CI.