CI test job times are becoming exceedingly poor, exceeding 10min and often 30min. Other jobs are all < 3 minutes.

Here are some recent workflow runs:

• PR #886 at 33 minutes.
• PR #896 at 38 minutes.

which are both running against the next branch with caching enabled and cache is hit. The bulk of the time is taken to compile internal crates again:

   Compiling miden-objects v0.6.0 (/home/runner/work/miden-base/miden-base/objects)
   Compiling miden-lib v0.6.0 (/home/runner/work/miden-base/miden-base/miden-lib)
   Compiling miden-tx v0.6.0 (/home/runner/work/miden-base/miden-base/miden-tx)
   Compiling miden-bench-tx v0.1.0 (/home/runner/work/miden-base/miden-base/bench-tx)
    Finished `test-release` profile [optimized + debuginfo] target(s) in 10m 05s

which occurs 3 times within the job. So the vast majority of the time is compiling despite the cache working.

What's interesting is that without the cache these multiple compilations don't occur. This CI run for #901 onto main has a full compile time of ~10minutes which it only pays once. The other two steps for some reason don't require a recompile. Compilation should be a once off; step 1 is build, and 2 & 3 are different test invocations. So it appears that this concept of build once, run twice is no longer working on next.

This probably indicates a change in workflow, features or dependencies between main and next which causes this triple compile.

Summary of initial findings:

• next branch has some difference to main which causes it to recompile between steps (probably, this is an assumption based on the slim evidence above).
• Cache is being hit on next but its not the dependencies which take long to compile. Its our own crates which take 10 minutes and are set to release which takes even longer. Essentially a full compile is 10 minutes which is roughly the same with or without dependencies precompiled.

Some initial thoughts:

• Figure out why main and next recompile differently between steps,
• Find different optimisation level which takes less time,
• Investigate what is taking so long to compile in our crates - it might very well be link time which could a simple matter of using mold.
• Consider moving proof related tests into a separate job which might run on a schedule instead of on each PR