### Problem to solve Engineering leadership (Director/VP/CTO) need to demonstrate the ROI from the investments in AI features. To quantify this impact they need to track the progress of Duo Root Cause Analysis adoption and what was the Impact of the SDLC perfromance. ### Overview **Hypothesis:** When pipelines are broken, end-user value (in the form of unshipped code) is blocked from moving into a production environment. The time it takes to troubleshoot and resolve broken pipelines directly correlates to Mean Time To Merge (MTTM) and Cycle Time (issue first mentioned in commit to issue being closed). At a more granular level, we can derive time savings by measuring the average time to resolve failed pipelines not using root cause analytics compared to the average time to resolved failed pipelines with root cause analysis. We can further measure adoption rates by counting unique users using RCA and RCA utilization rates on a macro level. **Important risk to be aware of:** If we expose these metrics to customers and there is no change in pipeline resolution time when using RCA, it will potentially negatively impact sales and adoption of GitLab Duo Enterprise. ### Propsal We can measure the impact of Root Cause Analysis with the following metrics: * **Root cause analysis of unique users** over `[insert period]` - we want to correlate the adoption of Duo RCA with decreasing lead and cycle time: * Count of returning users (unique user has used root cause analysis previously) * Count of new users (this is the first time the user has used RCA in `[insert period]`) * https://gitlab.com/gitlab-org/gitlab/-/issues/486523+s * **Total pipelines**: Count of pipelines run over `[insert period]` * **Failed pipelines:** Count of pipelines that failed over `[insert period]` * Measured by pipeline starts and pipeline fails before successfully completing. * Omit manually stopped pipelines and cancelled pipelines * https://gitlab.com/gitlab-org/gitlab/-/issues/486521+s * **Pipeline resolution time:** The average time it takes to resolve a pipeline after failure over `[insert period]` * Start event: A pipeline fails * End event: ~~The same pipeline passes with no failures~~ The pipeline for the same branch passes with no failures * **Root cause analysis pipeline resolution time:** The average time it takes to resolve a pipeline after failure when Root Cause Analysis was used over `[insert period]`. * Start event: A pipeline fails * Middle event: Root Cause Analysis button clicked * End event: The same pipeline passes with no failures * Display average time and % diff from pipeline resolution time without RCA * https://gitlab.com/gitlab-org/gitlab/-/issues/520332+s * **Improved Pipeline Success Ratio** (Measured through https://gitlab.com/gitlab-org/gitlab/-/pipelines/charts) * **Root Cause Analysis utilization:** The percentage of failed pipelines where RCA was used compared to overall failed pipelines over `[insert period]` * `Failed pipelines using RCA / Total failed pipelines` * https://gitlab.com/gitlab-org/gitlab/-/issues/520468+s * **Root Cause Analysis Time Saved:** Approximate hours/days saved by using root cause analysis * `(Pipeline resolution time (avg) - root cause analysis pipeline resolution time (avg)) * count of failed pipelines where root cause analysis was used to resolve the pipeline` * https://gitlab.com/gitlab-org/gitlab/-/issues/520469+s * **Correlated metrics:** * **Mean Time To Merge:** Average time from when an MR is opened to when an MR is merged. * **Cycle Time:** Average time from when a commit (or MR) first mentions an issue to when that issue is closed. Validation was done to \~6 customers, and they resonated well. ## Iteration Path - [ ] MVC1: https://gitlab.com/gitlab-org/gitlab/-/issues/486523+s - [ ] MVC2: https://gitlab.com/gitlab-org/gitlab/-/issues/486521+s - [ ] MVC3: https://gitlab.com/gitlab-org/gitlab/-/issues/486524+s - [ ] MVC4: **Improved Pipeline Success Ratio** - [ ] MVC5: **Root Cause Analysis utilization:** - [ ] MVC6: **Root Cause Analysis Time Saved:**