Polyrepo Workflows (#882) · Epics · Epics · GitLab.org

Polyrepo Workflows

*This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*  --- ## Problem to solve When an application lives in a single repository it is easy to test and integrate all the components in a single CI pipeline, making it simple to keep `master` green and be confident that any point if working. But software development often consists of many services and/of components that are need to be integrated to ship complex applications, but for reasons of scale, access control and history, it may not be possible or advisable to have many hundreds of projects operating in the same Git repository and the same GitLab project. Like when building an application in a single repository, I want to be able to be able to have a "master branch" for my application(s) that spans multiple projects that I can always keep green, and easily checkout any iteration of the "master branch" and be confident it works. # Polyrepo Workflows ## Further details Most software products are composed of many components that are spread across multiple projects/repositories. For example, GitLab itself has separate repositories for `gitaly`, `gitlab-workhorse` and `gitlab-shell`. Changes to these are coordinated by changing each component separately and integrating them into the `gitlab-ce` repository by incrementing the version. Consider a change to `gitaly` and `gitlab-ce` - although each of these changes exists in it's own merge request, it isn't easy for a reviewer to integrate and test these changes as a whole because no version has been released yet. However, releasing and managing versions of many components across many projects that consume them creates friction and slows velocity. Imagine a multi-billion dollar company with 10 products with 50 shared components and 20,000 engineers trying to release without breaking one or more of the products. Releasing a new version of each package, upgrading in the 10 products is a slow painful cycle. For this reason, a number of companies who are in this situation try to emulate the approach taken in smaller project where all components across all applications are continuously integrated. * all 10 products use the latest version of all 50 components * checking out any release, checks out the complete state of the product and it's dependencies at that version Some enterprise SCM tools achieve this with a global monotonically increasing commit ID, where essentially the whole server is a single Git repository. ## Sub-Problem Areas ### Identify all of the repositories that are part of the "product" [&17278](https://gitlab.com/groups/gitlab-org/-/epics/17278) Part of the structure of a polyrepo workflow is that all of the projects which make up the entire product need to be identified. This is typically done with a manifest file which contains details about each product's location and additional information about branches/SHAs in use by the default build. ### Tying changes across projects to a single piece of metadata [&17289](https://gitlab.com/groups/gitlab-org/-/epics/17289) The underlying theme of this problem is that once you produce changes across multiple repositories, GitLab needs a way to identify all these separate changes as a single object. This single object is required for understanding the merge requests, CI Jobs, and other aspects of the GitLab platform. When changes exist across multiple projects, a dashboard of "change sets" of merge requests is helpful to understanding the overall object. This unified view doesn't necessarily need to be a single "merge request" to review all the changes, but could be more of a status page that lists all the merge requests, their pipeline status, merge checks, approvals, etc... ### Merging all changes from projects in single action [&881](https://gitlab.com/groups/gitlab-org/-/epics/881) When all of the changes have been evaluated through merge requests and pipelines they all need to be merged through a single action. This action needs to support merging all of the changes at the same time since they all depend on each other for the overall application to work successfully. ### Reverting all changes from projects in single action [&17290](https://gitlab.com/groups/gitlab-org/-/epics/17290) In the event that a change needs to be reverted that has already been merged, all the changes that were previously part of a single object need to be reverted as a single object. This is to ensure that the changes come out of all places where they were made. ## End-to-End Workflow The GitLab polyrepo solution provides an integrated workflow for managing development across multiple repositories. Here's how the components work together to deliver a seamless experience: ```mermaid flowchart TD %% Simplified styling classDef setup fill:#fcf1d1 classDef dev fill:#d2e9fc classDef review fill:#e9d3fc classDef build fill:#d3fcde classDef integrate fill:#fcd3d3 classDef maintain fill:#fce6d3 %% Core nodes Product["#17278: Identify Projects that Make Up the 'Product'"] ChangeSet["#17289: Identify Changes via Single Piece of Metadata"] Review["#457: Review Changes Across Multiple Projects"] Build["#883: Build Changes with Centralized Pipeline"] Merge["#881: One-click Merge of Change Object"] Revert["#17290: Revert Support for Change Object"] %% Flow connections Product --> ChangeSet ChangeSet --> Review ChangeSet --> Build Review --> Build Build --> Merge Merge --> Revert Revert -.-> ChangeSet %% Apply classes class Product setup class ChangeSet dev class Review review class Build build class Merge integrate class Revert maintain ``` ### 1. Setup: Define Your Product Structure ([#17278](https://gitlab.com/groups/gitlab-org/-/epics/17278)) - Define which repositories constitute your complete "product" - Import existing project relationship definitions (like manifest files) - Establish dependency relationships between components - Create a first-class "Product" entity in GitLab ### 2. Development: Track Related Changes ([#17289](https://gitlab.com/groups/gitlab-org/-/epics/17289)) - Create a "Change Set" object to track related changes across repositories - Group multiple merge requests under a single logical change - Provide visibility into all components affected by a feature or bug fix - Tag changes with consistent metadata for tracking ### 3. Review: Evaluate Changes Holistically ([#457](https://gitlab.com/groups/gitlab-org/-/epics/457)) - Review all related changes across repositories in a unified interface - See the complete impact of changes across component boundaries - Coordinate approvals across interdependent changes - Maintain visibility into the broader system context of changes ### 4. Build & Test: Validate Against the Complete Product ([#883](https://gitlab.com/groups/gitlab-org/-/epics/883)) - Run centralized pipelines that build the entire product - Validate that all interdependent changes work correctly together - Coordinate CI/CD across repositories with dependencies - Ensure changes don't break cross-component functionality ### 5. Integration: Merge Changes as a Unit ([#881](https://gitlab.com/groups/gitlab-org/-/epics/881)) - Merge all related changes with a single action - Ensure atomic updates across repositories - Prevent partial application of interdependent changes - Maintain system integrity with coordinated merges ### 6. Maintenance: Support System Evolution ([#17290](https://gitlab.com/groups/gitlab-org/-/epics/17290)) - Revert multiple related changes as a unit if issues are discovered - Maintain system consistency during rollbacks - Ensure complete undoing of problematic changes across repositories ### Component Dependencies Each component in the workflow builds upon preceding components: - **Change Set Tracking** (#17289) depends on **Product Definition** (#17278) - **Cross-Project Review** (#457) depends on **Change Set Tracking** (#17289) - **Centralized Pipeline** (#883) depends on **Change Set Tracking** (#17289) - **One-Click Merge** (#881) depends on **Centralized Pipeline** (#883) and **Cross-Project Review** (#457) - **Revert Support** (#17290) depends on **One-Click Merge** (#881) ### Key Use Cases #### Automotive Software Development - **Industries:** Automotive, transportation systems - **Challenges:** Hardware variants, complex component interdependencies, safety-critical systems - **Benefits:** Consistent software across vehicle models, coordinated updates, reliable system integration #### Embedded Systems - **Industries:** IoT device manufacturers, hardware vendors - **Challenges:** Constrained resources, hardware/software co-development, variant management - **Benefits:** Simplified cross-component development, reduced integration issues, consistent firmware #### Large-Scale Applications - **Industries:** Enterprise software, cloud services - **Challenges:** API compatibility, distributed ownership, service dependencies - **Benefits:** Coordinated API changes, holistic feature development, system-wide consistency ## Customers * https://gitlab.my.salesforce.com/0016100001E1liJ?srPos=1&srKp=001 * https://gitlab.my.salesforce.com/00161000004bZPD ~"customer+" * https://gitlab.my.salesforce.com/00161000004yLEy ~"customer+" * https://gitlab.my.salesforce.com/00161000003bXej ~"customer+" - has invested significant effort implementing this into Gerrit * https://gitlab.my.salesforce.com/0016100000sPtsc ~customer - has invested significant effort implementing this into GitLab using Git submodules for creating the relationship between projects and defining the integration pipeline * https://gitlab.my.salesforce.com/00161000002xBfL ~customer * https://gitlab.my.salesforce.com/0016100000fDO7w ~customer * https://gitlab.my.salesforce.com/00161000017VeNW ~customer - source https://gitlab.com/gitlab-com/sales/issues/153 * https://gitlab.my.salesforce.com/0016100001PrBWK ~customer ## Links / References * [Multi-project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipelines.html) * [UX problem validation](https://gitlab.com/gitlab-org/ux-research/-/issues/972)

epic