This blog post discusses the strategies that Slack uses to manage the lifecycle (development, support, and eventual retirement) of infrastructure projects, through the lens of the migration through three successive internal \u201cplatform\u201d offerings.<\/p>\n
Circa 2020, our Cloud Engineering team (now evolved into multiple teams responsible for narrower aspects) was responsible for managing our Infrastructure-as-a-Service providers, compute environments like Chef and Kubernetes, and a large variety of related systems.<\/p>\n
This team had a broad remit, often with multiple systems and technologies serving overlapping purposes, and this broad remit caused several problems for us.<\/p>\n
These overlaps meant we would spend valuable time recreating new features in multiple projects, instead of writing new functionality just once.<\/p>\n
Retiring older technologies to focus on newer ones was challenging, because these older technologies still served important responsibilities, and we did not have a clear process for migration of those responsibilities to newer technologies.<\/p>\n
We also could not easily and consistently communicate with our peers about what they could expect if they chose a specific technology: Would we support them? Address bugs they found? Surprise them with a \u201cdeprecation\u201d warning a week after they went live?<\/p>\n
Not only did we not have clear answers to these questions, the un-clear answer a peer received would vary based on who on the team they talked to.<\/p>\n
Our peers didn\u2019t know which technologies they should \u2014 or could<\/i> \u2014 use.<\/p>\nGoals of a technology lifecycle<\/b><\/h2>\n
We set out to develop a model to help us track and talk about the technologies we supported, with a few goals in mind:<\/p>\n
We wanted to standardize the technology status and the expectations that we communicated to our peers, regardless of who they were talking to, but also across the technologies in our inventory.<\/p>\n
If we had two technologies that were slated for retirement, we wanted to handle incoming requests for them in the same way, and have concrete and executable plans for making forward progress to retire them.<\/p>\n
Without a framework to pin our technology status to, when our peers asked for advice about which they should select, our answers tended to be based in opinion \u2014 whether a choice is \u201cbad\u201d or \u201cgood\u201d, from the perspective of the answerer.<\/p>\n
We wanted instead to be able to concisely communicate important information about a technology \u2014 how we support it, what its future is, whether it is production ready or not, and what our peers might expect if they adopt it and then have feature requests or bug reports.<\/p>\n
This way, our customers could make their own decisions with factual information that\u2019s important to their own planning, rather than hard-to-weigh value judgments, or scattered and inconsistent factoids.<\/p>\n
Finally, the lifecycle would be a tool for us to retire technologies that we currently managed but for which there were (or would soon be) better options.<\/p>\n
Ultimately, our ability to fully retire a technology depends on the engineering teams that depend on it choosing to migrate off of it. Our peers in Engineering have the same full plates we do and the same planning cycles and constraints. So, by ensuring that the lifecycle is designed with these constraints in mind, we could be certain that as we move a technology through the lifecycle, our peers have ample time and forewarning to plan their own migrations or deprecations.<\/p>\n
Our project lifecycle has six stages. We expect most technologies to proceed through all six stages in the fullness of time; though many might skip stages for reasons discussed below.<\/p>\n
In each stage, we document what our peers can expect for:<\/p>\n
Technologies are assigned to a stage primarily through team consensus \u2014 we agree on: whether a system is feature complete and sufficiently bug free for broad use; whether we have the resources to provide support for it at a corresponding level; or in later stages, whether there is a feature-complete replacement and if we\u2019ve provided sufficient communication about retirement.<\/p>\n
Movement into and through later stages includes dialogue with the internal users of a system, for their awareness and to ensure that technologies we<\/i> think are a viable replacement do in fact meet their needs.<\/p>\n
Without further ado, the following stage descriptions are taken directly from our internal documentation, with some minor tweaks to wording or genericization of service names.<\/p>\n
An experiment. A new project or system that Cloud Engineering is working on. A pre-prototype, something that will very likely be discarded, but the “v2” might be Beta.<\/p>\n
General use is strongly<\/b> discouraged.<\/p>\nSupport Level<\/b><\/h4>\n
None. We reserve the right to summarily decline any bug reports or feature requests for Alpha projects, other than Security\/Compliance Bugs.<\/p>\n
Alpha projects will either advance to Beta or Retired, depending on the outcome of the experiment.<\/p>\n
A technology that Cloud Engineering is building that is intended to become an Active technology, but is not yet ready for general consumption and is considered Beta. A Prototype.<\/p>\n
Self-service production use is strongly discouraged. Limited production use in close consultation with Cloud Engineering. Available for non-production use to brave friends and early adopters.<\/p>\n
Supported only for the purpose of gathering feedback to advance the project. Documentation is likely sparse to nonexistent; Slack discussions are likely the best source of information. Any requests (other than Security\/Compliance) related to Beta systems will be classified as low priority and\/or scheduled on the project roadmap.<\/p>\n
Beta systems are on track to become Active. However, it’s always possible that these systems will instead go straight to Maintenance, or, very rarely, Deprecated.<\/p>\n
An Active technology is production grade, with staffed support from Cloud Engineering. Active technologies are our flagship technologies and products that we hope all of our friends across engineering will adopt.<\/p>\n
Active technologies are available for production use.<\/p>\n
Cloud Engineering sets aside resources to handle bug reports and feature requests for Active technologies. Requests will be prioritized according to normal procedures.<\/p>\n
Most Active technologies also have reserved resourcing to provide users of the systems with hands-on support and consultation. (Note that Cloud Engineering does not perform operational tasks for users of our services; but when you use Active technologies, we set aside time to work hand-in-hand with you to achieve your objectives.)<\/p>\n
Active technologies will eventually transition to Maintenance. Very rarely, an Active technology will transition straight to Deprecated, but never directly to Retired.<\/p>\n
Maintenance technologies are older technologies and projects. In general, a technology is moved to Maintenance when its replacement is late in the Beta stage, or very early in the Active stage.<\/p>\n
Maintenance technologies are available for production use. New use cases are discouraged but not prohibited \u2014 Active (or sometimes Beta) are preferred.<\/p>\n
Cloud Engineering sets aside resources to handle bug reports (but not feature requests) for Maintenance technologies. Requests will be prioritized according to normal procedures.<\/p>\n
Cloud Engineering may summarily decline feature requests for Maintenance technologies.<\/p>\n
Cloud Engineering will not in general provide hands-on support and consultation for Maintenance technologies, depending on the maturity of alternative systems. Cloud Infrastructure may scope such support to only designated office hours.<\/p>\n
Maintenance technologies may one day progress to Deprecated, or revived to Active if resources and prioritization allow.<\/p>\n
Deprecated technologies are those for which there is a feature-complete Active alternative.<\/p>\n
Frequently, these technologies have hard stop dates (for example, Ubuntu releases) that are external to Slack.<\/p>\n
Existing users of Deprecated technologies should migrate to Active technologies as quickly as possible.<\/p>\n
New use cases are prohibited from adopting Deprecated technologies.<\/p>\n
Minimal. Documentation is likely to be outdated or incorrect. Bug reports (other than Security\/Compliance) related to Deprecated systems will be classified as low priority.<\/p>\n
Cloud Engineering reserves the right to summarily decline feature requests for Deprecated systems.<\/p>\n
Deprecated systems will remain so for enough time that reasonable users will be able to migrate without undue haste. This typically means at least two full quarters after Deprecation (e.g., if a system is Deprecated during Q1, it will remain in this lifecycle state throughout the entirety of Q2 and Q3).<\/p>\n
After this time, and at Cloud Engineering’s discretion, the technology will be Retired.<\/p>\n
Retired systems are no longer supported and are not intended for use. Remnants may still exist, and it’s even possible that entire Retired systems will still function. However, Retired systems may fail at any moment, and if they do, the remnants will be deleted\/terminated, not repaired.<\/p>\n
Not available for any purpose. Do not use it for new use cases. Migrate existing use cases immediately.<\/p>\n
None. The only response even to Security\/Compliance requests will be to delete the related infrastructure. No bug reports or feature requests will be accepted.<\/p>\n
End of the line, folks. Get off here or you’re sleeping in the bus yard.<\/p>\n
We use the Technology Lifecycle framework as a communications and planning tool. It helped us migrate through three iterations of a platform-ized Compute offering by making it easier to communicate expectations with current and prospective customers of these platforms.<\/p>\n
Comparing our resource allocations to the technology\u2019s stage on the lifecycle also helped us see where we were over-investing or under-investing based on the projected future of a technology.<\/p>\n
We developed our first internal platform, BuiltBySlack; or \u201cBBS\u201d. This framework was rudimentary, consisting of a set of parameterized Terraform modules and Chef cookbooks used to deploy and run services bundled in an internally-developed packaging format.<\/p>\n
BBS helped us prove out the use case and appetite for a platform offering while also serving some pieces of critical infrastructure.<\/p>\n
After realizing some value from the abstraction and standardization BBS provided, we began working on a Kubernetes-based offering, \u201cBedrock\u201d.<\/p>\n
This first iteration of this, \u201cBedrock Classic\u201d, used many of the same infrastructure components as our current generation container platform, but required service owners to take responsibility for many more aspects of the container build and deploy process: Writing shell scripts to build images, push them to registries, and writing their own Kubernetes resources to run the images.<\/p>\n
We introduce the technology lifecycle. Bedrock Classic is described as Active in the first draft. BuiltBySlack is considered \u201cmaintenance\u201d. Its eventual replacement, Bedrock YAML, is Beta.<\/p>\n
BuiltBySlack is moved to Deprecated, since we and the remaining BBS customers agree that Bedrock is at feature parity.<\/p>\n
Bedrock YAML graduates to Active, now that it\u2019s ready for full production use. The Cloud Engineering team makes this determination, based on the experience of our Beta customers and the results of internal production readiness assessments.<\/p>\n
All BuiltBySlack customers have migrated off of BBS and onto Bedrock, and so BBS (and the packaging format it used) are moved to Retired.<\/p>\n
After many months of maintaining feature parity between Bedrock YAML and Bedrock Classic, that older iteration is moved to Deprecated. We communicate to the customers of Bedrock Classic that they should plan to migrate to Bedrock YAML.<\/p>\n
Through all of the stages of platform maturity (and a similar but more regimented process for, e.g., Ubuntu releases), a documented and concrete Lifecycle and Maturity Model for these products made it easy for us to plan for and communicate about our work; and ultimately to stop investing in technologies that did not fit in our vision of the future.<\/p>\n
Curtailing investments in systems that are slated to be deprecated and retired lets us focus a small and agile team onto high-impact work to support a substantially larger overall engineering team.<\/p>\n\t\t