This proposal on how the deployment of the package registry and package storage should change.
Problem
With the current way the package-storage is deployed, there is no easy way to have different environments for production, QA or snapshots. If someone builds a package and wants to share it for testing with others, the tester is required to setup the testing environment local and run it with a local registry.
There is also no process to have certain packages in a QA stage before they are shipped to production.
Goals
The following proposal dives into how the above can be solved. But even though multiple stages of deployment are introduced, it should still be as easy as possible to release a new package. The different stages of deployment can be followed but it is not a strict requirement.
Note: At the moment we have an additional "experimental" environment which goes away in the near future and is ignored on purpose in this proposal.
Environments
To make testing and snapshot builds possible, three environments are needed:
- production
- staging (QA)
- snapshot
Each of these environments is tied to a specific version of the package-storage and the package-registry and has different rules on how packages are added. These are described in more details below.
Snapshot environment
The snapshot environment is to share the latest versions of packages and testing it together with the snapshot builds of the Elastic Stack. New packages are added and updated at any time and in most cases fully automated. In the snapshot branch it is allowed to overwrite / update existing versions.
Addition of packages are done through direct commits to the branch. It is expected, that any package updates pushed to the branch are already pre-checked by CI by the contributor.
The branch used for the snapshot packages is called snapshot in the package-storage repositories. The related package-registry is the one in the master branch. Every time a new commit is pushed to the package-storage snapshot branch or the package-registry master branch a new build is kicked off. The build is expected to pass as precheckes should have happened, if not, the contributors are pinged and deployment does not happen.
Taking the integrations repository as an example, every time a PR is merged, a script will trigger a commit to the snapshot branch with the updated version. This will trigger a build and a new registry is deployed.
The snapshot registry is available under epr-snapshot.elastic.co.
Snapshot is NOT a package development environment. All changes and tests should happen outside the snapshot branch and only the final result of changes is pushed.
The snapshot registry is a combination of production + staging + snapshot packages. If the same version exists in production and snapshot, the production version is taken as these should not conflict. Having all packages together in snapshot also allows to do upgrade tests.
Staging environment
The staging environment is meant for testing packages which are ready for deployment. Talking in terms of directory, moving to staging is moving a directory from snapshot to staging, meaning the package will be removed from snpashot. In most cases, when a package enters staging, it is expected not to change anymore. But if issues are found, changes can be picked again from the snapshot branch. A script is expected to do the work taking a package from snapshot and pushing it to staging.
From the deployment perspective it is the same as the snapshot build, it is continously built.
It links to a specific release version of the registry. If the registry is updated, the registry reference has to be updated manually. This is to ensure all CI builds are consistent.
The staging environment is expected to be used on SNAPSHOT branches.
The url used is epr-staging.elastic.co.
Production environment
The production branch is used for all the released packages. These packages should never be changed / updated after they are released. Contributions to the production branch happend through a PR to make sure CI checks can be run in advance. These PR are normally opened by a script taking a package from the staging branch. As soon as it is merged into production, the staging version should be removed. For now the merging of these PR's need to happen manually, but it could be automated if CI is green for "trusted" contributors.
The registry is tied to a specific release tag. As soon as a PR is merged in the production branch, it is deployment automatically to epr.elastic.co.
Summary
In summary, the above gives us 3 different environments. Deployments of each environment happens fully automated as soon as a commit is added to the branch. Only in the case of production, the addition of a package has to go through a PR to have a previous CI check guaranteed.
Below is a summary table of the 3 environments:
|
Snapshot |
Staging |
Production |
| URL |
epr-snapshot.elastic.co |
epr-staging.elastic.co |
epr.elastic.co |
| Add package |
Commit |
Commit |
PR |
| Version overwrite |
yes |
if needed |
no |
| Stack Version |
*-SNAPSHOT |
*-SNAPSHOT |
Released (candidates) |
| Registry Version |
master |
Stable release |
Stable release |
| Branch |
snapshot |
staging |
production |
| Packages |
snapshot+staging+prod |
staging+production |
production |
| Release |
Automated |
Automated |
Automated |
| Docker image |
snapshot |
staging |
production |
How to get to this new deployment
Today all packages are in the master branch. To make the above possible, the integrations repository script has to be adjusted and an additional script has to be built, to move packages between the different version.
On the Kibana side, a way must be found the SNAPSHOT builds point to a different version of the registry then development. Or the production one could be just the default and it needs manual adjustment.
Why Git and branches
Instead of using Git for the above, it would also be possible to just use a directory structure on S3 or similar. But the nice thing about Git is that it shows us exactly what happened when and by who in the past, and in extreme casees allows us to roll back changes if needed. In addition for the production registry it allows us to the use a manual PR review to have CI checks in advance. The part that is a bit more combuersome is moving packages between branches, but scripts can do this for us.
This proposal on how the deployment of the package registry and package storage should change.
Problem
With the current way the package-storage is deployed, there is no easy way to have different environments for production, QA or snapshots. If someone builds a package and wants to share it for testing with others, the tester is required to setup the testing environment local and run it with a local registry.
There is also no process to have certain packages in a QA stage before they are shipped to production.
Goals
The following proposal dives into how the above can be solved. But even though multiple stages of deployment are introduced, it should still be as easy as possible to release a new package. The different stages of deployment can be followed but it is not a strict requirement.
Note: At the moment we have an additional "experimental" environment which goes away in the near future and is ignored on purpose in this proposal.
Environments
To make testing and snapshot builds possible, three environments are needed:
Each of these environments is tied to a specific version of the package-storage and the package-registry and has different rules on how packages are added. These are described in more details below.
Snapshot environment
The snapshot environment is to share the latest versions of packages and testing it together with the snapshot builds of the Elastic Stack. New packages are added and updated at any time and in most cases fully automated. In the snapshot branch it is allowed to overwrite / update existing versions.
Addition of packages are done through direct commits to the branch. It is expected, that any package updates pushed to the branch are already pre-checked by CI by the contributor.
The branch used for the snapshot packages is called
snapshotin the package-storage repositories. The related package-registry is the one in themasterbranch. Every time a new commit is pushed to the package-storage snapshot branch or the package-registry master branch a new build is kicked off. The build is expected to pass as precheckes should have happened, if not, the contributors are pinged and deployment does not happen.Taking the integrations repository as an example, every time a PR is merged, a script will trigger a commit to the
snapshotbranch with the updated version. This will trigger a build and a new registry is deployed.The snapshot registry is available under
epr-snapshot.elastic.co.Snapshot is NOT a package development environment. All changes and tests should happen outside the snapshot branch and only the final result of changes is pushed.
The snapshot registry is a combination of production + staging + snapshot packages. If the same version exists in production and snapshot, the production version is taken as these should not conflict. Having all packages together in snapshot also allows to do upgrade tests.
Staging environment
The staging environment is meant for testing packages which are ready for deployment. Talking in terms of directory, moving to staging is moving a directory from snapshot to staging, meaning the package will be removed from snpashot. In most cases, when a package enters staging, it is expected not to change anymore. But if issues are found, changes can be picked again from the snapshot branch. A script is expected to do the work taking a package from snapshot and pushing it to staging.
From the deployment perspective it is the same as the snapshot build, it is continously built.
It links to a specific release version of the registry. If the registry is updated, the registry reference has to be updated manually. This is to ensure all CI builds are consistent.
The staging environment is expected to be used on SNAPSHOT branches.
The url used is
epr-staging.elastic.co.Production environment
The production branch is used for all the released packages. These packages should never be changed / updated after they are released. Contributions to the production branch happend through a PR to make sure CI checks can be run in advance. These PR are normally opened by a script taking a package from the staging branch. As soon as it is merged into production, the staging version should be removed. For now the merging of these PR's need to happen manually, but it could be automated if CI is green for "trusted" contributors.
The registry is tied to a specific release tag. As soon as a PR is merged in the production branch, it is deployment automatically to
epr.elastic.co.Summary
In summary, the above gives us 3 different environments. Deployments of each environment happens fully automated as soon as a commit is added to the branch. Only in the case of production, the addition of a package has to go through a PR to have a previous CI check guaranteed.
Below is a summary table of the 3 environments:
How to get to this new deployment
Today all packages are in the
masterbranch. To make the above possible, theintegrationsrepository script has to be adjusted and an additional script has to be built, to move packages between the different version.On the Kibana side, a way must be found the SNAPSHOT builds point to a different version of the registry then development. Or the production one could be just the default and it needs manual adjustment.
Why Git and branches
Instead of using Git for the above, it would also be possible to just use a directory structure on S3 or similar. But the nice thing about Git is that it shows us exactly what happened when and by who in the past, and in extreme casees allows us to roll back changes if needed. In addition for the production registry it allows us to the use a manual PR review to have CI checks in advance. The part that is a bit more combuersome is moving packages between branches, but scripts can do this for us.