• Relates Test upgrade from previous version to latest version in CI #1831

It could be nice to have some feature to assist detecting backwards compatibility issues in packages. This could be part of elastic-package check, but it would have the particularity of needing to download (or checkout) previous versions of the package, so maybe it should be optional.

There are some things that could be automatically checked between versions:

• Incompatible changes in field data types.
• Removed fields.
• Changes in sets of dimensions.
• Changes in policy templates.
• Removed variables.
• Variables not referenced in any configuration.
• Removed data streams.
• If any of these issues are detected, check that there is at least a breaking change entry in the changelog.

Proposed solution

We will add a new elastic-package test compatibility subcommand. In general this command will be used to compare the compatibility of the definitions in two versions of a package. This subcommand would run tests decided by flags or the environment. It would have different modes:

• [--from-git <git repository>] [--from-git-ref <git ref>] Default execution would check if the current working copy is a git repository, with a branch forked from an upstream branch. If it is, it would checkout the package from the base commit to a temporary location and would compare with the local package. If it cannot obtain another package from the local repository, because it is not a repository or it doesn't contain a previous version of the package, the test does nothing. This could be extended with flags to select a source repository and commit.
• --from-registry <registry url> [--kibana-version <version>] [--all] would compare the local package with versions published in the indicated registry. By default it would compare with the latest version published, but could be extended to support comparing with the package of a given kibana version, or with all packages.
• --from-registry <registry url> --package <package> --from-registry-version <version> --to-registry-version <version> would compare package versions from the registry. It does not require a local copy. This could be used to check compatibility between versions already published.

For any of the modes, the compatibility checks would be performed in the following ways:

• Static checks based on comparing the definitions for each package, taking into account fields imported from ECS, ecs@mappings and so on. Comparisons would be done by looping over all definitions in the old package, one by one, and checking for the presence of a compatible definition in the new one. We would also need to look for removed definitions. This would be the default.
• --system System checks based on installing both package versions, dumping the generated assets, and comparing them following similar rules as the static checks. It would also do a dummy upgrade testing by installing one version after the other. The stack used would be the same as in other tests, the one in the current profile, or indicated by environment variables.

The specific checks that will be performed on each execution would be the ones mentioned in the top of the description. We will start by implementing the tests for field mappings compatibility, and would follow by implementing the ones about variables.

Packages will have the option to skip these tests following the common settings we have for that in other tests. Additionally, breaking change notices in the changelogs can be used to circunvent detected failures.

We would start implementing the local git mode, with static checks, what could be already useful to detect many issues in fields changes. Other modes and tests would be implemented on following phases.