…ents (#1078)

* chore: provide a fleet-server base image based on centos/debian with systemd

* WIP

* fix: remove duplicated fields after merge conflicts

* fix: update method call after merge conflicts

* chore: extract service name calculation to a method

* chore: extract container name calculation to a method

* chore: refactor get container name method

* chore: refactor method even more

* chore: use installer state to retrieve container name

* chore: use installer when calculating service name

* fix: adapt service names for fleet server

* chore: enrich log when creating an installer

* fix: use fleet server host when creating fleet config

* fix: use https when connecting to fleet-server

It's creating its own self-signed certs

* feat: bootstrap a fleet server before a regular agent is deployed to fleet

It will define the server host to be used when enrolling agents

* fix: use fleet policy for agents, not the server one

* fix: get different installers for fleet-server and agents

* fix: use the old step for deploying regular agents

* chore: rename variable with consistent name

* chore: rename fleet-server scenario

* fix: use proper container name for standalone mode

* chore: save two variables

* chore: rename standalone scenario for bootstrapping fleet-server

* chore: rename bootstrap methods

* chore: encapsulate bootstrap fleet-server logic

* Update fleet.go

* chore: remove Fleet Server CI parallel execution

* chore: remove feature file for fleet-server

* chore: boostrap fleet server only once

We want to have it bootstrapped for the entire test suite, not for each scenario

* fix: an agent was needed when adding integrations

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>
(cherry picked from commit 193337a)

* master:
  Support fleet-server-service-token (elastic#1096)
  fix: use wider selector for ARM workers (elastic#1103)
  feat: bootstrap fleet-server for the deployment of regular elastic-agents (elastic#1078)
  fix: use proper variable name (elastic#1102)
  fix: branch_specifier is needed (elastic#1097)
  Move kubernetes/kubectl/kind code to internal project layout (elastic#1092)
  fix: update JJBB with proper values (elastic#1093)
  feat: support building centos/debian Docker images in multiplatform format (elastic#1091)
  Kubernetes autodiscover suite (elastic#1064)

…ents (elastic#1078)

* chore: provide a fleet-server base image based on centos/debian with systemd

* WIP

* fix: remove duplicated fields after merge conflicts

* fix: update method call after merge conflicts

* chore: extract service name calculation to a method

* chore: extract container name calculation to a method

* chore: refactor get container name method

* chore: refactor method even more

* chore: use installer state to retrieve container name

* chore: use installer when calculating service name

* fix: adapt service names for fleet server

* chore: enrich log when creating an installer

* fix: use fleet server host when creating fleet config

* fix: use https when connecting to fleet-server

It's creating its own self-signed certs

* feat: bootstrap a fleet server before a regular agent is deployed to fleet

It will define the server host to be used when enrolling agents

* fix: use fleet policy for agents, not the server one

* fix: get different installers for fleet-server and agents

* fix: use the old step for deploying regular agents

* chore: rename variable with consistent name

* chore: rename fleet-server scenario

* fix: use proper container name for standalone mode

* chore: save two variables

* chore: rename standalone scenario for bootstrapping fleet-server

* chore: rename bootstrap methods

* chore: encapsulate bootstrap fleet-server logic

* Update fleet.go

* chore: remove Fleet Server CI parallel execution

* chore: remove feature file for fleet-server

* chore: boostrap fleet server only once

We want to have it bootstrapped for the entire test suite, not for each scenario

* fix: an agent was needed when adding integrations

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* feat: support building centos/debian Docker images in multiplatform format (#1091)

* chore: copy Centos+Systemd Dockerfile from origin

See https://github.com/CentOS/CentOS-Dockerfiles/tree/master/systemd/centos7

* chore: copy debian+systemd Dockerfile from origin

See https://github.com/alehaa/docker-debian-systemd

* chore: add script to build&push ARCH-based images for centos and debian

* chore: add script to push the multiplatform manifest for centos and debian

This script leverages infra's tool to write the manifest, which needs to
be ran right after the images have been built and pushed. Therefore, the
tool will write the manifest for both platforms (AMD/ARM), inspecting
the existing platform-specific repositories, combining them into the target.

FYI, the '-ARCH' placeholder will be replaced with the values in the
'--platforms' argument

* chore: add regular pipeline to build the docker images

* fix: default arch variable value

* chore: abstract image name from file system

* chore: couple agent's base box with stack platform

* Move kubernetes/kubectl/kind code to internal project layout (#1092)

This is mainly a cleanup to keep all internal related code that could be
reusable in our `internal` directory layout.

Next steps would be to take what's in `internal/kubectl` and merge with this code.

Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* feat: bootstrap fleet-server for the deployment of regular elastic-agents (#1078)

* chore: provide a fleet-server base image based on centos/debian with systemd

* WIP

* fix: remove duplicated fields after merge conflicts

* fix: update method call after merge conflicts

* chore: extract service name calculation to a method

* chore: extract container name calculation to a method

* chore: refactor get container name method

* chore: refactor method even more

* chore: use installer state to retrieve container name

* chore: use installer when calculating service name

* fix: adapt service names for fleet server

* chore: enrich log when creating an installer

* fix: use fleet server host when creating fleet config

* fix: use https when connecting to fleet-server

It's creating its own self-signed certs

* feat: bootstrap a fleet server before a regular agent is deployed to fleet

It will define the server host to be used when enrolling agents

* fix: use fleet policy for agents, not the server one

* fix: get different installers for fleet-server and agents

* fix: use the old step for deploying regular agents

* chore: rename variable with consistent name

* chore: rename fleet-server scenario

* fix: use proper container name for standalone mode

* chore: save two variables

* chore: rename standalone scenario for bootstrapping fleet-server

* chore: rename bootstrap methods

* chore: encapsulate bootstrap fleet-server logic

* Update fleet.go

* chore: remove Fleet Server CI parallel execution

* chore: remove feature file for fleet-server

* chore: boostrap fleet server only once

We want to have it bootstrapped for the entire test suite, not for each scenario

* fix: an agent was needed when adding integrations

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* apm-server tests (#1083)

* some tests for apm-server
* clean op dir on init instead of after

* fix agent uninstall (#1111)

* Auto bootstrap fleet during initialize scenario (#1116)

Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

* feat: support running k8s autodiscover suite for Beats PRs and local repositories (#1115)

* chore: add license

* chore: initialise configurations before test suite

* chore: use timeout_factor from env

* fix: tell kind to skip pulling beats images

* chore: add a method to load images into kind

* feat: support running k8s autodiscover for Beats PRs or local filesystem

* chore: add license header

* chore: expose logger and use it, simplifying initialisation

* fix: only run APM services for local APM environment

* Revert "chore: expose logger and use it, simplifying initialisation"

This reverts commit a89325c.

* chore: log scenario name

* fix: always cache beat version for podName

* chore: reduce log level

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>
Co-authored-by: Juan Álvarez <juan.alvarez@elastic.co>

…ents (elastic#1078)

* chore: provide a fleet-server base image based on centos/debian with systemd

* WIP

* fix: remove duplicated fields after merge conflicts

* fix: update method call after merge conflicts

* chore: extract service name calculation to a method

* chore: extract container name calculation to a method

* chore: refactor get container name method

* chore: refactor method even more

* chore: use installer state to retrieve container name

* chore: use installer when calculating service name

* fix: adapt service names for fleet server

* chore: enrich log when creating an installer

* fix: use fleet server host when creating fleet config

* fix: use https when connecting to fleet-server

It's creating its own self-signed certs

* feat: bootstrap a fleet server before a regular agent is deployed to fleet

It will define the server host to be used when enrolling agents

* fix: use fleet policy for agents, not the server one

* fix: get different installers for fleet-server and agents

* fix: use the old step for deploying regular agents

* chore: rename variable with consistent name

* chore: rename fleet-server scenario

* fix: use proper container name for standalone mode

* chore: save two variables

* chore: rename standalone scenario for bootstrapping fleet-server

* chore: rename bootstrap methods

* chore: encapsulate bootstrap fleet-server logic

* Update fleet.go

* chore: remove Fleet Server CI parallel execution

* chore: remove feature file for fleet-server

* chore: boostrap fleet server only once

We want to have it bootstrapped for the entire test suite, not for each scenario

* fix: an agent was needed when adding integrations

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* Move kubernetes/kubectl/kind code to internal project layout (#1092)

This is mainly a cleanup to keep all internal related code that could be
reusable in our `internal` directory layout.

Next steps would be to take what's in `internal/kubectl` and merge with this code.

Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* feat: bootstrap fleet-server for the deployment of regular elastic-agents (#1078)

* chore: provide a fleet-server base image based on centos/debian with systemd

* WIP

* fix: remove duplicated fields after merge conflicts

* fix: update method call after merge conflicts

* chore: extract service name calculation to a method

* chore: extract container name calculation to a method

* chore: refactor get container name method

* chore: refactor method even more

* chore: use installer state to retrieve container name

* chore: use installer when calculating service name

* fix: adapt service names for fleet server

* chore: enrich log when creating an installer

* fix: use fleet server host when creating fleet config

* fix: use https when connecting to fleet-server

It's creating its own self-signed certs

* feat: bootstrap a fleet server before a regular agent is deployed to fleet

It will define the server host to be used when enrolling agents

* fix: use fleet policy for agents, not the server one

* fix: get different installers for fleet-server and agents

* fix: use the old step for deploying regular agents

* chore: rename variable with consistent name

* chore: rename fleet-server scenario

* fix: use proper container name for standalone mode

* chore: save two variables

* chore: rename standalone scenario for bootstrapping fleet-server

* chore: rename bootstrap methods

* chore: encapsulate bootstrap fleet-server logic

* Update fleet.go

* chore: remove Fleet Server CI parallel execution

* chore: remove feature file for fleet-server

* chore: boostrap fleet server only once

We want to have it bootstrapped for the entire test suite, not for each scenario

* fix: an agent was needed when adding integrations

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* apm-server tests (#1083)

* some tests for apm-server
* clean op dir on init instead of after

* fix agent uninstall (#1111)

* Kubernetes Deployment (#1110)

* Kubernetes Deployment

Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* Expose hostPort for kibana, elasticsearch, fleet without needing ingress

This is nice for local development where you don't need an ingress and are
relatively sure that the host system has the required ports available to bind to.

Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* Auto bootstrap fleet during initialize scenario (#1116)

Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

* feat: support running k8s autodiscover suite for Beats PRs and local repositories (#1115)

* chore: add license

* chore: initialise configurations before test suite

* chore: use timeout_factor from env

* fix: tell kind to skip pulling beats images

* chore: add a method to load images into kind

* feat: support running k8s autodiscover for Beats PRs or local filesystem

* chore: add license header

* chore: expose logger and use it, simplifying initialisation

* fix: only run APM services for local APM environment

* Revert "chore: expose logger and use it, simplifying initialisation"

This reverts commit a89325c.

* chore: log scenario name

* fix: always cache beat version for podName

* chore: reduce log level

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* chore: initialise timeout factor next to the declaration (#1118)

* chore: initialise timeout factor on its own package

* chore: reuse timeout factor from common

* Unify fleet and stand-alone suites (#1112)

* fix agent uninstall

* unify fleet and stand alone suites

* move things around a bit more

* fixe bad merge

* simplify some things

* chore: remove unused code (#1119)

* chore: remove unused code

* chore: remove all references to fleet server hostname

Because we assume it's a runtime dependency, provided by the initial
compose file, we do not need to calculate service names, or URIs for the
fleet-service endpoint. Instead, we assume it's listening in the 8220 port
in the "fleet-server" hostname, which is accessible from the network
created by docker-compose.

* fix: use HTTP to connect to fleet-server

* chore: remove fleet server policy code

We do not need it anymore, as the fleet server is already bootstrapped

* chore: remove all policies but system and fleet_server

* Update policies.go

* Update fleet.go

* Update stand-alone.go

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* Support multiple deployment backends (#1130)

* Abstract out deployment

Provides ability to plugin different deployment backends for use in testing.
Current deployment backends supported are "docker" and "kubernetes"

* remove unused import
* remove unsetting of fleet server hostname as it's not needed
* add deployer support to stand-alone
* add elastic-agent to k8s deployment specs
* Update internal/docker/docker.go

Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>
Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

* fix: bump stale agent version to 7.12-snapshot

* chore: abstract process checks to the deployer (#1156)

* chore: abstract process checks to the deployer

* chore: rename variable in log entry

* docs: improve comment

* fix: go-fmt

* feat: simplify the initialisation of versions (#1159)

* chore: use fixed version in shell scripts

* chore: move retry to utils

We could move it to its own package, but at this moment it's very small

* chore: initialise stackVesion at one single place

* chore: initialise agent version base at one single place

* chore: initialise agent version at one single place

* chore: reduce the number of requests to Elastic's artifacts endpoint

* chore: rename AgentVersionBase variable to BeatVersionBase

* chore: rename AgentVersion variable to BeatVersion

* chore: use Beat version in metricbeat test suite

* chore: check if the version must use the fallback after coming from a Git SHA

* feat: support flavours in services, specially in the elastic-agent (#1162)

* chore: move compose to deploy package

* feat: use a ServiceRequest when adding services

* feat: add service flavour support

* chore: remove unused centos/debian services

* fixup: add service flavour

* chore: move docker client to the deploy package

We will need another abstraction to represent the Docker client operations,
as it's clear what is a deployment and what is an operation in the deployment.
Maybe a Client struct for each provider will help out in differenciate it

* chore: use ServiceRequest everywhere

* chore: run agent commands with a ServiceRequest

* chore: use ServiceRequest in metricbeat test suite

* chore: pass flavours to installers

* chore: add a step to install the agent for the underlying OS

* chore: always add flavour

* fix: use installer for fleet_mode when removing services at the end of the scenario

* fix: update broken references in metricbeat test suite

* fix: update broken references in helm test suite

* fix: standalone does not have an installer

* fix: use service instead of image to get a service request for the agent

* feat: support for scaling services in compose

* fix: run second agent using compose scale option

* fix: update kibana's default Docker namespace

* feat: make a stronger verification of fleet-server being bootstrapped (#1164)

* fix: resolve issues in k8s-autodiscover test suite (#1171)

* chore: use timeout factor when tagging docker images

* fix: resolve alias version in k8s-autodiscover test suite

* fix: use common versions for k8s-autodiscover

* fix: update background processes to 2 instances

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>
Co-authored-by: Juan Álvarez <juan.alvarez@elastic.co>