This article aims at introducing a small library to ease end-to-end testing of applications in Kubernetes environments.
An overview of what already existed
When typing “kubernetes e2e” or “kubernetes end to end” on Google, the first result I got was about testing a K8s cluster or component. It is what the project’s team is using to test the development of the Kubernetes project. This is not what I wanted. My goal was to test an application I packaged for K8s, not K8s itself.
Terratest is another solution I found. We have the same goal, but viewing this project made me realize I did not want a solution involving advanced programming. We have DevOps that can develop and maintain operative aspects. But they are not so many and most hardly know the Go language. All the team members learned kubectl and Helm commands easily. A scripting solution would be better. This would avoid the choice of a programming language (Go, Java…) and a thus a lot of arguing / reinventing.
Since we mostly had Helm packages, used by several internal projects, I tried to focus on Helm. I immediately found the solution used by the official Helm project. There are interesting parts, such as the linting and version checks. This can be convenient if you setup a internal Helm repository and that you want to make every chart use the same rules. There are also commands to check an installation. Anyway, this is tailored for a collection of Helm charts and still not adapted to what I wanted.
I then found the unit test plug-in for Helm. The principle is to create a YAML file that contains a set of tests. A Helm chart has to be deployed in the environment. The YAML files are passed to the chart that verifies them. This is an interesting solution, but it mostly tests the templating of your chart. Not the applicative behavior.
Testing an application in a Kubernetes environment means being able to deploy it, adapt the topology (scale replicas), verify everything works, execute scenarios and check assertions at various stages. The solution that fit the best this requirement was EUFT. This small project relies on BATS (Bash Automated Testing System), a script framework that allows to write and execute unit tests by using scripts. EUFT is in fact a solution to deploy a small K8s cluster and run BATS tests inside. Examples of tests are available in this repository. I also found out afterwards that Hashicorp was using the same technique for some of their Helm packages.
If I liked the principle of BATS, all the tests used by EUFT and Hashicorp are a little bit complex to maintain. Not everyone in our project is a script god. Besides, we do not want to deploy a K8s cluster in our tests: we want to use an existing one, with the same settings than our production one. This is important because of permissions and network policies. Running e2e tests in a ephemeral K8s installation is too limited. However, EUFT gave me a direction since I have not found anything else.
The DETIK library
I was not really inspired for a name…
DETIK stands for « DevOps End-to-End Testing In Kubernetes ». The idea is to write tests by using scripts, running them with BATS, and having a simple syntax, almost in natural language, to write assertions about resources in Kubernetes. With kubectl or Helm commands, a few knowledge in scripts (BASH, Ruby, Python, whatever…) and this library, anyone should be able to write applicative tests and be able to maintain them with very few efforts.
In addition to performing actions on the cluster, I also wanted to support the execution of scenarios. Scenarios can imply topology adaptations, but also user actions. BATS can integrate with many solutions, such as Selenium or Cypress for end-user scenarios, or Gatling for performance tests. With all these tools, it becomes possible to test an application from end-to-end in a K8s environment.
Example
The following example is taken from the Git repository.
It show the test of a Helm package. A part of the syntax comes from BATS.
#!/usr/bin/env bats
###########################################
# An example of tests for a Helm package
# that deploys Drupal and Varnish
# instances in a K8s cluster.
###########################################
load "/home/testing/lib/detik.bash"
DETIK_CLIENT_NAME="kubectl"
pck_version="1.0.1"
function setup() {
cd $BATS_TEST_DIRNAME
}
function verify_helm() {
helm template ../drupal | kubectl apply --dry-run -f -
}
@test "verify the linting of the chart" {
run helm lint ../drupal
[ "$status" -eq 0 ]
}
@test "verify the deployment of the chart in dry-run mode" {
run verify_helm
[ "$status" -eq 0 ]
}
@test "package the project" {
run helm -d /tmp package ../drupal
# Verifying the file was created is enough
[ -f /tmp/drupal-${pck_version}.tgz ]
}
@test "verify a real deployment" {
[ -f /tmp/drupal-${pck_version}.tgz ]
run helm install --name my-test \
--set varnish.ingressHost=varnish.test.local \
--set db.ip=10.234.121.117 \
--set db.port=44320 \
--tiller-namespace my-test-namespace \
/tmp/drupal-${pck_version}.tgz
[ "$status" -eq 0 ]
sleep 10
# PODs
run verify "there is 1 pod named 'my-test-drupal'"
[ "$status" -eq 0 ]
run verify "there is 1 pod named 'my-test-varnish'"
[ "$status" -eq 0 ]
# Postgres specifics
run verify "there is 1 service named 'my-test-postgres'"
[ "$status" -eq 0 ]
run verify "there is 1 ep named 'my-test-postgres'"
[ "$status" -eq 0 ]
run verify \
"'.subsets[*].ports[*].port' is '44320' " \
"for endpoints named 'my-test-postgres'"
[ "$status" -eq 0 ]
run verify \
"'.subsets[*].addresses[*].ip' is '10.234.121.117' " \
"for endpoints named 'my-test-postgres'"
[ "$status" -eq 0 ]
# Services
run verify "there is 1 service named 'my-test-drupal'"
[ "$status" -eq 0 ]
run verify "there is 1 service named 'my-test-varnish'"
[ "$status" -eq 0 ]
run verify "'port' is '80' for services named 'my-test-drupal'"
[ "$status" -eq 0 ]
run verify "'port' is '80' for services named 'my-test-varnish'"
[ "$status" -eq 0 ]
# Deployments
run verify "there is 1 deployment named 'my-test-drupal'"
[ "$status" -eq 0 ]
run verify "there is 1 deployment named 'my-test-varnish'"
[ "$status" -eq 0 ]
# Ingress
run verify "there is 1 ingress named 'my-test-varnish'"
[ "$status" -eq 0 ]
run verify \
"'.spec.rules[*].host' is 'varnish.test.local' " \
"for ingress named 'my-test-varnish'"
[ "$status" -eq 0 ]
run verify \
"'.spec.rules[*].http.paths[*].backend.serviceName' " \
"is 'my-test-varnish' for ingress named 'my-test-varnish'"
[ "$status" -eq 0 ]
# PODs should be started
run try "at most 5 times every 30s " \
"to get pods named 'my-test-drupal' " \
"and verify that 'status' is 'running'"
[ "$status" -eq 0 ]
run try "at most 5 times every 30s " \
"to get pods named 'my-test-varnish' " \
"and verify that 'status' is 'running'"
[ "$status" -eq 0 ]
# Indicate to other tests that the deployment succeeded
echo "started" > tests.status.tmp
}
@test "verify the deployed application" {
if [[ ! -f tests.status.tmp ]]; then
skip " The application was not correctly deployed... "
fi
rm -rf /tmp.drupal.html
curl -sL http://varnish.test.local -o /tmp/drupal.html
[ -f ${BATS_TMPDIR}/drupal.html ]
grep -q "<title>Choose language | Drupal</title>" /tmp/drupal.html
grep -q "Set up database" /tmp/drupal.html
grep -q "Install site" /tmp/drupal.html
grep -q "Save and continue" /tmp/drupal.html
}
@test "verify the undeployment" {
run helm del --purge my-test --tiller-namespace my-test-namespace
[ "$status" -eq 0 ]
[ "$output" == "release \"my-test\" deleted" ]
run verify "there is 0 service named 'my-test'"
[ "$status" -eq 0 ]
run verify "there is 0 deployment named 'my-test'"
[ "$status" -eq 0 ]
sleep 60
run verify "there is 0 pod named 'my-test'"
[ "$status" -eq 0 ]
}
@test "clean the test environment" {
rm -rf tests.status.tmp
}
These unit tests include the linting of the chart, a dry-run deployment, but also a real deployment with a basic topology. After deploying it, we verify assertions on K8s resources. Once the application (a simple Drupal) is started, we get the content of the web site and make sure it contains some expected words and sentences. We could replace it by a Selenium scenario.
Executing the bats my-test-file.bats command would start the execution.
A successful run would show the following output:
bats my-test-file.bats 1..7 ✓ 1 verify the linting of the chart ✓ 2 verify the deployment of the chart in dry-run mode ✓ 3 package the project ✓ 4 verify a real deployment ✓ 5 verify the deployed application ✓ 6 verify the undeployment ✓ 7 clean the test environment The command "bats my-test-file.bats" exited with 0.
Errors appear like below.
...
✗ 1 verify the linting of the chart
(in test file my-test(file.bats, line 14)
`[ "$status" -eq 0 ]' failed
...
Library Principles
Assertions are used to generate kubectl queries.
The output is extracted and compared to the values given as parameters.
There are very few queries in fact.
However, they work with all the kinds of resources of Kubernetes. That includes native K8s objects (POD, services….) but also OpenShift elements (routes, templates…) or custom resources (e.g. the upcoming Helm v3 objects).
Queries can be run with kubectl or with oc (the OpenShift client).
You only have to specify the client name in the DETIK_CLIENT_NAME variable (and make sure the client is available in the path).
With this, you can verify pre and post-conditions when using a Kubernetes client, Helm or even operators.
Usage
The library is available as a single file.
It can be donwloaded from this Github repository. The syntax is documented in the readme of the project.
A Dockerfile is provided as a basis in the project.
It embeds a kubectl client, a Helm client, BATS and the DETIK library. Depending on your cluster configuration, you might want to add other items (e.g. to log into your cluster).
Continuous Integration
The project is documented and explains how to execute (and write) such tests on your own machine. But the real interest of such tests is to be run in the last parts of an automated pipeline.
Here is a simple Jenkinsfile (for a Jenkins pipeline).
def label = "${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
podTemplate(label: label, containers: [
containerTemplate(
name: 'jnlp',
image: 'jnlp-slave-alpine:3.27-1-alpine'),
containerTemplate(
name: 'detik',
image: 'detik:LATEST',
ttyEnabled: true,
alwaysPullImage: true,
envVars: [
envVar(key: 'http_proxy', value: 'http://proxy.local:3128'),
envVar(key: 'https_proxy', value: 'http://proxy.local:3128'),
envVar(key: 'TILLER_NAMESPACE', value: 'my-test-namespace')
])
]) {
node(label) {
container(name: 'jnlp') {
stage('Checkout') {
checkout scm
}
}
container(name: 'ci-docker') {
stage('Login') {
withCredentials([usernamePassword(
credentialsId: 'k8s-credentials',
passwordVariable: 'K8S_PASSWORD',
usernameVariable: 'K8S_USERNAME')]) {
echo 'log into the cluster...'
// TODO: it depends on your cluster configuration
}
}
stage('Build and Test') {
sh 'bats tests/main.bats'
}
}
}
}
It can easily be adapted for Travis or GitLab CI.
You will find more examples on Github.
News from April 2020: the project has joined the BATS Core organization on Github.
Vincent Zurczak's Blog 