This project is deprecated in favor of the new Concourse Turbine project, based on the Turbine Framework. Turbine Framework is the new name for GBE (Gstack BOSH Environment). That's where future updates will be pushed.

This repository deploys a Concourse cluster behind a Træfik reverse-proxy.

It is a very first example of a GBE downstream repository, which means that it follows a GBE upstream repository. In this case, the upstream is the main Easy Foundry repository.

The Concourse Cluster from Easy Foundry has these charateristics:

• CredHub for storing pipeline secrets

• UAA for CredHub (not for autenticating users, contributions are welcome)

• Separate instance groups for CredHub and UAA (allows independent scaling of those components)

• Security: encryption all over the place

  • Encryption of data a the database level
  • HTTPS between all components
  • No HTTP anywhere

• Discovery (and failover) of components through BOSH DNS aliases (the current best practice, and most portable choice)

• Sanity tests as post-deploy hook (this fixes a missing piece, because no smoke tests are shipped by the Concourse BOSH Release, unfortunately)

Beyond these base characteristics, we have added here the following additions.

• Scale horizontally for limited downtime during upgrades.

  • Scale ATC (the web UI), CredHub, UAA and Concourse Worker to 2 instances, for limited downtime when upgrading the Concourse cluster.

  • Keep Postgres to 1 instance (because the standard and basic Postgres BOSH Release we use doesn't implement leader-follower replication, nor failover, which BTW are not trivial matters).

• Add the ATCs behind a Træfik reverse-proxy.

  • With circuit breaking: whenever those 2 ATCs produce more than 50% network errors, then a “Service Unavailable” error message is displayed by Træfik.

  • With Dynamic Round Robin load-balancing between ATC nodes, in case only one of them performs bad compared to the other.

  • With health checks, so that when an ATC node is shut down by BOSH during an upgrade, then no traffik is served to it anymore.

We first Git-clone GBE next to this repo, and we name it bosh-environment.

git clone https://github.com/gstackio/gstack-bosh-environment.git bosh-environment
git clone https://github.com/gstackio/concourse-gbeds.git
cd concourse-gbeds/
echo "--- {}" > concourse-bosh-env/conf/secrets.yml
chmod 600 concourse-bosh-env/conf/secrets.yml

Then we go check the GBE pre-requisites. Here the provided concourse-{bosh,garden}-env GBE environments are modeled after the hybrid topology to create the base BOSH environment.

We configure our BOSH environment as detailed in the GBE documentation. When ready, we create our environment.

export GBE_ENVIRONMENT=concourse-bosh-env
source <(./bin/gbe env)  # add 'gbe' to the $PATH
GBE_ENVIRONMENT=concourse-garden-env gbe up \
    && GBE_ENVIRONMENT=concourse-bosh-env gbe up
source <(./bin/gbe env)  # reload the updated environment variables

We could either create your BOSH environment with BUCC anyway, this would make no difference as long as the BUCC is properly targeted with source <(./bin/bucc env) for the following instructions below.

We update the Cloud and Runtime configs of our environment.

gbe update cloud-config runtime-config

We configure the external_host property in deployments/concourse-standalone/conf/spec.yml with a fully-qualified DNS name that points to the external_ip we have set in the concourse-{bosh,garden}-env/conf/spec.yml files. And yes, we need to set a DNS A record for this in our DNS zone.

Also in deployments/traefik-concourse/conf/spec.yml, we can configure these properties: web_backend_hostname, acme_certs_email and traefik_domain.

We are ready to converge the two GBE subsystems for Træfik and Concourse.

gbe converge -y traefik-concourse concourse-standalone

Assuming we have set external_ip: ci.example.com above and the DNS has converged, then we can open https://ci.example.com and use our Concourse. The first HTTPS request will provision a new TLS certificate. By default we use the staging Let's Encrypt API, so the certificate is reported untrusted by web browsers (red lock).

Later, when our setup is correct, we can opt for acme_staging: false in Træfik subsystem's spec.yml config. After this, the first request we make to https://ci.example.com will generate a green-lock HTTPS certificate with the Let's Encrypt production endpoint. We need to be careful though, because this production endpoint is subject to very strict API limitations.

Please feel free to submit issues and pull requests.

Copyright © 2018-2020, Benjamin Gandon, Gstack

Like BOSH and GBE, this example GBE downstream project is released under the terms of the Apache 2.0 license.