Play with Docker classroom

Medical physics with Geant4 + Docker = ❤️

Sun, 23 Feb 2020 00:00:00 +0000

In this lab we will learn about the basics of low and mid-energy radiation-matter interaction with the help of Geant4, the Geant4 binding for Python (g4py), and of course, our beloved Docker. This lab may fall into the recent (and praiseworthy) trend related to the so called reproducible research and Open Educational Resources as a Service. This PWD/Docker-based experience has been carried out with huge success at Universidad Internacional de La Rioja (UNIR), as described in this research paper.

Simulating the building blocks of Reality

To begin with, let’s start by summarising what is Geant4 and why it is so important in todays particle physics research. As very well stated by Agostinelli et al.:

Geant4 is a toolkit for simulating the passage of particles through matter. Its functionalities include tracking, geometry, physics models and hits. The physics processes take into account electromagnetic, hadronic and optical processes, as well as the surrounding materials, over a wide energy range. The toolkit is the result of a worldwide collaboration of physicists and software engineers. It has been implemented in the C++ programming language. It has been used in applications in Particle Physics, Nuclear Physics, accelerator design, Space Engineering and Medical Physics.

In other words, Geant4 implements our deepest (and best) level of knowledge about Nature… in the form of computer (C++) code. This deepest level is no other thing but the Standard Model of physics. This picture describes the building blocks of Reality (the subatomic particles) as well as the fundamental forces that govern their interactions. Geant4 acts as some sort of magnifying lens that amplifies what happens when a given particle (an electron, a kaon, a muon, or a photon) hits or, better said, interacts with its surroundings. This computer toolbox is able to simulate almost all possible scenarios, no matter the actors involved, the geometry or the energy of the event.

This turns out perfect if we happen to be particle physicists, but it may be felt as over complicated for those who just need to model humbler atomic interactions for specific fields. One if these specific fields is Medical Physics and, in this case, Geant4 has a wonderful and easy to use Python binding, g4py. With the help of Docker, g4py is even easier to use.

Particle Physics in Medicine

The field of Medical Physics has its own complexity. Subatomic particles can be used to treat cancer or tumours or can be used to image (probe) a patient’s body, i.e., to see its inner parts. This last application is the one that will be tackled in this interactive lab. Normally, the type of imaging particle will be the (X-ray) photon, although you’ll be able to play with other probing rays such as electrons or positrons. Photons can be defined as particles of light and were originally proposed by Einstein in one of his 1905 paper An Heuristic Point of View Toward the Emission and Transformation of Light.

In Medical Imaging, mid-energy photons (a few keV) are projected against the patient. Some of these electromagnetic entities are completely absorbed (photoelectric effect), others are diverted (Compton scattering) by atoms, and others (if their energy is high enough) can even transform themselves into a pair of new objects composed of a electron and an anti-electron (a positron). This last process is called pair production. In turn, when a positron accidentally hits and electron, they both disappear back into two photons which travel in opposite directions. This is called annihilation and it’s the foundation of PET imaging (which produces beautiful and lifesaving images such as the one shown above).

Geant4 and Medical Physics

As stated above, Geant4 has a difficult-to-grasp C++ heart, but thanks to g4py, its easy to design simple scenarios related to the field of Medical Physics which radiologists and doctors can even used in their daily practice. However, installing and configuring Geant4 and g4py can turn out awkward. We can alleviate this problem with Docker and better yet, we can play with a simple, yet interesting simulation thanks to Play with Docker. This simulation consists of the following ingredients:

A world, or the simulation limits. This world has the shape of an empty elongated cuboid. The longest axis is the z (or longitudinal) axis.
A beam of particles that are initially projected along the aforementioned z-axis. This beam has a nature (type of particles), an energy (in mega-electronvolts or MeV, for short) and a count (i.e., how many particles). In a real particle physics experiment or setup, generating (and conforming) such a beam has its complexities. However, in a simulated world, we don’t have to worry about those.
A target, or something the particles can hit in their way. This target has also cuboid shape, with a width (thickness) and location (both also defined along the same z-axis as the world) and is composed of a given (and selectable) material. In Medical Physics, the target is also commonly known as phantom.

You can take a 3D look at this world in the following image:

Or you can also surf a live WebGL version by clicking in this link. A quick (and tiny) animation is also displayed above.

We will now launch our own simulations. In order to achieve so, let us first download the Docker image that contains the necessary infrastructure. Remember, you can just click in the code cells to have them (one by one) executed ▶️ in the terminal on the left side of this page:

docker pull pammacdotnet/g4lpwd

It may take a minute 🕐 or so to download the whole image… so please, be (slightly) patient.

Download the simulation code from Github

The code of the simple simulation described above can be downloaded by clicking in the next code cell:

curl https://raw.githubusercontent.com/pammacdotnet/FFRepo/master/simulation.py -o simulation.py -s

We recommend you take a quick look (even if you are not a Python or particle physics enthusiast) so that you realise how it is very easy to experiment with modern computer simulations thanks to awesome tools such as g4py:

pip install pygments
pygmentize -g simulation.py

Also, download (and enable the execution of) this simple Python script to convert from WRL data to x3dom-enabled HTML content:

curl https://raw.githubusercontent.com/pammacdotnet/FFRepo/master/wrl2html.py -o wrl2html.py -s
chmod +x wrl2html.py

This will allow us to visualise 3D scenes (like the one linked above) in the same Web browser. BTW, this last command (wrl2html.py) uses this online converter made by the company instantreality (part of the Visual Computing System Technologies Group which, in turn, belongs to the reputed Fraunhofer Institute).

First simulation

Let’s run our first simulation. It will consist of a beam of 20 photons, each of 1 MeV that hits a water filled target (with a thickness of 20 cm):

docker run -v ${PWD}:/root pammacdotnet/g4lpwd

You’ll see a bunch of information appearing in the terminal:

**************************************************************
 Geant4 version Name: geant4-10-04-patch-02    (25-May-2018)
                       Copyright : Geant4 Collaboration
                      References : NIM A 506 (2003), 250-303
                                 : IEEE-TNS 53 (2006), 270-278
                                 : NIM A 835 (2016), 186-225
                             WWW : http://geant4.org/
**************************************************************

Visualization Manager instantiating with verbosity "warnings (3)"...
/tracking/storeTrajectory 1
/tracking/storeTrajectory 2

phot:   for  gamma    SubType= 12  BuildTable= 0
      LambdaPrime table from 200 keV to 100 TeV in 61 bins 
      ===== EM models for the G4Region  DefaultRegionForTheWorld ======
       PhotoElectric :  Emin=        0 eV    Emax=      100 TeV   AngularGenSauterGavrila

compt:   for  gamma    SubType= 13  BuildTable= 1
      Lambda table from 100 eV  to 1 MeV, 7 bins per decade, spline: 1
      LambdaPrime table from 1 MeV to 100 TeV in 56 bins 
      ===== EM models for the G4Region  DefaultRegionForTheWorld ======
…
…

This is Geant4 telling you all the calculations it has had to carry out for you in behind the scenes in order to generate a final (and more intuitive) file called simulation.wrl (default name). It is now time to launch a simple web server that will let us download or view the results:

python3 -m http.server 80 > /dev/null 2>&1 &

By using the & symbol we state we want the web server to run in the background and with the expression > /dev/null 2>&1 we guarantee that its output won’t bother us. Once started, the previous file can be downloaded from here ⬇️. It’ll take some time because a VRLM file can be quite big.

Once in your hard drive, you can 3D-navigate/fly through this file (or any VRML scene) with any of the following software:

Here you can see a screenshot of view3dscene (Windows version):

However, it is better to transform the WRL file into WebGL-enabled HTML with the wrl2html.py command tackled above:

./wrl2html.py

By default, this last command assumes the input WRL file to have the name simulation.wrl and outputs an HTML file called, by default, simulation.html. You previous instruction is equivalent to:

./wrl2html.py --wrl simulation.wrl --html simulation.html

You can fly through this 3D HTML version by clicking here ↗️, or you can list all files 📁.

More simulations

Let’s explore more simulations! You achieve this by altering any of the configurable parameters:

Particle type. Available with the option --type. Possible values are e-, e+ or gamma. The default value is gamma (also known as photons or, simply, light).
Particle energy. Available with the option --energy. All values are expressed in MeV and should be integer values. The default values is 1 (MeV). Bear in mind that X-rays have more or less 0.1 MeV of energy.
Phantom material. Available with the option --material. Geant4 defines many materials: G4_Al, G4_Si, G4_Ar, G4_Cu, G4_Fe, G4_Ge, G4_Ag, G4_W, G4_Au, G4_Pb, G4_AIR, G4_Galactic, G4_WATER, G4_CESIUM_IODIDE, G4_SODIUM_IODIDE, G4_PLASTIC_SC_VINYLTOLUENE, G4_MYLAR, etc. The default value in our simulations is water (G4_WATER). Do you remember that the human body is composed of 70% water?
Phantom thickness. Available with the option --size. It is the size (along the longitudinal axis) of the phantom, that is, how much matter the beam of particle will have to traverse.
Output file. Available with the option --wrl. It represents the name of the VRML file with a 3D representation of the simulation. The default value is simulation.wrl.
Particle count. Available with the option --count. It is the number of particles within the simulated beam. The default value is 20. We suggest that you keep this value under 💯. Otherwise you’ll end up with a huge WRL file, very difficult to manage and visually analyse.

Antimatter to the rescue!

For instance, the following command:

docker run -v ${PWD}:/root pammacdotnet/g4lpwd --type=e+

will exchange photons for positrons. Positrons are the antiparticles of electrons. The major difference from electrons is their positive charge. You can either download again the WRL file ⬇️ or generate a new HTML file. If you run again the wrl2html.py command with no arguments, the original simulation.html file will be overwritten ⚠️. However you can change the name of the resulting HTML file:

./wrl2html.py --html simulation2.html

This file is still browsable ↗️.

Positrons tend to annihilate with electrons. When this happens, two photons emerge in opposite directions. As you can see from the following picture, the positrons (yellow traces) suddenly seem to disappear. What is happening is that those positrons find a corresponding electron (attached to the atoms that compose the material) and the mass of both particles is transformed into two photons emerging in any random direction (but opposite ways).

Light seems to zigzag!

Another effect that you will see in the previous simulation is that light rays seem to bend with sharp angles. This is due to the Compton scattering. In this process, the path of a photon is diverted by a charged particle, usually an electron. In each scattering event, the ray of light loses energy (decreases its frequency) and its finally absorbed by the material.

Superman (o Supergirl)-approved shielding!

Let’s now change the particle type to a more common one (electrons) and let’s have them collide against a lead target:

docker run -v ${PWD}:/root pammacdotnet/g4lpwd --type=e- --material=G4_Pb --wrl simulation3.wrl
./wrl2html.py --wrl simulation3.wrl --html simulation3.html

The results are displayed here ↗️. Again, you can specify a new WRL file (with the --wrl option). As you can see, 1 MeV electrons are unable to penetrate the material. On the contrary, some of them seem to bounce back. This is due to electrostatic repulsion against the electronic cloud covering this lead shield.

Beware of radiation!

Let’s repeat the same simulation with a (virtual) water target and with a greater energy:

docker run -v ${PWD}:/root pammacdotnet/g4lpwd --type=e- --energy=100 --wrl simulation4.wrl
./wrl2html.py --wrl simulation4.wrl --html simulation4.html

If you carefully observe the simulation results ↗️, you’ll notice the famous electromagnetic showers. This is the same process that occurs when a cosmic ray hits the upper atmosphere and triggers the generation of new particles. The primary process that we are watching here is bremsstrahlung, which consists in the production of electromagnetic radiation (a photon, represented by a white line) when an electron decelerates by the presence of matter.

Remember you can either download the resulting WRL file or navigate through the x3dom-enabled HTML file.

Ionization

Many of the already proposed simulations exhibit ionisations. In this type of interaction, a photon hits an electron (attached to an atom) and produces an ion (i.e., an atom negatively charged).

The previous simulation was obtained with the following command (and whose results can be explored here ↗️):

docker run -v ${PWD}:/root pammacdotnet/g4lpwd --type=gamma --energy=10 --material=G4_Al --wrl simulation5.wrl
./wrl2html.py --wrl simulation5.wrl --html simulation5.html

Ionisations can also take place when a moving electron hits an atom at rest (well, one of its orbiting electrons), that is, the ionisation particle can be of the same nature as the removed one.

Simulation limits and photoelectric absorption

Let’s go back a water-filled cuboid and low energies, but let’s insanely increase the number of particles (photons). Run the following command in the terminal:

docker run -v ${PWD}:/root pammacdotnet/g4lpwd --type=gamma --energy=2 --count=100 --wrl simulation6.wrl
./wrl2html.py --wrl simulation6.wrl --html simulation6.html

You’ll see an scenario ↗️ very similar to this one, where the photons, after traveling in straight lines and/or interacting with matter, seem to disappear:

Well, where did all the photons go? Well, some of them reach the world limits… Do you remember the movie The 13th Floor?

When this happens, Geant4 stops calculating any further interactions and any particle trace that crosses the boundaries of the simulation seems to suddenly vanish. However, even inside the simulation realm, some photons seem to evaporate. What’s happening? The answer is simple: they are absorbed by the material. With more detail, these absorption processes correspond to the photoelectric effect: photons are completely absorbed by electrons.

Here you have a second simulation that better shows the previous processes:

This scene can be recreated with the following command, where a beam of just a few electrons hits an iron target:

docker run -v ${PWD}:/root pammacdotnet/g4lpwd --type=e- --material=G4_Fe --count=10 --energy=50 --wrl simulation7.wrl
./wrl2html.py --wrl simulation7.wrl --html simulation7.html

Each electron produces in turn an electromagnetic shower. These showers consist of a bunch of photons that end up photoelectrically absorbed. Take a look ↗️ for yourself (of course, after running the simulation)!

Energy is a kind of matter

Yes, energy can be transformed into matter (and back!)… as discovered by Einstein in his famous 1905 paper: On the Electrodynamics of Moving Bodies. BTW, Einstein was also responsible for the explanation of the photoelectric effect, as commented in the introduction above.

(Not that Einstein!)

Run the following simulation:

docker run -v ${PWD}:/root pammacdotnet/g4lpwd --type=gamma --material=G4_WATER --count=10 --energy=15 --wrl simulation8.wrl
./wrl2html.py --wrl simulation8.wrl --html simulation8.html

You will see ↗️ some events like the following, where a photon (with no mass and thus made of pure energy), after colliding with an atom, transfers all its momentum to the creation of two particles: an electron and anti-electron (positron).

That is: energy (E) is eventually converted into matter (m), following the famous Einstein equation: E=m·c·c, where c is the speed of light. This process is known as pair production.

Final words

Now it’s time for you to play with your own simulations. This includes other materials, energies, particles, etc. As stated above, you can also increase the number of particles, but we suggest you keep this number below 100. This lab is based on the article X-ray imaging virtual online laboratory for engineering undergraduates (by Alberto Corbi, Daniel Burgos, Frank Vidal, Francisco Albiol and Alberto Albiol) and it has been carried out with success in the first semesters of the Computing Science degree at the School of Engineering of Universidad Internacional de La Rioja (UNIR).

.NET Conf 2019

Wed, 04 Sep 2019 00:00:00 +0000

Welcome to Docker’s .NET Conf challenge!

This lab gets you using .NET Core in Docker containers. You’ll experience compiling code, packaging apps and running containers, using the latest .NET Core 3.0 release.

Difficulty: Beginner (assumes no familiarity with Docker)

Time: Approximately 5 minutes

Tasks:

Task 1: Grab the code

Task 2: Build the application image

Task 3: Run a .NET Core container!

Task 1: Grab the code

The Play-with-Docker environment is already set up with Docker and Git, so you’re good to go.

Clone the source code from GitHub

Clone the application source into your local session:

Just click the text in these boxes to send the command to your terminal

git clone https://github.com/dockersamples/dotnetconf19.git

Now browse to the source code folder dotnetconf19:

cd dotnetconf19

In there you’ll see a folder called src which contains the .NET code and a file called Dockerfile whch contains the instructions to build and package the app:

ls

Task 2: Build the application image

The Dockerfile has dotnet commands to restore NuGet packages and publish the app:

cat Dockerfile

Even if you’re not familiar with the Dockerfile syntax, you can kind of work out that this is a script to compile the app and package it up to run

You run the script with the docker image build command, which produces a container package called a Docker image:

docker image build --tag dotnetconf:19 .

You’ll see lots of download progress bars, and some familiar output from MSBuild.

The final message Successfully tagged dotnetconf:19 tells you the image has been built and given the tag dotnetconf:19 - which is just the image name.

Task 3: Run a .NET Core container!

A Docker image is a complete packaged app. You can share it on Docker Hub, which is how thousands of open-source and commercial projects now distribute their software.

Your image contains the .NET Core 3.0 runtime, together with the assemblies and configuration for the demo app.

You run the app by running a container from the image:

docker container run dotnetconf:19

Scroll up to read the message from .NET Bot - that’s your .NET Conf code!

If you want to learn more about what’s happened here and how you can use Docker to build cross-platform apps that run on Windows, Linux, Itenal and Arm - check out this blog post:

Docker + Arm Virtual Meetup Recap: Building Multi-arch Apps with Buildx.

You’re done!

Enjoy .NET Conf :)

The Play with Docker Training site is always on, and there are plenty more labs you can try at home.

Docker for Beginners - Linux

Thu, 01 Aug 2019 00:00:00 +0000

In this lab, we will look at some basic Docker commands and a simple build-ship-run workflow. We’ll start by running some simple containers, then we’ll use a Dockerfile to build a custom app. Finally, we’ll look at how to use bind mounts to modify a running container as you might if you were actively developing using Docker.

Difficulty: Beginner (assumes no familiarity with Docker)

Time: Approximately 30 minutes

Tasks:

Task 0: Prerequisites

Task 1: Run some simple Docker containers

Task 2: Package and run a custom app using Docker

Task 3: Modify a Running Website

Task 0: Prerequisites

You will need all of the following to complete this lab:

A clone of the lab’s GitHub repo.
A DockerID.

Clone the Lab’s GitHub Repo

Use the following command to clone the lab’s repo from GitHub (you can click the command or manually type it). This will make a copy of the lab’s repo in a new sub-directory called linux_tweet_app.

    git clone https://github.com/dockersamples/linux_tweet_app

Make sure you have a DockerID

If you do not have a DockerID (a free login used to access Docker Hub), please visit Docker Hub and register for one. You will need this for later steps.

Task 1: Run some simple Docker containers

There are different ways to use containers. These include:

To run a single task: This could be a shell script or a custom app.
Interactively: This connects you to the container similar to the way you SSH into a remote server.
In the background: For long-running services like websites and databases.

In this section you’ll try each of those options and see how Docker manages the workload.

Run a single task in an Alpine Linux container

In this step we’re going to start a new container and tell it to run the hostname command. The container will start, execute the hostname command, then exit.

Run the following command in your Linux console.

 docker container run alpine hostname

The output below shows that the alpine:latest image could not be found locally. When this happens, Docker automatically pulls it from Docker Hub.

After the image is pulled, the container’s hostname is displayed (888e89a3b36b in the example below).

 Unable to find image 'alpine:latest' locally
 latest: Pulling from library/alpine
 88286f41530e: Pull complete
 Digest: sha256:f006ecbb824d87947d0b51ab8488634bf69fe4094959d935c0c103f4820a417d
 Status: Downloaded newer image for alpine:latest
 888e89a3b36b

Docker keeps a container running as long as the process it started inside the container is still running. In this case the hostname process exits as soon as the output is written. This means the container stops. However, Docker doesn’t delete resources by default, so the container still exists in the Exited state.

List all containers.
```
 docker container ls --all
```
Notice that your Alpine Linux container is in the Exited state.
```
 CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS            PORTS               NAMES
 888e89a3b36b        alpine              "hostname"          50 seconds ago      Exited (0) 49 seconds ago                       awesome_elion
```
Note: The container ID is the hostname that the container displayed. In the example above it’s 888e89a3b36b.

Containers which do one task and then exit can be very useful. You could build a Docker image that executes a script to configure something. Anyone can execute that task just by running the container - they don’t need the actual scripts or configuration information.

Run an interactive Ubuntu container

You can run a container based on a different version of Linux than is running on your Docker host.

In the next example, we are going to run an Ubuntu Linux container on top of an Alpine Linux Docker host (Play With Docker uses Alpine Linux for its nodes).

Run a Docker container and access its shell.
```
 docker container run --interactive --tty --rm ubuntu bash
```
In this example, we’re giving Docker three parameters:
- --interactive says you want an interactive session.
- --tty allocates a pseudo-tty.
- --rm tells Docker to go ahead and remove the container when it’s done executing.
The first two parameters allow you to interact with the Docker container.

We’re also telling the container to run bash as its main process (PID 1).

When the container starts you’ll drop into the bash shell with the default prompt root@<container id>:/#. Docker has attached to the shell in the container, relaying input and output between your local session and the shell session in the container.
Run the following commands in the container.

ls / will list the contents of the root directory in the container, ps aux will show running processes in the container, cat /etc/issue will show which Linux distro the container is running, in this case Ubuntu 20.04.3 LTS.
```
ls /
```
```
ps aux
```
```
cat /etc/issue
```
Type exit to leave the shell session. This will terminate the bash process, causing the container to exit.
```
 exit
```
Note: As we used the --rm flag when we started the container, Docker removed the container when it stopped. This means if you run another docker container ls --all you won’t see the Ubuntu container.

For fun, let’s check the version of our host VM.

 cat /etc/issue

You should see:

 Welcome to Alpine Linux 3.8
 Kernel \r on an \m (\l)

Notice that our host VM is running Alpine Linux, yet we were able to run an Ubuntu container. As previously mentioned, the distribution of Linux inside the container does not need to match the distribution of Linux running on the Docker host.

However, Linux containers require the Docker host to be running a Linux kernel. For example, Linux containers cannot run directly on Windows Docker hosts. The same is true of Windows containers - they need to run on a Docker host with a Windows kernel.

Interactive containers are useful when you are putting together your own image. You can run a container and verify all the steps you need to deploy your app, and capture them in a Dockerfile.

You can commit a container to make an image from it - but you should avoid that wherever possible. It’s much better to use a repeatable Dockerfile to build your image. You’ll see that shortly.

Run a background MySQL container

Background containers are how you’ll run most applications. Here’s a simple example using MySQL.

Run a new MySQL container with the following command.

 docker container run \
 --detach \
 --name mydb \
 -e MYSQL_ROOT_PASSWORD=my-secret-pw \
 mysql:latest

--detach will run the container in the background.
--name will name it mydb.
-e will use an environment variable to specify the root password (NOTE: This should never be done in production).

As the MySQL image was not available locally, Docker automatically pulled it from Docker Hub.

 Unable to find image 'mysql:latest' locallylatest: Pulling from library/mysql
 aa18ad1a0d33: Pull complete
 fdb8d83dece3: Pull complete
 75b6ce7b50d3: Pull complete
 ed1d0a3a64e4: Pull complete
 8eb36a82c85b: Pull complete
 41be6f1a1c40: Pull complete
 0e1b414eac71: Pull complete
 914c28654a91: Pull complete
 587693eb988c: Pull complete
 b183c3585729: Pull complete
 315e21657aa4: Pull complete
 Digest: sha256:0dc3dacb751ef46a6647234abdec2d47400f0dfbe77ab490b02bffdae57846ed
 Status: Downloaded newer image for mysql:latest
 41d6157c9f7d1529a6c922acb8167ca66f167119df0fe3d86964db6c0d7ba4e0

As long as the MySQL process is running, Docker will keep the container running in the background.

List the running containers.

 docker container ls

Notice your container is running.

 CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS            NAMES
 3f4e8da0caf7        mysql:latest        "docker-entrypoint..."   52 seconds ago      Up 51 seconds       3306/tcp            mydb

You can check what’s happening in your containers by using a couple of built-in Docker commands: docker container logs and docker container top.

 docker container logs mydb

This shows the logs from the MySQL Docker container.

   <output truncated>
   2017-09-29T16:02:58.605004Z 0 [Note] Executing 'SELECT * FROM INFORMATION_SCHEMA.TABLES;' to get a list of tables using the deprecated partition engine. You may use the startup option '--disable-partition-engine-check' to skip this check.
   2017-09-29T16:02:58.605026Z 0 [Note] Beginning of list of non-natively partitioned tables
   2017-09-29T16:02:58.616575Z 0 [Note] End of list of non-natively partitioned tables

Let’s look at the processes running inside the container.

   docker container top mydb

You should see the MySQL daemon (mysqld) is running in the container.

 PID                 USER                TIME                COMMAND
 2876                999                 0:00                mysqld

Although MySQL is running, it is isolated within the container because no network ports have been published to the host. Network traffic cannot reach containers from the host unless ports are explicitly published.

List the MySQL version using docker container exec.

docker container exec allows you to run a command inside a container. In this example, we’ll use docker container exec to run the command-line equivalent of mysql --user=root --password=$MYSQL_ROOT_PASSWORD --version inside our MySQL container.
```
 docker exec -it mydb \
 mysql --user=root --password=$MYSQL_ROOT_PASSWORD --version
```
You will see the MySQL version number, as well as a handy warning.
```
 mysql: [Warning] Using a password on the command line interface can be insecure.
 mysql  Ver 14.14 Distrib 5.7.19, for Linux (x86_64) using  EditLine wrapper
```
You can also use docker container exec to connect to a new shell process inside an already-running container. Executing the command below will give you an interactive shell (sh) inside your MySQL container.
```
 docker exec -it mydb sh
```
Notice that your shell prompt has changed. This is because your shell is now connected to the sh process running inside of your container.
Let’s check the version number by running the same command again, only this time from within the new shell session in the container.
```
 mysql --user=root --password=$MYSQL_ROOT_PASSWORD --version
```
Notice the output is the same as before.
Type exit to leave the interactive shell session.
```
 exit
```

Task 2: Package and run a custom app using Docker

In this step you’ll learn how to package your own apps as Docker images using a Dockerfile.

The Dockerfile syntax is straightforward. In this task, we’re going to create a simple NGINX website from a Dockerfile.

Build a simple website image

Let’s have a look at the Dockerfile we’ll be using, which builds a simple website that allows you to send a tweet.

Make sure you’re in the linux_tweet_app directory.
```
 cd ~/linux_tweet_app
```
Display the contents of the Dockerfile.
```
 cat Dockerfile
```
```
 FROM nginx:latest

 COPY index.html /usr/share/nginx/html
 COPY linux.png /usr/share/nginx/html

 EXPOSE 80 443     

 CMD ["nginx", "-g", "daemon off;"]
```
Let’s see what each of these lines in the Dockerfile do.
- FROM specifies the base image to use as the starting point for this new image you’re creating. For this example we’re starting from nginx:latest.
- COPY copies files from the Docker host into the image, at a known location. In this example, COPY is used to copy two files into the image: index.html. and a graphic that will be used on our webpage.
- EXPOSE documents which ports the application uses.
- CMD specifies what command to run when a container is started from the image. Notice that we can specify the command, as well as run-time arguments.
In order to make the following commands more copy/paste friendly, export an environment variable containing your DockerID (if you don’t have a DockerID you can get one for free via Docker Hub).

You will have to manually type this command as it requires your unique DockerID.

export DOCKERID=<your docker id>
Echo the value of the variable back to the terminal to ensure it was stored correctly.
```
 echo $DOCKERID
```

Use the docker image build command to create a new Docker image using the instructions in the Dockerfile.

--tag allows us to give the image a custom name. In this case it’s comprised of our DockerID, the application name, and a version. Having the Docker ID attached to the name will allow us to store it on Docker Hub in a later step
. tells Docker to use the current directory as the build context

Be sure to include period (.) at the end of the command.

 docker image build --tag $DOCKERID/linux_tweet_app:1.0 .

The output below shows the Docker daemon executing each line in the Dockerfile

 Sending build context to Docker daemon  32.77kB
 Step 1/5 : FROM nginx:latest
 latest: Pulling from library/nginx
 afeb2bfd31c0: Pull complete
 7ff5d10493db: Pull complete
 d2562f1ae1d0: Pull complete
 Digest: sha256:af32e714a9cc3157157374e68c818b05ebe9e0737aac06b55a09da374209a8f9
 Status: Downloaded newer image for nginx:latest
 ---> da5939581ac8
 Step 2/5 : COPY index.html /usr/share/nginx/html
 ---> eba2eec2bea9
 Step 3/5 : COPY linux.png /usr/share/nginx/html
 ---> 4d080f499b53
 Step 4/5 : EXPOSE 80 443
 ---> Running in 47232cb5699f
 ---> 74c968a9165f
 Removing intermediate container 47232cb5699f
 Step 5/5 : CMD nginx -g daemon off;
 ---> Running in 4623761274ac
 ---> 12045a0df899
 Removing intermediate container 4623761274ac
 Successfully built 12045a0df899
 Successfully tagged <your docker ID>/linux_tweet_app:latest

Use the docker container run command to start a new container from the image you created.

As this container will be running an NGINX web server, we’ll use the --publish flag to publish port 80 inside the container onto port 80 on the host. This will allow traffic coming in to the Docker host on port 80 to be directed to port 80 in the container. The format of the --publish flag is host_port:container_port.
```
 docker container run \
 --detach \
 --publish 80:80 \
 --name linux_tweet_app \
 $DOCKERID/linux_tweet_app:1.0
```
Any external traffic coming into the server on port 80 will now be directed into the container on port 80.

In a later step you will see how to map traffic from two different ports - this is necessary when two containers use the same port to communicate since you can only expose the port once on the host.
Click here to load the website which should be running.
Once you’ve accessed your website, shut it down and remove it.
```
 docker container rm --force linux_tweet_app
```
Note: We used the --force parameter to remove the running container without shutting it down. This will ungracefully shutdown the container and permanently remove it from the Docker host.

In a production environment you may want to use docker container stop to gracefully stop the container and leave it on the host. You can then use docker container rm to permanently remove it.

Task 3: Modify a running website

When you’re actively working on an application it is inconvenient to have to stop the container, rebuild the image, and run a new version every time you make a change to your source code.

One way to streamline this process is to mount the source code directory on the local machine into the running container. This will allow any changes made to the files on the host to be immediately reflected in the container.

We do this using something called a bind mount.

When you use a bind mount, a file or directory on the host machine is mounted into a container running on the same host.

Start our web app with a bind mount

Let’s start the web app and mount the current directory into the container.

In this example we’ll use the --mount flag to mount the current directory on the host into /usr/share/nginx/html inside the container.

Be sure to run this command from within the linux_tweet_app directory on your Docker host.
```
 docker container run \
 --detach \
 --publish 80:80 \
 --name linux_tweet_app \
 --mount type=bind,source="$(pwd)",target=/usr/share/nginx/html \
 $DOCKERID/linux_tweet_app:1.0
```
Remember from the Dockerfile, /usr/share/nginx/html is where the html files are stored for the web app.
The website should be running.

Modify the running website

Bind mounts mean that any changes made to the local file system are immediately reflected in the running container.

Copy a new index.html into the container.

The Git repo that you pulled earlier contains several different versions of an index.html file. You can manually run an ls command from within the ~/linux_tweet_app directory to see a list of them. In this step we’ll replace index.html with index-new.html.
```
 cp index-new.html index.html
```
Go to the running website and refresh the page. Notice that the site has changed.

If you are comfortable with vi you can use it to load the local index.html file and make additional changes. Those too would be reflected when you reload the webpage. If you are really adventurous, why not try using exec to access the running container and modify the files stored there.

Even though we’ve modified the index.html local filesystem and seen it reflected in the running container, we’ve not actually changed the Docker image that the container was started from.

To show this, stop the current container and re-run the 1.0 image without a bind mount.

Stop and remove the currently running container.
```
 docker rm --force linux_tweet_app
```

Rerun the current version without a bind mount.

 docker container run \
 --detach \
 --publish 80:80 \
 --name linux_tweet_app \
 $DOCKERID/linux_tweet_app:1.0

Notice the website is back to the original version.
Stop and remove the current container
```
docker rm --force linux_tweet_app
```

Update the image

To persist the changes you made to the index.html file into the image, you need to build a new version of the image.

Build a new image and tag it as 2.0

Remember that you previously modified the index.html file on the Docker hosts local filesystem. This means that running another docker image build command will build a new image with the updated index.html

Be sure to include the period (.) at the end of the command.
```
 docker image build --tag $DOCKERID/linux_tweet_app:2.0 .
```
Notice how fast that built! This is because Docker only modified the portion of the image that changed vs. rebuilding the whole image.

Let’s look at the images on the system.

 docker image ls

You now have both versions of the web app on your host.

 REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
 <docker id>/linux_tweet_app    2.0                 01612e05312b        16 seconds ago      108MB
 <docker id>/linux_tweet_app    1.0                 bb32b5783cd3        4 minutes ago       108MB
 mysql                          latest              b4e78b89bcf3        2 weeks ago         412MB
 ubuntu                         latest              2d696327ab2e        2 weeks ago         122MB
 nginx                          latest              da5939581ac8        3 weeks ago         108MB
 alpine                         latest              76da55c8019d        3 weeks ago         3.97MB

Test the new version

Run a new container from the new version of the image.

 docker container run \
 --detach \
 --publish 80:80 \
 --name linux_tweet_app \
 $DOCKERID/linux_tweet_app:2.0

Check the new version of the website (You may need to refresh your browser to get the new version to load).

The web page will have an orange background.

We can run both versions side by side. The only thing we need to be aware of is that we cannot have two containers using port 80 on the same host.

As we’re already using port 80 for the container running from the 2.0 version of the image, we will start a new container and publish it on port 8080. Additionally, we need to give our container a unique name (old_linux_tweet_app)
Run another new container, this time from the old version of the image.

Notice that this command maps the new container to port 8080 on the host. This is because two containers cannot map to the same port on a single Docker host.
```
 docker container run \
 --detach \
 --publish 8080:80 \
 --name old_linux_tweet_app \
 $DOCKERID/linux_tweet_app:1.0
```
View the old version of the website.

Push your images to Docker Hub

List the images on your Docker host.
```
 docker image ls -f reference="$DOCKERID/*"
```
You will see that you now have two linux_tweet_app images - one tagged as 1.0 and the other as 2.0.
```
 REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
 <docker id>/linux_tweet_app    2.0                 01612e05312b        3 minutes ago       108MB
 <docker id>/linux_tweet_app    1.0                 bb32b5783cd3        7 minutes ago       108MB
```
These images are only stored in your Docker hosts local repository. Your Docker host will be deleted after the workshop. In this step we’ll push the images to a public repository so you can run them from any Linux machine with Docker.

Distribution is built into the Docker platform. You can build images locally and push them to a public or private registry, making them available to other users. Anyone with access can pull that image and run a container from it. The behavior of the app in the container will be the same for everyone, because the image contains the fully-configured app - the only requirements to run it are Linux and Docker.

Docker Hub is the default public registry for Docker images.
Before you can push your images, you will need to log into Docker Hub.
```
 docker login
```
You will need to supply your Docker ID credentials when prompted.
```
 Username: <your docker id>
 Password: <your docker id password>
 Login Succeeded
```

Push version 1.0 of your web app using docker image push.

 docker image push $DOCKERID/linux_tweet_app:1.0

You’ll see the progress as the image is pushed up to Docker Hub.

 The push refers to a repository [docker.io/<your docker id>/linux_tweet_app]
 910e84bcef7a: Pushed
 1dee161c8ba4: Pushed
 110566462efa: Pushed
 305e2b6ef454: Pushed
 24e065a5f328: Pushed
 1.0: digest: sha256:51e937ec18c7757879722f15fa1044cbfbf2f6b7eaeeb578c7c352baba9aa6dc size: 1363

Now push version 2.0.

 docker image push $DOCKERID/linux_tweet_app:2.0

Notice that several lines of the output say Layer already exists. This is because Docker will leverage read-only layers that are the same as any previously uploaded image layers.

 The push refers to a repository [docker.io/<your docker id>/linux_tweet_app]
 0b171f8fbe22: Pushed
 70d38c767c00: Pushed
 110566462efa: Layer already exists
 305e2b6ef454: Layer already exists
 24e065a5f328: Layer already exists
 2.0: digest: sha256:7c51f77f90b81e5a598a13f129c95543172bae8f5850537225eae0c78e4f3add size: 1363

You can browse to https://hub.docker.com/r/<your docker id>/ and see your newly-pushed Docker images. These are public repositories, so anyone can pull the image - you don’t even need a Docker ID to pull public images. Docker Hub also supports private repositories.

Next Step

Check out the introduction to a multi-service application stack orchestration in the Application Containerization and Microservice Orchestration tutorial.

Application Containerization and Microservice Orchestration

Sat, 22 Sep 2018 00:00:00 +0000

In this tutorial we will learn about basic application containerization using Docker and running various components of an application as microservices. We will utilize Docker Compose for orchestration during the development. This tutorial is targeted for beginners who have the basic familiarity with Docker. If you are new to Docker, we recommend you check out Docker for Beginners tutorial first.

We will start from a basic Python script that scrapes links from a given web page and gradually evolve it into a multi-service application stack. The demo code is available in the Link Extractor repo. The code is organized in steps that incrementally introduce changes and new concepts. After completion, the application stack will contain the following microservices:

A web application written in PHP and served using Apache that takes a URL as the input and summarizes extracted links from it
The web application talks to an API server written in Python (and Ruby) that takes care of the link extraction and returns a JSON response
A Redis cache that is used by the API server to avoid repeated fetch and link extraction for pages that are already scraped

The API server will only load the page of the input link from the web if it is not in the cache. The stack will eventually look like the figure below:

This tutorial was initially developed for a colloquium in the Computer Science Department of the Old Dominion University, Norfolk, Virginia. A video recording, presentation slides, and a brief description of the talk can be found in a blog post.

Steps:

Stage Setup

Step 0: Basic Link Extractor Script

Step 1: Containerized Link Extractor Script

Step 2: Link Extractor Module with Full URI and Anchor Text

Step 3: Link Extractor API Service

Step 4: Link Extractor API and Web Front End Services

Step 5: Redis Service for Caching

Step 6: Swap Python API Service with Ruby

Conclusions

Stage Setup

Let’s get started by first cloning the demo code repository, changing the working directory, and checking the demo branch out.

git clone https://github.com/ibnesayeed/linkextractor.git
cd linkextractor
git checkout demo

Step 0: Basic Link Extractor Script

Checkout the step0 branch and list files in it.

git checkout step0
tree

.
├── README.md
└── linkextractor.py

0 directories, 2 files

The linkextractor.py file is the interesting one here, so let’s look at its contents:

cat linkextractor.py

#!/usr/bin/env python

import sys
import requests
from bs4 import BeautifulSoup

res = requests.get(sys.argv[-1])
soup = BeautifulSoup(res.text, "html.parser")
for link in soup.find_all("a"):
    print(link.get("href"))

This is a simple Python script that imports three packages: sys from the standard library and two popular third-party packages requests and bs4. User-supplied command line argument (which is expected to be a URL to an HTML page) is used to fetch the page using the requests package, then parsed using the BeautifulSoup. The parsed object is then iterated over to find all the anchor elements (i.e., <a> tags) and print the value of their href attribute that contains the hyperlink.

However, this seemingly simple script might not be the easiest one to run on a machine that does not meet its requirements. The README.md file suggests how to run it, so let’s give it a try:

./linkextractor.py http://example.com/

bash: ./linkextractor.py: Permission denied

When we tried to execute it as a script, we got the Permission denied error. Let’s check the current permissions on this file:

ls -l linkextractor.py

-rw-r--r--    1 root     root           220 Sep 23 16:26 linkextractor.py

This current permission -rw-r--r-- indicates that the script is not set to be executable. We can either change it by running chmod a+x linkextractor.py or run it as a Python program instead of a self-executing script as illustrated below:

python3 linkextractor.py

Traceback (most recent call last):
  File "linkextractor.py", line 5, in <module>
    from bs4 import BeautifulSoup
ImportError: No module named bs4

Here we got the first ImportError message because we are missing the third-party package needed by the script. We can install that Python package (and potentially other missing packages) using one of the many techniques to make it work, but it is too much work for such a simple script, which might not be obvious for those who are not familiar with Python’s ecosystem.

Depending on which machine and operating system you are trying to run this script on, what software is already installed, and how much access you have, you might face some of these potential difficulties:

Is the script executable?
Is Python installed on the machine?
Can you install software on the machine?
Is pip installed?
Are requests and beautifulsoup4 Python libraries installed?

This is where application containerization tools like Docker come in handy. In the next step we will try to containerize this script and make it easier to execute.

Step 1: Containerized Link Extractor Script

Checkout the step1 branch and list files in it.

git checkout step1
tree

.
├── Dockerfile
├── README.md
└── linkextractor.py

0 directories, 3 files

We have added one new file (i.e., Dockerfile) in this step. Let’s look into its contents:

cat Dockerfile

FROM       python:3
LABEL      maintainer="Sawood Alam <@ibnesayeed>"

RUN        pip install beautifulsoup4
RUN        pip install requests

WORKDIR    /app
COPY       linkextractor.py /app/
RUN        chmod a+x linkextractor.py

ENTRYPOINT ["./linkextractor.py"]

Using this Dockerfile we can prepare a Docker image for this script. We start from the official python Docker image that contains Python’s run-time environment as well as necessary tools to install Python packages and dependencies. We then add some metadata as labels (this step is not essential, but is a good practice nonetheless). Next two instructions run the pip install command to install the two third-party packages needed for the script to function properly. We then create a working directory /app, copy the linkextractor.py file in it, and change its permissions to make it an executable script. Finally, we set the script as the entrypoint for the image.

So far, we have just described how we want our Docker image to be like, but didn’t really build one. So let’s do just that:

docker image build -t linkextractor:step1 .

This command should yield an output as illustrated below:

Sending build context to Docker daemon  171.5kB
Step 1/8 : FROM       python:3

... [OUTPUT REDACTED] ...

Successfully built 226196ada9ab
Successfully tagged linkextractor:step1

We have created a Docker image named linkextractor:step1 based on the Dockerfile illustrated above. If the build was successful, we should be able to see it in the list of image:

docker image ls

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
linkextractor       step1               e067c677be37        2 seconds ago       931MB
python              3                   a9d071760c82        2 weeks ago         923MB

This image should have all the necessary ingredients packaged in it to run the script anywhere on a machine that supports Docker. Now, let’s run a one-off container with this image and extract links from some live web pages:

docker container run -it --rm linkextractor:step1 http://example.com/

This outputs a single link that is present in the simple example.com web page:

http://www.iana.org/domains/example

Let’s try it on a web page with more links in it:

docker container run -it --rm linkextractor:step1 https://training.play-with-docker.com/

/
/about/
#ops
#dev
/ops-stage1
/ops-stage2
/ops-stage3
/dev-stage1
/dev-stage2
/dev-stage3
/alacart
https://twitter.com/intent/tweet?text=Play with Docker Classroom&url=https://training.play-with-docker.com/&via=docker&related=docker
https://facebook.com/sharer.php?u=https://training.play-with-docker.com/
https://plus.google.com/share?url=https://training.play-with-docker.com/
http://www.linkedin.com/shareArticle?mini=true&url=https://training.play-with-docker.com/&title=Play%20with%20Docker%20Classroom&source=https://training.play-with-docker.com
https://2018.dockercon.com/
https://2018.dockercon.com/
https://success.docker.com/training/
https://community.docker.com/registrations/groups/4316
https://docker.com
https://www.docker.com
https://www.facebook.com/docker.run
https://twitter.com/docker
https://www.github.com/play-with-docker/play-with-docker.github.io

This looks good, but we can improve the output. For example, some links are relative, we can convert them into full URLs and also provide the anchor text they are linked to. In the next step we will make these changes and some other improvements to the script.

Step 2: Link Extractor Module with Full URI and Anchor Text

Checkout the step2 branch and list files in it.

git checkout step2
tree

.
├── Dockerfile
├── README.md
└── linkextractor.py

0 directories, 3 files

In this step the linkextractor.py script is updated with the following functional changes:

Paths are normalized to full URLs
Reporting both links and anchor texts
Usable as a module in other scripts

Let’s have a look at the updated script:

cat linkextractor.py

#!/usr/bin/env python

import sys
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

def extract_links(url):
    res = requests.get(url)
    soup = BeautifulSoup(res.text, "html.parser")
    base = url
    # TODO: Update base if a <base> element is present with the href attribute
    links = []
    for link in soup.find_all("a"):
        links.append({
            "text": " ".join(link.text.split()) or "[IMG]",
            "href": urljoin(base, link.get("href"))
        })
    return links

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("\nUsage:\n\t{} <URL>\n".format(sys.argv[0]))
        sys.exit(1)
    for link in extract_links(sys.argv[-1]):
        print("[{}]({})".format(link["text"], link["href"]))

The link extraction logic is abstracted into a function extract_links that accepts a URL as a parameter and returns a list of objects containing anchor texts and normalized hyperlinks. This functionality can now be imported into other scripts as a module (which we will utilize in the next step).

Now, let’s build a new image and see these changes in effect:

docker image build -t linkextractor:step2 .

We have used a new tag linkextractor:step2 for this image so that we don’t overwrite the image from the step1 to illustrate that they can co-exist and containers can be run using either of these images.

docker image ls

REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
linkextractor       step2               be2939eada96        3 seconds ago        931MB
linkextractor       step1               673d045a822f        About a minute ago   931MB
python              3                   a9d071760c82        2 weeks ago          923MB

Running a one-off container using the linkextractor:step2 image should now yield an improved output:

docker container run -it --rm linkextractor:step2 https://training.play-with-docker.com/

[Play with Docker classroom](https://training.play-with-docker.com/)
[About](https://training.play-with-docker.com/about/)
[IT Pros and System Administrators](https://training.play-with-docker.com/#ops)
[Developers](https://training.play-with-docker.com/#dev)
[Stage 1: The Basics](https://training.play-with-docker.com/ops-stage1)
[Stage 2: Digging Deeper](https://training.play-with-docker.com/ops-stage2)
[Stage 3: Moving to Production](https://training.play-with-docker.com/ops-stage3)
[Stage 1: The Basics](https://training.play-with-docker.com/dev-stage1)
[Stage 2: Digging Deeper](https://training.play-with-docker.com/dev-stage2)
[Stage 3: Moving to Staging](https://training.play-with-docker.com/dev-stage3)
[Full list of individual labs](https://training.play-with-docker.com/alacart)
[[IMG]](https://twitter.com/intent/tweet?text=Play with Docker Classroom&url=https://training.play-with-docker.com/&via=docker&related=docker)
[[IMG]](https://facebook.com/sharer.php?u=https://training.play-with-docker.com/)
[[IMG]](https://plus.google.com/share?url=https://training.play-with-docker.com/)
[[IMG]](http://www.linkedin.com/shareArticle?mini=true&url=https://training.play-with-docker.com/&title=Play%20with%20Docker%20Classroom&source=https://training.play-with-docker.com)
[[IMG]](https://2018.dockercon.com/)
[DockerCon 2018 in San Francisco](https://2018.dockercon.com/)
[training.docker.com](https://success.docker.com/training/)
[Register here](https://community.docker.com/registrations/groups/4316)
[Docker, Inc.](https://docker.com)
[[IMG]](https://www.docker.com)
[[IMG]](https://www.facebook.com/docker.run)
[[IMG]](https://twitter.com/docker)
[[IMG]](https://www.github.com/play-with-docker/play-with-docker.github.io)

Running a container using the previous image linkextractor:step1 should still result in the old output:

docker container run -it --rm linkextractor:step1 https://training.play-with-docker.com/

So far, we have learned how to containerize a script with its necessary dependencies to make it more portable. We have also learned how to make changes in the application and build different variants of Docker images that can co-exist. In the next step we will build a web service that will utilize this script and will make the service run inside a Docker container.

Step 3: Link Extractor API Service

Checkout the step3 branch and list files in it.

git checkout step3
tree

.
├── Dockerfile
├── README.md
├── linkextractor.py
├── main.py
└── requirements.txt

0 directories, 5 files

The following changes have been made in this step:

Added a server script main.py that utilizes the link extraction module written in the last step
The Dockerfile is updated to refer to the main.py file instead
Server is accessible as a WEB API at http://<hostname>[:<prt>]/api/<url>
Dependencies are moved to the requirements.txt file
Needs port mapping to make the service accessible outside of the container (the Flask server used here listens on port 5000 by default)

Let’s first look at the Dockerfile for changes:

cat Dockerfile

FROM       python:3
LABEL      maintainer="Sawood Alam <@ibnesayeed>"

WORKDIR    /app
COPY       requirements.txt /app/
RUN        pip install -r requirements.txt

COPY       *.py /app/
RUN        chmod a+x *.py

CMD        ["./main.py"]

Since we have started using requirements.txt for dependencies, we no longer need to run pip install command for individual packages. The ENTRYPOINT directive is replaced with the CMD and it is referring to the main.py script that has the server code it because we do not want to use this image for one-off commands now.

The linkextractor.py module remains unchanged in this step, so let’s look into the newly added main.py file:

cat main.py

#!/usr/bin/env python

from flask import Flask
from flask import request
from flask import jsonify
from linkextractor import extract_links

app = Flask(__name__)

@app.route("/")
def index():
    return "Usage: http://<hostname>[:<prt>]/api/<url>"

@app.route("/api/<path:url>")
def api(url):
    qs = request.query_string.decode("utf-8")
    if qs != "":
        url += "?" + qs
    links = extract_links(url)
    return jsonify(links)

app.run(host="0.0.0.0")

Here, we are importing extract_links function from the linkextractor module and converting the returned list of objects into a JSON response.

It’s time to build a new image with these changes in place:

docker image build -t linkextractor:step3 .

Then run the container in detached mode (-d flag) so that the terminal is available for other commands while the container is still running. Note that we are mapping the port 5000 of the container with the 5000 of the host (using -p 5000:5000 argument) to make it accessible from the host. We are also assigning a name (--name=linkextractor) to the container to make it easier to see logs and kill or remove the container.

docker container run -d -p 5000:5000 --name=linkextractor linkextractor:step3

If things go well, we should be able to see the container being listed in Up condition:

docker container ls

CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUSPORTS                    NAMES
d69c0150a754        linkextractor:step3   "./main.py"         9 seconds ago       Up 8 seconds0.0.0.0:5000->5000/tcp   linkextractor

We can now make an HTTP request in the form /api/<url> to talk to this server and fetch the response containing extracted links:

curl -i http://localhost:5000/api/http://example.com/

HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 78
Server: Werkzeug/0.14.1 Python/3.7.0
Date: Sun, 23 Sep 2018 20:52:56 GMT

[{"href":"http://www.iana.org/domains/example","text":"More information..."}]

Now, we have the API service running that accepts requests in the form /api/<url> and responds with a JSON containing hyperlinks and anchor texts of all the links present in the web page at give <url>.

Since the container is running in detached mode, so we can’t see what’s happening inside, but we can see logs using the name linkextractor we assigned to our container:

docker container logs linkextractor

* Serving Flask app "main" (lazy loading)
* Environment: production
  WARNING: Do not use the development server in a production environment.
  Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
172.17.0.1 - - [23/Sep/2018 20:52:56] "GET /api/http://example.com/ HTTP/1.1" 200 -

We can see the messages logged when the server came up, and an entry of the request log when we ran the curl command. Now we can kill and remove this container:

docker container rm -f linkextractor

In this step we have successfully ran an API service listening on port 5000. This is great, but APIs and JSON responses are for machines, so in the next step we will run a web service with a human-friendly web interface in addition to this API service.

Step 4: Link Extractor API and Web Front End Services

Checkout the step4 branch and list files in it.

git checkout step4
tree

.
├── README.md
├── api
│   ├── Dockerfile
│   ├── linkextractor.py
│   ├── main.py
│   └── requirements.txt
├── docker-compose.yml
└── www
    └── index.php

2 directories, 7 files

In this step the following changes have been made since the last step:

The link extractor JSON API service (written in Python) is moved in a separate ./api folder that has the exact same code as in the previous step
A web front-end application is written in PHP under ./www folder that talks to the JSON API
The PHP application is mounted inside the official php:7-apache Docker image for easier modification during the development
The web application is made accessible at http://<hostname>[:<prt>]/?url=<url-encoded-url>
An environment variable API_ENDPOINT is used inside the PHP application to configure it to talk to the JSON API server
A docker-compose.yml file is written to build various components and glue them together

In this step we are planning to run two separate containers, one for the API and the other for the web interface. The latter needs a way to talk to the API server. For the two containers to be able to talk to each other, we can either map their ports on the host machine and use that for request routing or we can place the containers in a single private network and access directly. Docker has excellent support for networking and provides helpful commands for dealing with networks. Additionally, in a Docker network containers identify themselves using their names as hostnames to avoid hunting for their IP addresses in the private network. However, we are not going to do any of this manually, instead we will be using Docker Compose to automate many of these tasks.

So let’s look at the docker-compose.yml file we have:

cat docker-compose.yml

version: '3'

services:
  api:
    image: linkextractor-api:step4-python
    build: ./api
    ports:
      - "5000:5000"
  web:
    image: php:7-apache
    ports:
      - "80:80"
    environment:
      - API_ENDPOINT=http://api:5000/api/
    volumes:
      - ./www:/var/www/html

This is a simple YAML file that describes the two services api and web. The api service will use the linkextractor-api:step4-python image that is not built yet, but will be built on-demand using the Dockerfile from the ./api directory. This service will be exposed on the port 5000 of the host.

The second service named web will use official php:7-apache image directly from the DockerHub, that’s why we do not have a Dockerfile for it. The service will be exposed on the default HTTP port (i.e., 80). We will supply an environment variable named API_ENDPOINT with the value http://api:5000/api/ to tell the PHP script where to connect to for the API access. Notice that we are not using an IP address here, instead, api:5000 is being used because we will have a dynamic hostname entry in the private network for the API service matching its service name. Finally, we will bind mount the ./www folder to make the index.php file available inside of the web service container at /var/www/html, which is the default web root for the Apache web server.

Now, let’s have a look at the user-facing www/index.php file:

cat www/index.php

This is a long file that mainly contains all the markup and styles of the page. However, the important block of code is in the beginning of the file as illustrated below:

$api_endpoint = $_ENV["API_ENDPOINT"] ?: "http://localhost:5000/api/";
$url = "";
if(isset($_GET["url"]) && $_GET["url"] != "") {
  $url = $_GET["url"];
  $json = @file_get_contents($api_endpoint . $url);
  if($json == false) {
    $err = "Something is wrong with the URL: " . $url;
  } else {
    $links = json_decode($json, true);
    $domains = [];
    foreach($links as $link) {
      array_push($domains, parse_url($link["href"], PHP_URL_HOST));
    }
    $domainct = @array_count_values($domains);
    arsort($domainct);
  }
}

The $api_endpoint variable is initialized with the value of the environment variable supplied from the docker-compose.yml file as $_ENV["API_ENDPOINT"] (otherwise falls back to a default value of http://localhost:5000/api/). The request is made using file_get_contents function that uses the $api_endpoint variable and user supplied URL from $_GET["url"]. Some analysis and transformations are performed on the received response that are later used in the markup to populate the page.

Let’s bring these services up in detached mode using docker-compose utility:

docker-compose up -d --build

Creating network "linkextractor_default" with the default driver
Pulling web (php:7-apache)...
7-apache: Pulling from library/php

... [OUTPUT REDACTED] ...

Status: Downloaded newer image for php:7-apache
Building api
Step 1/8 : FROM       python:3

... [OUTPUT REDACTED] ...

Successfully built 1f419be1c2bf
Successfully tagged linkextractor-api:step4-python
Creating linkextractor_web_1 ... done
Creating linkextractor_api_1 ... done

This output shows that Docker Compose automatically created a network named linkextractor_default, pulled php:7-apache image from DockerHub, built api:python image using our local Dockerfile, and finally, spun two containers linkextractor_web_1 and linkextractor_api_1 that correspond to the two services we have defined in the YAML file above.

Checking for the list of running containers confirms that the two services are indeed running:

docker container ls

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS   PORTS                    NAMES
268b021b5a2c        php:7-apache                     "docker-php-entrypoi…"   3 minutes ago       Up 3 minutes        0.0.0.0:80->80/tcp       linkextractor_web_1
5bc266b4e43d        linkextractor-api:step4-python   "./main.py"              3 minutes ago       Up 3 minutes        0.0.0.0:5000->5000/tcp   linkextractor_api_1

We should now be able to talk to the API service as before:

curl -i http://localhost:5000/api/http://example.com/

To access the web interface click to open the Link Extractor. Then fill the form with https://training.play-with-docker.com/ (or any HTML page URL of your choice) and submit to extract links from it.

We have just created an application with microservice architecture, isolating individual tasks in separate services as opposed to monolithic applications where everything is put together in a single unit. Microservice applications are relatively easier to scale, maintains, and move around. They also allow easy swapping of components with an equivalent service. More on that later.

Now, let’s modify the www/index.php file to replace all occurrences of Link Extractor with Super Link Extractor:

sed -i 's/Link Extractor/Super Link Extractor/g' www/index.php

Reloading the web interface of the application (or clicking here) should now reflect this change in the title, header, and footer. This is happening because the ./www folder is bind mounted inside of the container, so any changes made outside will reflect inside the container or the vice versa. This approach is very helpful in development, but in the production environment we would prefer our Docker images to be self-contained. Let’s revert these changes now to clean the Git tracking:

git reset --hard

Before we move on to the next step we need to shut these services down, but Docker Compose can help us take care of it very easily:

docker-compose down

Stopping linkextractor_api_1 ... done
Stopping linkextractor_web_1 ... done
Removing linkextractor_api_1 ... done
Removing linkextractor_web_1 ... done
Removing network linkextractor_default

In the next step we will add one more service to our stack and will build a self-contained custom image for our web interface service.

Step 5: Redis Service for Caching

Checkout the step5 branch and list files in it.

git checkout step5
tree

.
├── README.md
├── api
│   ├── Dockerfile
│   ├── linkextractor.py
│   ├── main.py
│   └── requirements.txt
├── docker-compose.yml
└── www
    ├── Dockerfile
    └── index.php

2 directories, 8 files

Some noticeable changes from the previous step are as following:

Another Dockerfile is added in the ./www folder for the PHP web application to build a self-contained image and avoid live file mounting
A Redis container is added for caching using the official Redis Docker image
The API service talks to the Redis service to avoid downloading and parsing pages that were already scraped before
A REDIS_URL environment variable is added to the API service to allow it to connect to the Redis cache

Let’s first inspect the newly added Dockerfile under the ./www folder:

cat www/Dockerfile

FROM       php:7-apache
LABEL      maintainer="Sawood Alam <@ibnesayeed>"

ENV        API_ENDPOINT="http://localhost:5000/api/"

COPY       . /var/www/html/

This is a rather simple Dockerfile that uses the official php:7-apache image as the base and copies all the files from the ./www folder into the /var/www/html/ folder of the image. This is exactly what was happening in the previous step, but that was bind mounted using a volume, while here we are making the code part of the self-contained image. We have also added the API_ENDPOINT environment variable here with a default value, which implicitly suggests that this is an important information that needs to be present in order for the service to function properly (and should be customized at run time with an appropriate value).

Next, we will look at the API server’s api/main.py file where we are utilizing the Redis cache:

cat api/main.py

The file has many lines, but the important bits are as illustrated below:

redis_conn = redis.from_url(os.getenv("REDIS_URL", "redis://localhost:6379"))
# ...
    jsonlinks = redis.get(url)
    if not jsonlinks:
        links = extract_links(url)
        jsonlinks = json.dumps(links, indent=2)
        redis.set(url, jsonlinks)

This time the API service needs to know how to connect to the Redis instance as it is going to use it for caching. This information can be made available at run time using the REDIS_URL environment variable. A corresponding ENV entry is also added in the Dockerfile of the API service with a default value.

A redis client instance is created using the hostname redis (same as the name of the service as we will see later) and the default Redis port 6379. We are first trying to see if a cache is present in the Redis store for a given URL, if not then we use the extract_links function as before and populate the cache for future attempts.

Now, let’s look into the updated docker-compose.yml file:

cat docker-compose.yml

version: '3'

services:
  api:
    image: linkextractor-api:step5-python
    build: ./api
    ports:
      - "5000:5000"
    environment:
      - REDIS_URL=redis://redis:6379
  web:
    image: linkextractor-web:step5-php
    build: ./www
    ports:
      - "80:80"
    environment:
      - API_ENDPOINT=http://api:5000/api/
  redis:
    image: redis

The api service configuration largely remains the same as before, except the updated image tag and added environment variable REDIS_URL that points to the Redis service. For the web service, we are using the custom linkextractor-web:step5-php image that will be built using the newly added Dockerfile in the ./www folder. We are no longer mounting the ./www folder using the volumes config. Finally, a new service named redis is added that will use the official image from DockerHub and needs no specific configurations for now. This service is accessible to the Python API using its service name, the same way the API service is accessible to the PHP front-end service.

Let’s boot these services up:

docker-compose up -d --build

... [OUTPUT REDACTED] ...

Creating linkextractor_web_1   ... done
Creating linkextractor_api_1   ... done
Creating linkextractor_redis_1 ... done

Now, that all three services are up, access the web interface by clicking the Link Extractor. There should be no visual difference from the previous step. However, if you extract links from a page with a lot of links, the first time it should take longer, but the successive attempts to the same page should return the response fairly quickly. To check whether or not the Redis service is being utilized, we can use docker-compose exec followed by the redis service name and the Redis CLI’s monitor command:

docker-compose exec redis redis-cli monitor

Now, try to extract links from some web pages using the web interface and see the difference in Redis log entries for pages that are scraped the first time and those that are repeated. Before continuing further with the tutorial, stop the interactive monitor stream as a result of the above redis-cli command by pressing Ctrl + C keys while the interactive terminal is in focus.

Now that we are not mounting the /www folder inside the container, local changes should not reflect in the running service:

sed -i 's/Link Extractor/Super Link Extractor/g' www/index.php

Verify that the changes made locally do not reflect in the running service by reloading the web interface and then revert changes:

git reset --hard

Now, shut these services down and get ready for the next step:

docker-compose down

Stopping linkextractor_web_1   ... done
Stopping linkextractor_redis_1 ... done
Stopping linkextractor_api_1   ... done
Removing linkextractor_web_1   ... done
Removing linkextractor_redis_1 ... done
Removing linkextractor_api_1   ... done
Removing network linkextractor_default

We have successfully orchestrated three microservices to compose our Link Extractor application. We now have an application stack that represents the architecture illustrated in the figure shown in the introduction of this tutorial. In the next step we will explore how easy it is to swap components from an application with the microservice architecture.

Step 6: Swap Python API Service with Ruby

Checkout the step6 branch and list files in it.

git checkout step6
tree

.
├── README.md
├── api
│   ├── Dockerfile
│   ├── Gemfile
│   └── linkextractor.rb
├── docker-compose.yml
├── logs
└── www
    ├── Dockerfile
    └── index.php

3 directories, 7 files

Some significant changes from the previous step include:

The API service written in Python is replaced with a similar Ruby implementation
The API_ENDPOINT environment variable is updated to point to the new Ruby API service
The link extraction cache event (HIT/MISS) is logged and is persisted using volumes

Notice that the ./api folder does not contain any Python scripts, instead, it now has a Ruby file and a Gemfile to manage dependencies.

Let’s have a quick walk through the changed files:

cat api/linkextractor.rb

#!/usr/bin/env ruby
# encoding: utf-8

require "sinatra"
require "open-uri"
require "uri"
require "nokogiri"
require "json"
require "redis"

set :protection, :except=>:path_traversal

redis = Redis.new(url: ENV["REDIS_URL"] || "redis://localhost:6379")

Dir.mkdir("logs") unless Dir.exist?("logs")

get "/" do
  "Usage: http://<hostname>[:<prt>]/api/<url>"
end

get "/api/*" do
  url = [params['splat'].first, request.query_string].reject(&:empty?).join("?")
  cache_status = "HIT"
  jsonlinks = redis.get(url)
  if jsonlinks.nil?
    cache_status = "MISS"
    jsonlinks = JSON.pretty_generate(extract_links(url))
    redis.set(url, jsonlinks)
  end

  cache_log = File.open("logs/extraction.log", "a")
  cache_log.puts "#{Time.now.to_i}\t#{cache_status}\t#{url}"
  cache_log.close

  status 200
  headers "content-type" => "application/json"
  body jsonlinks
end

def extract_links(url)
  links = []
  doc = Nokogiri::HTML(open(url))
  doc.css("a").each do |link|
    text = link.text.strip.split.join(" ")
    begin
      links.push({
        text: text.empty? ? "[IMG]" : text,
        href: URI.join(url, link["href"])
      })
    rescue
    end
  end
  links
end

This Ruby file is almost equivalent to what we had in Python before, except, in addition to that it also logs the link extraction requests and corresponding cache events. In a microservice architecture application swapping components with an equivalent one is easy as long as the expectations of consumers of the component are maintained.

cat api/Dockerfile

FROM       ruby:2.6
LABEL      maintainer="Sawood Alam <@ibnesayeed>"

ENV        LANG C.UTF-8
ENV        REDIS_URL="redis://localhost:6379"

WORKDIR    /app
COPY       Gemfile /app/
RUN        bundle install

COPY       linkextractor.rb /app/
RUN        chmod a+x linkextractor.rb

CMD        ["./linkextractor.rb", "-o", "0.0.0.0"]

Above Dockerfile is written for the Ruby script and it is pretty much self-explanatory.

cat docker-compose.yml

version: '3'

services:
  api:
    image: linkextractor-api:step6-ruby
    build: ./api
    ports:
      - "4567:4567"
    environment:
      - REDIS_URL=redis://redis:6379
    volumes:
      - ./logs:/app/logs
  web:
    image: linkextractor-web:step6-php
    build: ./www
    ports:
      - "80:80"
    environment:
      - API_ENDPOINT=http://api:4567/api/
  redis:
    image: redis

The docker-compose.yml file has a few minor changes in it. The api service image is now named linkextractor-api:step6-ruby, the port mapping is changed from 5000 to 4567 (which is the default port for Sinatra server), and the API_ENDPOINT environment variable in the web service is updated accordingly so that the PHP code can talk to it.

With these in place, let’s boot our service stack:

docker-compose up -d --build

... [OUTPUT REDACTED] ...

Successfully built b713eef49f55
Successfully tagged linkextractor-api:step6-ruby
Creating linkextractor_web_1   ... done
Creating linkextractor_api_1   ... done
Creating linkextractor_redis_1 ... done

We should now be able to access the API (using the updated port number):

curl -i http://localhost:4567/api/http://example.com/

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 96
X-Content-Type-Options: nosniff
Server: WEBrick/1.4.2 (Ruby/2.5.1/2018-03-29)
Date: Mon, 24 Sep 2018 01:41:35 GMT
Connection: Keep-Alive

[
  {
    "text": "More information...",
    "href": "http://www.iana.org/domains/example"
  }
]

Now, open the web interface by clicking the Link Extractor and extract links of a few URLs. Also, try to repeat these attempts for some URLs. If everything is alright, the web application should behave as before without noticing any changes in the API service (which is completely replaced).

We can use the tail command with the -f or --follow option to follow the log output live.

tail -f logs/extraction.log

Try a few more URLs in the web interface. You should see the new log entries appear in the terminal.

To stop following the log, press Ctrl + C keys while the interactive terminal is in focus.

We can shut the stack down now:

docker-compose down

Since we have persisted logs, they should still be available after the services are gone:

cat logs/extraction.log

1537753295      MISS    http://example.com/
1537753600      HIT     http://example.com/
1537753635      MISS    https://training.play-with-docker.com/

This illustrates that the caching is functional as the second attempt to the http://example.com/ resulted in a cache HIT.

In this step we explored the possibility of swapping components of an application with microservice architecture with their equivalents without impacting rest of the parts of the stack. We have also explored data persistence using bind mount volumes that persists even after the containers writing to the volume are gone.

So far, we have used docker-compose utility to orchestrate the application stack, which is good for development environment, but for production environment we use docker stack deploy command to run the application in a Docker Swarm Cluster. It is left for you as an assignment to deploy this application in a Docker Swarm Cluster.

Conclusions

We started this tutorial with a simple Python script that scrapes links from a given web page URL. We demonstrated various difficulties in running the script. We then illustrated how easy to run and portable the script becomes onces it is containerized. In the later steps we gradually evolved the script into a multi-service application stack. In the process we explored various concepts of microservice architecture and how Docker tools can be helpful in orchestrating a multi-service stack. Finally, we demonstrated the ease of microservice component swapping and data persistence.

The next step would be to learn how to deploy such service stacks in a Docker Swarm Cluster.

As an aside, here are some introductory Docker slides.

Introducing Docker - Application Containerization & Service Orchestration by Sawood Alam

Node.js with SQL Server on Docker

Tue, 28 Nov 2017 00:00:00 +0000

This lab walks through the evolution of a simple Node.js bulletin board application, running on Docker. You’ll start with a simple app that uses hard-coded data, then add SQL Server for persistent storage, and a proxy to improve web performance.

You’ll learn about packaging applications in Docker images, running distributed applications across multiple containers, and adding instrumentation to your containers so you can see the health of your application. You’ll use the Docker command line, Docker Compose and Docker swarm for running the app.

Difficulty: Beginner (assumes no familiarity with Docker)

Time: Approximately 60 minutes

Tasks:

Task 0: Prerequisites

Task 1: Run v1 of the app in a container

Task 2: Add a SQL Server database container for storage

Task 3: Switch to high availability in swarm mode

Task 4: Add a reverse proxy to improve performance

Task 5: Add monitoring and an application dashboard

Task 0: Prerequisites

You will need:

a copy of the application source code
a Docker ID

Clone the source code from GitHub

Use the following command to clone the application source code from GitHub (you can click the command or manually type it). This will make a copy of the lab’s repo in a new sub-directory called node-bulletin-board.

git clone https://github.com/dockersamples/node-bulletin-board.git

And browse to the source code folder:

cd node-bulletin-board

Save your Docker ID

You need a Docker ID to push your images to Docker Hub. If you don’t have one, create a free Docker ID at Docker Hub.

Now save your Docker ID in an environment variable - you need to type this command manually with your own Docker ID:

export dockerId='your-docker-id'

Be sure to use your own Docker ID. Mine is sixeyed, so the command I run is export dockerId='sixeyed'.

Check your Docker ID gets displayed when you read the variable:

echo $dockerId

Task 1: Run v1 of the app in a container

The first version of the application uses a single container, running the Node.js application, and the data is only stored on the client’s browser.

Switch to the v1 source code branch:

git checkout v1

Now build the Docker image, which uses this Dockerfile to package the source code on top of the official Node.js image:

docker image build --tag $dockerId/bb-app:v1 --file bulletin-board-app/Dockerfile ./bulletin-board-app

When that completes you will have version 1 of the app in an image stored locally. Run a container from that image to start the app:

docker container run --detach --publish 8080:8080 $dockerId/bb-app:v1

Docker will start a container from the application image, which runs npm start to start the app. You can browse to the application on port 8080:

Click here for v1 of the app

You’ll see the bulletin board application, and you can add and remove events:

If you make some changes and refresh the browser, you’ll see your changes get lost. That’s because the events are only stored in memory on the client.

In the next step you’ll fix that.

Task 2: Add a SQL Server database container for storage

Storing data in client memory is only good for proof-of-concept applications. In this step you’ll build and deploy version 2 of the app, which uses a SQL Server database to store the events.

First switch to the v2 code branch:

git checkout v2

Now clear up all the containers from the previous part:

docker container rm --force $(docker container ls --quiet)

You’ll use Docker Compose to build and run the application. The compose file specifies the database and application containers to run, and how to configure them.

The compose file also contains the path to the application Dockerfile and to the database Dockerfile, so you can build the database and application images with one command:

docker-compose build

When that completes, you’ll have two Docker images:

<your-docker-id>/bulletin-board-db:v2 - which is based on Microsoft’s SQL Server image and packages the database schema for the bulletin board
<your-docker-id>/bulletin-board-app:v2 - which is the new version of the Node.js application, using SQL Server to store events

You can start the whole app with Docker Compose:

docker-compose up -d

You’ll see compose starts the database first, because it’s specified as a dependency for the application container. Then it starts the app container.

If you list all containers, you’ll see there are two instances of the app container. One container started before the database was ready, so it failed - and then Docker Compose started a replacement container, which did connect to the database.

docker container ls --all

Click here for v2 of the app

You’ll see it’s the same user interface, but now you can add and delete events and when you refresh the page they’re still there. The data is persisted in SQL Server.

The SQL Server database is not publicly available. In the docker-compose.yml file, the web container, the port 8080 is published so you can send traffic in, but no ports are published for the database. It’s only available to other containers and to Docker.

Task 3: Switch to high availability in swarm mode

Swarm mode lets you join several Docker servers together and treat them as a single unit. You deploy your app as services to the swarm, and Docker runs containers across all the servers.

You can run multiple instances of a container to deal with scale, and if a server goes down and you lose containers, Docker starts replacement containers on other servers.

First clear down all the containers from part 2:

docker container rm --force $(docker container ls --quiet)

Now switch to swarm mode:

docker swarm init --advertise-addr $(hostname -i)

This creates a single-node swarm. The output of the command shows you how to join other Docker servers to the swarm - all you need are more servers running Docker in the same network. You can scale Docker swarm up to hundreds of nodes.

The normal docker commands still work in swarm mode. Switch to the v3 source code branch:

git checkout v3

And build the application with Docker Compose:

docker-compose build

Version 3 has the same source code, but the Dockerfile for v3 of the the web app includes a HEALTHCHECK instruction. That tells Docker how to test if the application is healthy, and unhealthy containers are stopped and replaced with new ones.

You use the same Docker Compose file format to deploy in swarm mode, and there are some additional options available. Deploy version 3 of the app using the docker-stack.yml file:

docker stack deploy -c docker-stack.yml bb

A stack is a way to group many services together, so you can manage them as one unit. You can see the services in the stack, which tells you if the application is up:

docker stack services bb

Click here for v3 of the app

You’ll see the application behaviour is exactly the same - containers are running from the same Docker images, but now they’re being scheduled by Docker swarm.

Docker swarm also supports rolling updates for applications running as stacks. In the next part you’ll add more functionality to the app, by running a web proxy.

Task 4: Add a reverse proxy to improve performance

Node.js is a good server platform, but it’s easy to improve performance by putting a reverse proxy in front of the Node.js application. The proxy is the public entrypoint to the app, and it handles requests from users.

Nginx is a popular open source web server which you can easily configure as a reverse proxy. The Nginx configuration in this part makes use of browser and server caching, which reduces the load on the web application and improves performance.

Switch to the v4 code branch:

git checkout v4

And use Docker Compose to build the application:

docker-compose build

Now you have v4 Docker images for all the application parts, you can upgrade the running stack using the new docker-stack.yml file:

docker stack deploy -c docker-stack.yml bb

Version 4 adds a proxy server to the stack which publishes port 80, so now you can browse to the app on the standard HTTP port:

Click here for v4 of the app

The web application looks the same, but behind the scenes all the hard work is being done by the Nginx proxy. You can open developer tools on your browser and inspect the network responses - Nginx has added browser caching hints, and it’s also using a local cache to reduce traffic to the Node.js app.

There are also several instances of the proxy container running - Docker swarm load-balances incoming requests between those containers. If you had multiple servers in the swarm, you would be able to scale up to handle your incoming workload.

In the final part you’ll add monitoring to the application, so you can see what the Node.js container is doing.

Task 5: Add monitoring and an application dashboard

Docker swarm makes it super easy to scale containers, but before you go to production witrh a Dockerized application, you’ll want monitoring in place so you can see what all those containers are doing.

Two open-source technologies are very popular in the Docker ecosystem for monitoring containers. Prometheus is an instrumentation server that collects and stores metrics from your containers, and Grafana is an analytics UI that plugs into Prometheus to show dashboards.

In this part you’ll add Prometheus and Grafana to your application.

First switch to the v5 code branch:

git checkout v5

Now build the application, which will build images from the Prometheus Dockerfile and the Grafana Dockerfile:

docker-compose build

You have v5 images for all the application components now. Upgrade the stack to the v5 docker-stack.yml file:

docker stack deploy -c docker-stack.yml bb

Click here for v5 of the app

The UX is the same, but now the Prometheus container is scraping metrics from the Node.js container, every 5 seconds.

To see the application metrics in Grafana, you need to configure the dashboard:

Click here for Grafana

Add a new data source with the following details:

Name: prometheus
Type: Prometheus
URL: http://bb-metrics:9090

From the Grafana icon, click Dashboards… Import and load the JSON dashboard file from v5 /dashboard.json. Select the Prometheus data store.

You’ll now see the application dashboard - send some load into the app by refreshing the browser, and the graphs will be populated:

Cleanup

You can easily remove the whole application by removing the stack:

docker stack rm bb

And you can leave swarm mode to return to a single-server Docker host:

docker swarm leave --force

Thanks for completing the Node.js and SQL Server lab! You’ve learned how to build and run applications with Docker and Docker Compose, how to achieve high availability with Docker Swarm and how to get your application production ready by adding a proxy and a metrics dashboard.

The Play with Docker Training site is always on, and there are plenty more labs you can try at home.

Docker for IT Pros and System Administrators Stage 1

Tue, 19 Sep 2017 00:00:00 +0000

Welcome to the first stage of your Docker learning journey. This stage will

Get you familiar with the core concepts of Docker
Help you understand why other IT leaders are rapidly adopting Docker
Help you see how Docker can help your organization

For an introduction to Docker specifically geared towards for sys admins and IT Pros, check out Mike Coleman’s talk Docker?!? But I’m a SYSADMIN!

Hands-on Learning

Give Docker a try with these three beginning tutorials:

Your First Linux Containers: In this lab you will explore the basics of running containers: pulling images from a registry, running an containerized application, and container instances and isolation.
Customizing Docker Images: Move on to building your own custom Docker images and explore the Docker concept of image layers and the Dockerfile.
Deploy and Managing Multiple Containers: Real apps consist of multiple components. In this lab you will begin to explore running multiple services as a single stack with Docker Swarm.

Doing More With Docker Images

Tue, 19 Sep 2017 00:00:00 +0000

In the previous exercise you pulled down images from Docker Store to run in your containers. Then you ran multiple instances and noted how each instance was isolated from the others. We hinted that this is used in many production IT environments every day but obviously we need a few more tools in our belt to get to the point where Docker can become a true time & money saver.

First thing you may want to do is figure out how to create our own images. While there are over 700K images on Docker Store it is almost certain that none of them are exactly what you run in your data center today. Even something as common as a Windows OS image would get its own tweaks before you actually run it in production. In the first lab, we created a file called “hello.txt” in one of our container instances. If that instance of our Alpine container was something we wanted to re-use in future containers and share with others, we would need to create a custom image that everyone could use.

We will start with the simplest form of image creation, in which we simply commit one of our container instances as an image. Then we will explore a much more powerful and useful method for creating images: the Dockerfile.

We will then see how to get the details of an image through the inspection and explore the filesystem to have a better understanding of what happens under the hood.

Image creation from a container

Let’s start by running an interactive shell in a ubuntu container:

docker container run -ti ubuntu bash

As you know from earlier labs, you just grabbed the image called “ubuntu” from Docker Store and are now running the bash shell inside that container.¹

To customize things a little bit we will install a package called figlet in this container. Your container should still be running so type the following commands at your ubuntu container command line:

apt-get update
apt-get install -y figlet
figlet "hello docker"

You should see the words “hello docker” printed out in large ascii characters on the screen. Go ahead and exit from this container

exit

Now let us pretend this new figlet application is quite useful and you want to share it with the rest of your team. You could tell them to do exactly what you did above and install figlet in to their own container, which is simple enough in this example. But if this was a real world application where you had just installed several packages and run through a number of configuration steps the process could get cumbersome and become quite error prone. Instead, it would be easier to create an image you can share with your team.

To start, we need to get the ID of this container using the ls command (do not forget the -a option as the non running container are not returned by the ls command).

docker container ls -a

Before we create our own image, we might want to inspect all the changes we made. Try typing the command docker container diff <container ID> for the container you just created. You should see a list of all the files that were added to or changed in the container when you installed figlet. Docker keeps track of all of this information for us. This is part of the layer concept we will explore in a few minutes.

Now, to create an image we need to “commit” this container. Commit creates an image locally on the system running the Docker engine. Run the following command, using the container ID you retrieved, in order to commit the container and create an image out of it.

docker container commit CONTAINER_ID

That’s it - you have created your first image! Once it has been commited, we can see the newly created image in the list of available images.

docker image ls

You should see something like this:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
<none>              <none>              a104f9ae9c37        46 seconds ago      160MB
ubuntu              latest              14f60031763d        4 days ago          120MB

Note that the image we pulled down in the first step (ubuntu) is listed here along with our own custom image. Except our custom image has no information in the REPOSITORY or TAG columns, which would make it tough to identify exactly what was in this container if we wanted to share amongst multiple team members.

Adding this information to an image is known as tagging an image. From the previous command, get the ID of the newly created image and tag it so it’s named ourfiglet:

docker image tag <IMAGE_ID> ourfiglet
docker image ls

Now we have the more friendly name “ourfiglet” that we can use to identify our image.

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ourfiglet           latest              a104f9ae9c37        5 minutes ago       160MB
ubuntu              latest              14f60031763d        4 days ago          120MB

Here is a graphical view of what we just completed:

Now we will run a container based on the newly created ourfiglet image:

docker container run ourfiglet figlet hello

As the figlet package is present in our ourfiglet image, the command returns the following output:

 _          _ _
| |__   ___| | | ___
| '_ \ / _ \ | |/ _ \
| | | |  __/ | | (_) |
|_| |_|\___|_|_|\___/

This example shows that we can create a container, add all the libraries and binaries in it and then commit it in order to create an image. We can then use that image just as we would for images pulled down from the Docker Store. We still have a slight issue in that our image is only stored locally. To share the image we would want to push the image to a registry somewhere. This is beyond the scope of this lab (and you should not enter any personal login information in these labs) but you can get a free Docker ID, run these labs, and push to the Docker Community Hub from your own system using Docker for Windows or Docker for Mac if you want to try this out.

As mentioned above, this approach of manually installing software in a container and then committing it to a custom image is just one way to create an image. It works fine and is quite common. However, there is a more powerful way to create images. In the following exercise we will see how images are created using a Dockerfile, which is a text file that contains all the instructions to build an image.

Image creation using a Dockerfile

Instead of creating a static binary image, we can use a file called a Dockerfile to create an image. The final result is essentially the same, but with a Dockerfile we are supplying the instructions for building the image, rather than just the raw binary files. This is useful because it becomes much easier to manage changes, especially as your images get bigger and more complex.

For example, if a new version of figlet is released we would either have to re-create our image from scratch, or run our image and upgrade the installed version of figlet. In contrast, a Dockerfile would include the apt-get commands we used to install figlet so that we - or anybody using the Dockerfile - could simply recompose the image using those instructions.

It is kind of like the old adage:

Give a sysadmin an image and their app will be up-to-date for a day, give a sysadmin a Dockerfile and their app will always be up-to-date.

Ok, maybe that’s a bit of a stretch but Dockerfiles are powerful because they allow us to manage how an image is built, rather than just managing binaries. In practice, Dockerfiles can be managed the same way you might manage source code: they are simply text files so almost any version control system can be used to manage Dockerfiles over time.

We will use a simple example in this section and build a “hello world” application in Node.js. Do not be concerned if you are not familiar with Node.js: Docker (and this exercise) does not require you to know all these details.

We will start by creating a file in which we retrieve the hostname and display it. NOTE: You should be at the Docker host’s command line ($). If you see a command line that looks similar to root@abcd1234567:/# then you are probably still inside your ubuntu container from the previous exercise. Type exit to return to the host command line.

Type the following content into a file named index.js. You can use vi, vim or several other Linux editors in this exercise. If you need assistance with the Linux editor commands to do this follow this footnote².

var os = require("os");
var hostname = os.hostname();
console.log("hello from " + hostname);

The file we just created is the javascript code for our server. As you can probably guess, Node.js will simply print out a “hello” message. We will Docker-ize this application by creating a Dockerfile. We will use alpine as the base OS image, add a Node.js runtime and then copy our source code in to the container. We will also specify the default command to be run upon container creation.

Create a file named Dockerfile and copy the following content into it. Again, help creating this file with Linux editors is here ³.

FROM alpine
RUN apk update && apk add nodejs
COPY . /app
WORKDIR /app
CMD ["node","index.js"]

Let’s build our first image out of this Dockerfile and name it hello:v0.1:

docker image build -t hello:v0.1 .

This is what you just completed:

We then start a container to check that our applications runs correctly:

docker container run hello:v0.1

You should then have an output similar to the following one (the ID will be different though).

hello from 92d79b6de29f

What just happened? We created two files: our application code (index.js) is a simple bit of javascript code that prints out a message. And the Dockerfile is the instructions for Docker engine to create our custom container. This Dockerfile does the following:

Specifies a base image to pull FROM - the alpine image we used in earlier labs.
Then it RUNs two commands (apk update and apk add) inside that container which installs the Node.js server.
Then we told it to COPY files from our working directory in to the container. The only file we have right now is our index.js.
Next we specify the WORKDIR - the directory the container should use when it starts up
And finally, we gave our container a command (CMD) to run when the container starts.

Recall that in previous labs we put commands like echo "hello world" on the command line. With a Dockerfile we can specify precise commands to run for everyone who uses this container. Other users do not have to build the container themselves once you push your container up to a repository (which we will cover later) or even know what commands are used. The Dockerfile allows us to specify how to build a container so that we can repeat those steps precisely everytime and we can specify what the container should do when it runs. There are actually multiple methods for specifying the commands and accepting parameters a container will use, but for now it is enough to know that you have the tools to create some pretty powerful containers.

Image layers

There is something else interesting about the images we build with Docker. When running they appear to be a single OS and application. But the images themselves are actually built in layers. If you scroll back and look at the output from your docker image build command you will notice that there were 5 steps and each step had several tasks. You should see several “fetch” and “pull” tasks where Docker is grabbing various bits from Docker Store or other places. These bits were used to create one or more container layers. Layers are an important concept. To explore this, we will go through another set of exercises.

First, check out the image you created earlier by using the history command (remember to use the docker image ls command from earlier exercises to find your image IDs):

docker image history <image ID>

What you see is the list of intermediate container images that were built along the way to creating your final Node.js app image. Some of these intermediate images will become layers in your final container image. In the history command output, the original Alpine layers are at the bottom of the list and then each customization we added in our Dockerfile is its own step in the output. This is a powerful concept because it means that if we need to make a change to our application, it may only affect a single layer! To see this, we will modify our app a bit and create a new image.

Type the following in to your console window:

echo "console.log(\"this is v0.2\");" >> index.js

This will add a new line to the bottom of your index.js file from earlier so your application will output one additional line of text. Now we will build a new image using our updated code. We will also tag our new image to mark it as a new version so that anybody consuming our images later can identify the correct version to use:

docker image build -t hello:v0.2 .

You should see output similar to this:

Sending build context to Docker daemon  86.15MB
Step 1/5 : FROM alpine
 ---> 7328f6f8b418
Step 2/5 : RUN apk update && apk add nodejs
 ---> Using cache
 ---> 2707762fca63
Step 3/5 : COPY . /app
 ---> 07b2e2127db4
Removing intermediate container 84eb9c31320d
Step 4/5 : WORKDIR /app
 ---> 6630eb76312c
Removing intermediate container ee6c9e7a5337
Step 5/5 : CMD node index.js
 ---> Running in e079fb6000a3
 ---> e536b9dadd2f
Removing intermediate container e079fb6000a3
Successfully built e536b9dadd2f
Successfully tagged hello:v0.2

Notice something interesting in the build steps this time. In the output it goes through the same five steps, but notice that in some steps it says Using cache.

Docker recognized that we had already built some of these layers in our earlier image builds and since nothing had changed in those layers it could simply use a cached version of the layer, rather than pulling down code a second time and running those steps. Docker’s layer management is very useful to IT teams when patching systems, updating or upgrading to the latest version of code, or making configuration changes to applications. Docker is intelligent enough to build the container in the most efficient way possible, as opposed to repeatedly building an image from the ground up each and every time.

Image Inspection

Now let us reverse our thinking a bit. What if we get a container from Docker Store or another registry and want to know a bit about what is inside the container we are consuming? Docker has an inspect command for images and it returns details on the container image, the commands it runs, the OS and more.

The alpine image should already be present locally from the exercises above (use docker image ls to confirm), if it’s not, run the following command to pull it down:

docker image pull alpine

Once we are sure it is there let’s inspect it.

docker image inspect alpine

There is a lot of information in there:

the layers the image is composed of
the driver used to store the layers
the architecture / OS it has been created for
metadata of the image
…

We will not go into all the details here but we can use some filters to just inspect particular details about the image. You may have noticed that the image information is in JSON format. We can take advantage of that to use the inspect command with some filtering info to just get specific data from the image.

Let’s get the list of layers:

docker image inspect --format "{{ json .RootFS.Layers }}" alpine

Alpine is just a small base OS image so there’s just one layer:

["sha256:60ab55d3379d47c1ba6b6225d59d10e1f52096ee9d5c816e42c635ccc57a5a2b"]

Now let’s look at our custom Hello image. You will need the image ID (use docker image ls if you need to look it up):

docker image inspect --format "{{ json .RootFS.Layers }}" <image ID>

Our Hello image is a bit more interesting (your sha256 hashes will vary):

["sha256:5bef08742407efd622d243692b79ba0055383bbce12900324f75e56f589aedb0","sha256:5ac283aaea742f843c869d28bbeaf5000c08685b5f7ba01431094a207b8a1df9","sha256:2ecb254be0603a2c76880be45a5c2b028f6208714aec770d49c9eff4cbc3cf25"]

We have three layers in our application. Recall that we had the base Alpine image (the FROM command in our Dockerfile), then we had a RUN command to install some packages, then we had a COPY command to add in our javascript code. Those are our layers! If you look closely, you can even see that both alpine and hello are using the same base layer, which we know because they have the same sha256 hash.

The tools and commands we explored in this lab are just the beginning. Docker Enterprise Edition includes private Trusted Registries with Security Scanning and Image Signing capabilities so you can further inspect and authenticate your images. In addition, there are policy controls to specify which users have access to various images, who can push and pull images, and much more.

Another important note about layers: each layer is immutable. As an image is created and successive layers are added, the new layers keep track of the changes from the layer below. When you start the container running there is an additional layer used to keep track of any changes that occur as the application runs (like the “hello.txt” file we created in the earlier exercises). This design principle is important for both security and data management. If someone mistakenly or maliciously changes something in a running container, you can very easily revert back to its original state because the base layers cannot be changed. Or you can simply start a new container instance which will start fresh from your pristine image. And applications that create and store data (databases, for example) can store their data in a special kind of Docker object called a volume, so that data can persist and be shared with other containers. We will explore volumes in a later lab.

Up next, we will look at more sophisticated applications that run across several containers and use Docker Compose and Docker Swarm to define our architecture and manage it.

Terminology

Layers - A Docker image is built up from a series of layers. Each layer represents an instruction in the image’s Dockerfile. Each layer except the last one is read-only.
Dockerfile - A text file that contains all the commands, in order, needed to build a given image. The Dockerfile reference page lists the various commands and format details for Dockerfiles.
Volumes - A special Docker container layer that allows data to persist and be shared separately from the container itself. Think of volumes as a way to abstract and manage your persistent data separately from the application itself.

Footnotes

A note on images and the public Docker Store (AKA Docker Hub): Docker registries are subdivided in to many repositories. This is the same for both our public registries like Docker Store / Docker Hub, as well as Docker Trusted Registries that you might run in your own environment. Image names must be unique and are specified in the format <repository>/<image>:<tag>. In our exercises, we pulled images called “ubuntu” and “alpine”. Since there is no repository specified we pulled from a default public repository called “library” which is maintained by us at Docker. And since we did not specify a tag, the default is to look for a tag named “latest” and use that. The tags generally specify versions (although this is not a requirement). ↩
Type vi index.js then once the editor loads hit the i key. You can now type each of the commands as shown in the example. When you are finished hit the <esc> key then type :wq and that will save the file and take you back to the command prompt. You can type ls at the command prompt to ensure your index.js file is there or type cat index.js to make sure all the code is in the file. If you make a mistake in the editor and you have a hard time navigating the editor it might be easier to start fresh: simply type <esc> and then :wq if you are in the editor and then when you are back to the command line type rm index.js to delete the file and then start again. ↩
Type vi Dockerfile then once the editor loads hit the i key. Type in each line of the Dockerfile code as shown in the example - capitalization is important! - then hit the <esc> key followed by :wq. To verify your Dockerfile exists and is correct type cat Dockerfile. If you make a mistake in the editor and you have a hard time navigating the editor it might be easier to start fresh: simply type <esc> and then :wq if you are in the editor and then when you are back to the command line type rm Dockerfile and then start again. ↩

First Alpine Linux Containers

Tue, 19 Sep 2017 00:00:00 +0000

In this lab you will run a popular, free, lightweight container and explore the basics of how containers work, how the Docker Engine executes and isolates containers from each other. If you already have experience running containers and basic Docker commands you can probably skip this intro exercise.

Concepts in this exercise:

Docker engine
Containers & images
Image registries and Docker Hub
Container isolation

Tips:

Code snippets are shown in one of three ways throughout this environment:

Code that looks like this is sample code snippets that is usually part of an explanation.
Code that appears in box like the one below can be clicked on and it will automatically be typed in to the appropriate terminal window:
```
uname -a
```
Code appearing in windows like the one below is code that you should type in yourself. Usually there will be a unique ID or other bit your need to enter which we cannot supply. Items appearing in <> are the pieces you should substitute based on the instructions.
```
docker container start <container ID>
```

1.0 Running your first container

It’s time to get your hands dirty! As with all things technical, a “hello world” app is good place to start. Type or click the code below to run your first Docker container:

docker container run hello-world

That’s it: your first container. The hello-world container output tells you a bit about what just happened. Essentially, the Docker engine running in your terminal tried to find an image named hello-world. Since you just got started there are no images stored locally (Unable to find image...) so Docker engine goes to its default Docker registry, which is Docker Hub, to look for an image named “hello-world”. It finds the image there, pulls it down, and then runs it in a container. And hello-world’s only function is to output the text you see in your terminal, after which the container exits.

If you are familiar with VMs, you may be thinking this is pretty much just like running a virtual machine, except with a central repository of VM images. And in this simple example, that is basically true. But as you go through these exercises you will start to see important ways that Docker and containers differ from VMs. For now, the simple explanation is this:

The VM is a hardware abstraction: it takes physical CPUs and RAM from a host, and divides and shares it across several smaller virtual machines. There is an OS and application running inside the VM, but the virtualization software usually has no real knowledge of that.
A container is an application abstraction: the focus is really on the OS and the application, and not so much the hardware abstraction. Many customers actually use both VMs and containers today in their environments and, in fact, may run containers inside of VMs.

1.1 Docker Images

In this rest of this lab, you are going to run an Alpine Linux container. Alpine is a lightweight Linux distribution so it is quick to pull down and run, making it a popular starting point for many other images.

To get started, let’s run the following in our terminal:

docker image pull alpine

The pull command fetches the alpine image from the Docker registry and saves it in our system. In this case the registry is Docker Hub. You can change the registry, but that’s a different lab.

You can use the docker image command to see a list of all images on your system.

docker image ls

REPOSITORY              TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
alpine                 latest              c51f86c28340        4 weeks ago         1.109 MB
hello-world             latest              690ed74de00f        5 months ago        960 B

1.1 Docker Container Run

Great! Let’s now run a Docker container based on this image. To do that you are going to use the docker container run command.

docker container run alpine ls -l

total 48
drwxr-xr-x    2 root     root          4096 Mar  2 16:20 bin
drwxr-xr-x    5 root     root           360 Mar 18 09:47 dev
drwxr-xr-x   13 root     root          4096 Mar 18 09:47 etc
drwxr-xr-x    2 root     root          4096 Mar  2 16:20 home
drwxr-xr-x    5 root     root          4096 Mar  2 16:20 lib
......
......

While the output of the ls command may not be all that exciting, behind the scenes quite a few things just took place. When you call run, the Docker client finds the image (alpine in this case), creates the container and then runs a command in that container. When you run docker container run alpine, you provided a command (ls -l), so Docker executed this command inside the container for which you saw the directory listing. After the ls command finished, the container shut down.

The fact that the container exited after running our command is important, as you will start to see. Let’s try something more exciting. Type in the following:

docker container run alpine echo "hello from alpine"

And you should get the following output:

hello from alpine

In this case, the Docker client dutifully ran the echo command inside our alpine container and then exited. If you noticed, all of that happened pretty quickly and again our container exited. As you will see in a few more steps, the echo command ran in a separate container instance. Imagine booting up a virtual machine (VM), running a command and then killing it; it would take a minute or two just to boot the VM before running the command. A VM has to emulate a full hardware stack, boot an operating system, and then launch your app - it’s a virtualized hardware environment. Docker containers function at the application layer so they skip most of the steps VMs require and just run what is required for the app. Now you know why they say containers are fast!

Try another command.

docker container run alpine /bin/sh

Wait, nothing happened! Is that a bug? No! In fact, something did happen. You started a 3rd instance of the alpine container and it ran the command /bin/sh and then exited. You did not supply any additional commands to /bin/sh so it just launched the shell, exited the shell, and then stopped the container. What you might have expected was an interactive shell where you could type some commands. Docker has a facility for that by adding a flag to run the container in an interactive terminal. For this example, type the following:

 docker container run -it alpine /bin/sh

You are now inside the container running a Linux shell and you can try out a few commands like ls -l, uname -a and others. Note that Alpine is a small Linux OS so several commands might be missing. Exit out of the shell and container by typing the exit command.

Ok, we said that we had run each of our commands above in a separate container instance. We can see these instances using the docker container ls command. The docker container ls command by itself shows you all containers that are currently running:

docker container ls

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Since no containers are running, you see a blank line. Let’s try a more useful variant: docker container ls -a

docker container ls -a

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS               NAMES
36171a5da744        alpine              "/bin/sh"                5 minutes ago       Exited (0) 2 minutes ago                        fervent_newton
a6a9d46d0b2f        alpine             "echo 'hello from alp"    6 minutes ago       Exited (0) 6 minutes ago                        lonely_kilby
ff0a5c3750b9        alpine             "ls -l"                   8 minutes ago       Exited (0) 8 minutes ago                        elated_ramanujan
c317d0a9e3d2        hello-world         "/hello"                 34 seconds ago      Exited (0) 12 minutes ago                       stupefied_mcclintock

What you see now is a list of all containers that you ran. Notice that the STATUS column shows that these containers exited some time ago.

Here is the same output of the docker container ls -a command, shown diagrammatically (note that your container IDs and names will be different):

It makes sense to spend some time getting comfortable with the docker run commands. To find out more about run, use docker container run --help to see a list of all flags it supports. As you proceed further, we’ll see a few more variants of docker container run but feel free to experiment here before proceeding.

1.2 Container Isolation

In the steps above we ran several commands via container instances with the help of docker container run. The docker container ls -a command showed us that there were several containers listed. Why are there so many containers listed if they are all from the alpine image?

This is a critical security concept in the world of Docker containers! Even though each docker container run command used the same alpine image, each execution was a separate, isolated container. Each container has a separate filesystem and runs in a different namespace; by default a container has no way of interacting with other containers, even those from the same image. Let’s try another exercise to learn more about isolation.

docker container run -it alpine /bin/ash

The /bin/ash is another type of shell available in the alpine image. Once the container launches and you are at the container’s command prompt type the following commands:

 echo "hello world" > hello.txt

 ls

The first echo command creates a file called “hello.txt” with the words “hello world” inside it. The second command gives you a directory listing of the files and should show your newly created “hello.txt” file. Now type exit to leave this container.

To show how isolation works, run the following:

docker container run alpine ls

It is the same ls command we used inside the container’s interactive ash shell, but this time, did you notice that your “hello.txt” file is missing? That’s isolation! Your command ran in a new and separate instance, even though it is based on the same image. The 2nd instance has no way of interacting with the 1st instance because the Docker Engine keeps them separated and we have not setup any extra parameters that would enable these two instances to interact.

In every day work, Docker users take advantage of this feature not only for security, but to test the effects of making application changes. Isolation allows users to quickly create separate, isolated test copies of an application or service and have them run side-by-side without interfering with one another. In fact, there is a whole lifecycle where users take their changes and move them up to production using this basic concept and the built-in capabilities of Docker Enteprise. We will explore more of that in later exercises.

Right now, the obvious question is “how do I get back to the container that has my ‘hello.txt’ file?”

Once again run the

docker container ls -a

command again and you should see output similar to the following:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS               NAMES
36171a5da744        alpine              "ls"                     2 minutes ago       Exited (0) 2 minutes ago                        distracted_bhaskara
3030c9c91e12        alpine              "/bin/ash"               5 minutes ago       Exited (0) 2 minutes ago                        fervent_newton
a6a9d46d0b2f        alpine             "echo 'hello from alp"    6 minutes ago       Exited (0) 6 minutes ago                        lonely_kilby
ff0a5c3750b9        alpine             "ls -l"                   8 minutes ago       Exited (0) 8 minutes ago                        elated_ramanujan
c317d0a9e3d2        hello-world         "/hello"                 34 seconds ago      Exited (0) 12 minutes ago                       stupefied_mcclintock

Graphically this is what happened on our Docker Engine:

The container in which we created the “hello.txt” file is the same one where we used the /bin/ash shell, which we can see listed in the “COMMAND” column. The Container ID number from the first column uniquely identifies that particular container instance. In the sample output above the container ID is 3030c9c91e12. We can use a slightly different command to tell Docker to run this specific container instance. Try typing:

docker container start <container ID>

Pro tip: Instead of using the full container ID you can use just the first few characters, as long as they are enough to uniquely ID a container. So we could simply use “3030” to identify the container instance in the example above, since no other containers in this list start with these characters.

Now use the docker container ls command again to list the running containers.

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS               NAMES
3030c9c91e12        alpine              "/bin/ash"                2 minutes ago       Up 14 seconds                        distracted_bhaskara

Notice this time that our container instance is still running. We used the ash shell this time so the rather than simply exiting the way /bin/sh did earlier, ash waits for a command. We can send a command in to the container to run by using the exec command, as follows:

docker container exec <container ID> ls

This time we get a directory listing and it shows our “hello.txt” file because we used the container instance where we created that file.

Now you are starting to see some of the important concepts of containers. In the next exercise we will start to see how you can create your own Docker images and how to use a Dockerfile to standardize images such that you can create larger, more complex images in a simple, automated manner.

1.3 Terminology

In the last section, you saw a lot of Docker-specific jargon which might be confusing to some. So before you go further, let’s clarify some terminology that is used frequently in the Docker ecosystem.

Images - The file system and configuration of our application which are used to create containers. To find out more about a Docker image, run docker image inspect alpine. In the demo above, you used the docker image pull command to download the alpine image. When you executed the command docker container run hello-world, it also did a docker image pull behind the scenes to download the hello-world image.
Containers - Running instances of Docker images — containers run the actual applications. A container includes an application and all of its dependencies. It shares the kernel with other containers, and runs as an isolated process in user space on the host OS. You created a container using docker run which you did using the alpine image that you downloaded. A list of running containers can be seen using the docker container ls command.
Docker daemon - The background service running on the host that manages building, running and distributing Docker containers.
Docker client - The command line tool that allows the user to interact with the Docker daemon.
Docker Hub - Store is, among other things, a registry of Docker images. You can think of the registry as a directory of all available Docker images. You’ll be using this later in this tutorial.

Where do images get pulled from by default when not found locally?

( ) Docker Trusted Registry
(x) Docker Hub
( ) There is no default
( ) Docker Store

Which command lists your Docker images?

(x) docker image ls
( ) docker run
( ) docker container ls

Swarm Mode Introduction for IT Pros

Tue, 12 Sep 2017 00:00:00 +0000

So far we have explored using single instances of containers running on a single host, much like a developer might do when working on a single service application or like an IT adminstrator might do on a test rig. Production applications are usually much more complex and this single server model will not work to coordinate 10s or 100s of containers and the network connections amongst them, not to mention the need to ensure availability and the ability to scale.

For real applications IT users and app teams need more sophisticated tools. Docker supplies two such tools: Docker Compose and Docker Swarm Mode. The two tools have some similarities but some important differences:

Compose is used to control multiple containers on a single system. Much like the Dockerfile we looked at to build an image, there is a text file that describes the application: which images to use, how many instances, the network connections, etc. But Compose only runs on a single system so while it is useful, we are going to skip Compose¹ and go straight to Docker Swarm Mode.
Swarm Mode tells Docker that you will be running many Docker engines and you want to coordinate operations across all of them. Swarm mode combines the ability to not only define the application architecture, like Compose, but to define and maintain high availability levels, scaling, load balancing, and more. With all this functionality, Swarm mode is used more often in production environments than its more simplistic cousin, Compose.

The application

The voting app is a multi-container application often used for demo purposes during Docker meetups and conferences. It basically allow users to vote between two choices, the default being “cat” and “dog”, but could be “space” or “tab”, too, if you feel like it. This application is available on Github and updated very frequently when new features are developed.

Initialize Your Swarm

First thing we need to do is tell our Docker hosts we want to use Docker Swarm Mode. Swarms can be just a single node, but that is unusual as you would have no high availability capabilities and you would severely limit your scalability. Most production swarms have at least three manager nodes in them and many worker nodes. Three managers is the minimum to have a true high-availability cluster with quorum. Note that manager nodes can run your container tasks the same as a worker node, but this functionality can also be separated so that managers only perform the management tasks.

Initializing Docker Swarm Mode is easy. In the first terminal window labeled [node1] enter the following:

docker swarm init --advertise-addr $(hostname -i)

That’s it - you now have your first Swarm manager and it is listening on the IP address returned by the (hostname -i) command. You should see some output that looks like this:

Swarm initialized: current node (tjocs7ul557phkmp6mkpjmu3f) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token <token> <host>

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

In the output of your swarm init, you are given a command in the middle that looks like docker swarm join -token SWMTKN-X-abcdef..... which you use to join workers nodes to the swarm. You are also given a second command docker swarm join-token manager for adding additional managers.

We are going to add a worker. Copy the “docker swarm join…” command from your manager’s output and paste it in the 2nd terminal window on your screen. Make sure to copy the entire command - it’s likely to break across multiple lines - and do not copy the sample command above because your token will be different.

You now officially have a Docker Swarm! Currently, you have one manager and one worker. As hinted at above, you would almost always have 3 or more manager nodes and several worker nodes in order to maintain high availability and scalability, but one of each is enough to get started.

Show Swarm Members

From the first terminal window, check the number of nodes in the swarm (running this command from the second terminal worker node will fail as swarm related commands need to be issued against a swarm manager).

docker node ls

The above command should output 2 nodes, the first one being the manager, and the second one a worker. You should see that your manager node is also the “Leader”. This is because you only have one manager node. The Leader is just what it sounds like: the main control node for all the managers. If your Leader node goes down for some reason, the other manager nodes will elect a new leader; just one reason why you would always have multiple manager nodes in true production.

Here is a view of managers and workers in Docker Swarm Mode. In our exercise, we have just one manager and one worker, but you can see how multiple managers and workers interact in the diagram:

Clone the Voting App

Now to do something interesting we will retrieve the sample voting app code from Github.

Ensure you are in the first terminal - the manager - and enter the following two commands:

git clone https://github.com/docker/example-voting-app
cd example-voting-app

Deploy a Stack

A stack is a group of services that are deployed together: multiple containerized components of an application that run in separate instances. Each individual service can actually be made up of one or more containers, called tasks and then all the tasks & services together make up a stack.

As with Dockerfiles and the Compose files, the file that defines a stack is a plain text file that is easy to edit and track. In our exercise, there is a file called docker-stack.yml in the current folder which will be used to deploy the voting app as a stack. Enter the following to investigate the docker-stack.yml file:

cat docker-stack.yml

This YAML file defines our entire stack: the architecture of the services, number of instances, how everything is wired together, how to handle updates to each service. It is the source code for our application design. A few items of particular note:

Near the top of the file you will see the line “services:”. These are the individual application components. In the voting app we have redis, db, vote, result, worker, and visualizer as our services.
Beneath each service are lines that specify how that service should run:
- Notice the familiar term image from earlier labs? Same idea here: this is the container image to use for a particular service.
- Ports and networks are mostly self-explanatory although it is worth pointing out that these networks and ports can be privately used within the stack or they can allow external communication to and from a stack.²
- Note that some services have a line labeled replicas: this indicates the number of instances, or tasks, of this service that the Swarm managers should start when the stack is brought up. The Docker engine is intelligent enough to automatically load balance between multiple replicas using built-in load balancers. (The built-in load balancer can, of course, be swapped out for something else.)

Ensure you are in the [node1] manager terminal and do the following:

docker stack deploy --compose-file=docker-stack.yml voting_stack

You can see if the stack deployed from the [node1] manager terminal

docker stack ls

The output should be the following. It indicates the 6 services of the voting app’s stack (named voting_stack) have been deployed.

NAME          SERVICES
voting_stack  6

We can get details on each service within the stack with the following:

docker stack services voting_stack

The output should be similar to the following, although naturally your IDs will be unique:

ID            NAME                     MODE        REPLICAS  IMAGE
10rt1wczotze  voting_stack_visualizer  replicated  1/1       dockersample
s/visualizer:stable
8lqj31k3q5ek  voting_stack_redis       replicated  1/1       redis:alpine
nhb4igkkyg4y  voting_stack_result      replicated  1/1       dockersample
s/examplevotingapp_result:before
nv8d2z2qhlx4  voting_stack_db          replicated  1/1       postgres:9.4
ou47zdyf6cd0  voting_stack_vote        replicated  2/2       dockersample
s/examplevotingapp_vote:before
rpnxwmoipagq  voting_stack_worker      replicated  1/1       dockersample
s/examplevotingapp_worker:latest

If you see that there are 0 replicas just wait a few seconds and enter the command again. The Swarm will eventually get all the replicas running for you. Just like our docker-stack file specified, there are two replicas of the voting_stack_vote service and one of each of the others.

Let’s list the tasks of the vote service.

docker service ps voting_stack_vote

You should get an output like the following one where the 2 tasks (replicas) of the service are listed.

ID            NAME                 IMAGE
      NODE   DESIRED STATE  CURRENT STATE           ERROR  PORTS
my7jqgze7pgg  voting_stack_vote.1  dockersamples/examplevotingapp_vote:be
fore  node1  Running        Running 56 seconds ago
3jzgk39dyr6d  voting_stack_vote.2  dockersamples/examplevotingapp_vote:be
fore  node2  Running        Running 58 seconds ago

From the NODE column, we can see one task is running on each node. This app happens to have a built-in SWARM VISUALIZER to show you how the app is setup and running. You can also access the front-end web UI of the app to cast your vote for dogs or cats, and track how the votes are going on the result page. Try opening the front-end several times so you can cast multiple votes. You should see that the “container ID” listed at the bottom of the voting page changes since we have two replicas running.

The SWARM VISUALIZER gives you the physical layout of the stack, but here is a logical interpretation of how stacks, services and tasks are inter-related:

Scaling An Application

Let us pretend that our cats vs. dogs vote has gone viral and our two front-end web servers are no longer able to handle the load. How can we tell our app to add more replicas of our vote service? In production you might automate it through Docker’s APIs but for now we will do it manually. You could also edit the docker-stack.yml file and change the specs if you wanted to make the scale size more permanent. Type the following at the [node1] terminal:

docker service scale voting_stack_vote=5

Now enter your docker stack services voting_stack command again. You should see the number of replicas for the vote service increase to 5 and in a few seconds Swarm will have all of them running. Go back to your front-end voting UI and refresh the page a few times. You should see the container ID listed at the bottom cycle through all 5 of your containers. If you go back and refresh your SWARM VISUALIZER you should see your updated architecture there as well.

Here’s our new architecture after scaling:

That’s all there is to it! Docker Swarm can easily and quickly scale your application’s services up and down as needs require. Again, in many situations you would probably want to automate this rather than manually scaling, which is pretty easy through the Docker APIs. You also have the option to swap out the built-in load balancer for something with additional controls, like an F5 or Citrix NetScaler or some other software you prefer.

Conclusion

Using only a couple of commands enables you to deploy a stack of services using Docker Swarm Mode to orchestrate the entire stack, all maintained in the simple, human readable Docker Compose file format.

What next?

Hopefully these first few labs have given you some familiarity with Docker, containers, and Docker Swarm Mode orchestration. We encourage you to keep Playing With Docker to learn more. There are several things you can do to continue your learning:

Keep learning right here in the Play With Docker classroom with Phase 2 and Phase 3 of the Docker for IT Pros and System Administrators learning path.
- In Phase 2 you will learn more about orchestration, security, and networking
- Phase 3 will bring it all together as you implement a proof-of-concept application
Run the labs locally on your machine. You can download the free Docker for Mac or Docker for Windows clients and run some of these labs right on your own machine.
- You can also sign up for a free Docker Store account. You get a free Docker repository of your very own so that you can try taking the images we created in these labs and storing them in the community hub.

Footnotes:

1: If you want to go through a lab using Docker Compose the Orchestration, Part 1 lab on play-with-docker will guide you through the basics.↩

2: Networking is an important concept and there are many more details available on Docker Docs ↩

What is a stack?

(x) a multi-service app running on a Swarm
( ) a way of creating multiple nodes
( ) a method of using multiple compose files to run an app

A stack can:

(x) be deployed from the commandline
(x) use the compose file format to deploy
( ) run a Dockerfile
( ) be used to manage your hosts
(x) be used to manage services over multiple nodes

Docker for IT Pros and System Administrators

Tue, 25 Jul 2017 00:00:00 +0000

This self-paced learning journey is designed for IT Pros and System Administrators and will introduce you to Docker containers and the ways IT organizations like yours have started using Docker in their environment. We’ve designed three stages to guide you through this journey:

Stage 1: The Basics

This stage will

Get you familiar with the core concepts of Docker
Help you understand the fundamental value proposition for Docker
Help you see how Docker can help your organization

Stage 2: Digging Deeper

This stage will help you

Understand the architecture of Docker, and the core features
Understand how to integrate Docker into your existing application infrastructure
Develop a proof of concept application deployment

Stage 3: Moving to Production

This final stage will help you

Prepare to implement a full proof of concept application
Develop a strategy for integrating Docker into your existing production environment
Become recognized as a leader in your organization on implementing Docker

Reducing nodejs Docker images size by %50 using multi-stage builds and Zeit pkg.

Thu, 04 May 2017 00:00:00 +0000

This is a followup interactive tutorial from this blogpost to reduce docker nodejs images by 50% using Zeit pkg and docker multi-stage builds.

The traditional way

Before jumping into the magic trick, we’ll build a nodejs docker image as we would do traditionally to show the space that we’re saving by using this new approach.

Let’s create a simple node application:

echo 'console.log("Hello, pkg")' | tee index.js

We’ll now package it using a very basic dockerfile

echo 'FROM node:boron-slim
COPY . /app
CMD ["node", "/app/index.js"] ' | tee Dockerfile

Finally, we’ll build our image and check it’s size:

docker build -t myapp .
docker image ls | grep myapp

You can see that our myapp is 200+MB

Using PKG and multi-stage builds

Let’s create now a new Dockerfile that bundles our app with pkg and uses multi-stage builds to package it in a smaller docker image

echo 'FROM node:boron
RUN npm install -g pkg pkg-fetch

ENV NODE node6
ENV PLATFORM linux
ENV ARCH x64

RUN /usr/local/bin/pkg-fetch ${NODE} ${PLATFORM} ${ARCH}

COPY . /app
WORKDIR /app
RUN /usr/local/bin/pkg --targets ${NODE}-${PLATFORM}-${ARCH} index.js

FROM debian:jessie-slim
COPY --from=0  /app/index /
CMD ["/index"]' | tee Dockerfile-pkg

We’ll build our app the same way as before but using our new Dockerfile:

docker build -t myapp_pkg -f Dockerfile-pkg .
docker image ls | grep myapp_pkg

As you can see, our new docker image is around 100MB and it works as expected:

docker run myapp_pkg

Service Discovery under Docker Swarm Mode

Wed, 12 Apr 2017 00:00:00 +0000

Service Discovery under Swarm Mode.

Purpose

The purpose of this lab is to illustrate how Service Discovery works under Swarm Mode.

The application

WordPress is an open-source content management system (CMS) based on PHP and MySQL. It is a very simple containers application often used for demo purposes during meetup and conferences.

Init your swarm

Let’s create a Docker Swarm first. Open up the first instance and initiate Swarm mode cluster.

docker swarm init --advertise-addr $(hostname -i)

This node becomes a master node. The output displays a command to add a worker node to this swarm as shown below:

Swarm initialized: current node (xf323rkhg80qy2pywkjkxqusp) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-089phhmfamjor1o1qj8s0l4wdhyvegphg6vtt9p3s8c35upltk-eecvhhtz1f2vpjhvc70v6v
vzb \
    10.0.50.3:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructi
ons.

The above token ID is unique for every swarm mode cluster and hence might differ for your setup. From the output above, copy the join command (watch out for newlines).

Next, Open up the new instance and paste the below command. This should join the new node to the swarm mode cluster and this new node becomes a worker node. In my case, the command would look something like this:

 docker swarm join \
    --token SWMTKN-1-089phhmfamjor1o1qj8s0l4wdhyvegphg6vtt9p3s8c35upltk-eecvhhtz1f2vpjhvc70v6v
vzb \
    10.0.50.3:2377

Output:

$ docker swarm join --token SWMTKN-1-089phhmfamjor1o1qj8s0l4wdhyvegphg6vtt9p3s8c35upltk-eecvhh
tz1f2vpjhvc70v6vvzb 10.0.50.3:2377
This node joined a swarm as a worker.

Show members of swarm

Type the below command in the first terminal:

docker node ls

The output shows you both the manager and worker node indicating 2-node cluster:

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
xf323rkhg80qy2pywkjkxqusp *  node1     Ready   Active        Leader
za75md1p0hpc2qswefj8uyktk    node2     Ready   Active

If you try to execute an administrative command in a non-leader node worker, you’ll get an error. Try it here:

docker node ls

Create an overlay network

docker network create -d overlay net1

The above command generates an ID:

4md6wyy0pdpdzku6dj2z7yxjf

List out the newly created overlay network using the below command:

docker network ls

The output should show the newly added network called “net1” holding swarm scope .

NETWORK ID          NAME                DRIVER              SCOPE
c30f13d9c242        bridge              bridge              local
990fa0ad6ab6        docker_gwbridge     bridge              local
c60123ff7abf        host                host                local
v7sp7ev6xfoo        ingress             overlay             swarm
4md6wyy0pdpd        net1                overlay             swarm
333c7d045239        none                null

Creating MYSQL service

docker service create \
           --replicas 1 \
           --name wordpressdb \
           --network net1 \
           --env MYSQL_ROOT_PASSWORD=mysql123 \
           --env MYSQL_DATABASE=wordpress \
          mysql:latest

The above command creates a service named “wordpressdb” which belongs to “net1” network which runs a single replica of the container. It displays service ID as an output as shown:

ip9a8zl9rke256q92itgrm8ov

Run the below command to list out the service:

docker service ls

The output should be like the following one (your ID will display different though).

ID                  NAME                MODE                REPLICAS            IMAGE
ip9a8zl9rke2        wordpressdb         replicated          1/1                 mysql:latest

Let’s list the tasks of the wordpressdb service.

docker service ps wordpressdb

You should get an output like the following one where the 1 task of the service are listed.

ID                  NAME                IMAGE               NODE                DESIRED STATE
      CURRENT STATE                ERROR               PORTS
puoe9lvfkcia        wordpressdb.1       mysql:latest        node1               Running
      Running about a minute ago

Creating WordPress service

docker service create \
           --replicas 4 \
           --name wordpressapp \
           --network net1 \
           --env WORDPRESS_DB_HOST=wordpressdb \
           --env WORDPRESS_DB_PASSWORD=mysql123 \
          wordpress:latest

The above command creates a service named “wordpressapp” which belongs to “net1” network which runs 4 copies of wordpressapp container. As output, this command displays a service ID as:

m4hca6rliz8wer2aojayv01r5

Listing out the services:

docker service ls

Output:

ID                  NAME                MODE                REPLICAS            IMAGE
ID                  NAME                MODE                REPLICAS            IMAGE
ip9a8zl9rke2        wordpressdb         replicated          1/1                 mysql:latest
m4hca6rliz8w        wordpressapp        replicated          4/4                 wordpress:late
st

You can list the tasks of the wordpressapp service using the command:

docker service ps wordpressapp

Output:

ID                  NAME                IMAGE               NODE                DESIRED STATE
      CURRENT STATE                ERROR               PORTS
zg7wpvs1rbki        wordpressapp.1      wordpress:latest    node2               Running
      Running 58 seconds ago
8rybe5m4urik        wordpressapp.2      wordpress:latest    node1               Running
      Running about a minute ago
scia4v5i1znj        wordpressapp.3      wordpress:latest    node2               Running
      Running 58 seconds ago
4avyixggcb8n        wordpressapp.4      wordpress:latest    node1               Running
      Running about a minute ago
      

Service Discovery

Let us try to discover wordpressdb service from within one of wordpressapp container. Open up the manager node instance and run the below command:

Open up instance of worker node and verify what containers are running:

docker ps

This should display number of tasks(containers) running on the worker node locally:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS
           PORTS               NAMES
52f16028e12c        wordpress:latest    "docker-entrypoint..."   2 minutes ago       Up 2 minu
tes        80/tcp              wordpressapp.1.zg7wpvs1rbkiy4zwo71yk031i
f3271e89d54e        wordpress:latest    "docker-entrypoint..."   2 minutes ago       Up 2 minu
tes        80/tcp              wordpressapp.3.scia4v5i1znj378gujluad2ku

As shown above, there are 2 instances of wordpressapp task(container) running on the worker node.

Now, Open up manager node and confirm what task are running:

docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS
           PORTS               NAMES
b68d99cad3da        wordpress:latest    "docker-entrypoint..."   5 minutes ago       Up 4 minu
tes        80/tcp              wordpressapp.2.8rybe5m4urikqsqje6hcpou9t
657cff3e37d5        wordpress:latest    "docker-entrypoint..."   5 minutes ago       Up 4 minu
tes        80/tcp              wordpressapp.4.4avyixggcb8neej1h395ognt2
e71c164c36b3        mysql:latest        "docker-entrypoint..."   10 minutes ago      Up 10 min
utes       3306/tcp            wordpressdb.1.puoe9lvfkciavkrzrkbrhrl6e

As we notice, there are 2 instances of wordpressapp task(container) running on the manager node(shown above) and 1 instance of wordpressdb.

Let’s pick up the wordpressdb task running on the manager node and try to reach out to wordpressapp running on the remote worker node. Because the container is missing the ping command we need to install it first:

docker exec -it e71 bash -c "apt update && apt -y install iputils-ping"

Once installed, we can ping wordpressapp as shown below:

docker exec -it e71 ping wordpressapp

This should work successfully and be able to ping the wordpressapp as service name.

PING wordpressapp (10.0.0.4): 56 data bytes
64 bytes from 10.0.0.4: icmp_seq=0 ttl=64 time=0.052 ms
^C--- wordpressapp ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.052/0.052/0.052/0.000 ms

Let us try to reach out to remote wordpressapp container from one of the wordpressdb instance running on the worker node by its hostname:

docker exec -it e71 ping wordpressapp.3.scia4v5i1znj378gujluad2ku

Output:

PING wordpressapp.3.scia4v5i1znj378gujluad2ku (10.0.0.5): 56 data bytes
64 bytes from 10.0.0.5: icmp_seq=0 ttl=64 time=6.175 ms
64 bytes from 10.0.0.5: icmp_seq=1 ttl=64 time=0.131 ms
^C--- wordpressapp.3.scia4v5i1znj378gujluad2ku ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.131/3.153/6.175/3.022 ms

Voila ! We are able to ping wordpressapp service from container(running wordpresdb task) using the service name.Also, we were successful in reaching out to remote wordpressapp container using its hostname from one of wordpressdb container running in maanager node.

What features are available under Docker Swarm Mode:

Docker volume sshfs

Fri, 07 Apr 2017 00:00:00 +0000

Docker volume plugins

Docker Engine volume plugins enable Engine deployments to be integrated with external storage systems such as Amazon EBS, and enable data volumes to persist beyond the lifetime of a single Docker host. See the plugin documentation for more information.

Setting up

First we install openssh and configure test user with password testpassword in the second node

apk add --no-cache openssh
adduser -D test
echo "test:testpassword" | chpasswd
ssh-keygen -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key
/usr/sbin/sshd

You can safely ignore any Could not load host key error messages.

Now, let’s install the plugin and create a docker volume with it

mkdir /var/lib/docker/plugins # Shouldn't have to do this if graph folder is somewhere else
docker plugin install --grant-all-permissions vieux/sshfs
docker volume create -d vieux/sshfs -o sshcmd=test@node2:/home/test -o password=testpassword sshvolume

We can now use the plugin volume in a container as usual

docker run -it -v sshvolume:/test alpine sh -c 'echo "Hello world" > /test/somefile'

Now we check that the file has been created in the second node

cat /home/test/somefile

Swarm synchronous services

Mon, 03 Apr 2017 00:00:00 +0000

Synchronous service create and service update

A nice PATCH has been merged into Docker a few minutes ago that allows service creations and updated to be executed synchronously.

Note: If you don’t see the progress bars in the terminal on the right, try resizing the pane to make the term bigger.

Creating a synchronous service

Initialize your swarm

docker swarm init --advertise-addr eth1

Create a new synchronous serivce using the new -d flag

docker service create -d=false --name top --replicas 5 busybox top

You should an output similar to the following in the terminal:

mmsdrpbigre7ls9vp6mhig3vz
overall progress: 5 out of 5 tasks
1/5: running   [==================================================>]
2/5: running   [==================================================>]
3/5: running   [==================================================>]
4/5: running   [==================================================>]
5/5: running   [==================================================>]
verify: Waiting 1 seconds to verify that tasks are stable...

As you can see, a nice progress bar will display the status of the overall deployment.

Note: If you press Ctrl+C while the service is being created, it will be sent to background automatically

We’ll check how update also works with the --detach paramter now.

docker service update -d=false --force --update-parallelism 0 top

WebApps with Docker Flow Proxy

Sat, 01 Apr 2017 05:21:47 +0000

Before anything How to use this course

Please, validate the Google capcha to activate the shell on the right. Then, you can either copy the commands yourself, or simply click on the grey boxes to automatically copy commands into the terminal.

echo 'execute command on node1!!'

echo 'execute command on node2!!'

Please note that this platform is not secure and you should not store personal datas
the instance will be removed after few hours

Predictive Load-balancing name using Docker Flow Proxy

In this course, we will leverage the power of Docker Swarm Mode, released with Docker 1.13, and the great features of vfarcic Docker Flow Proxy which provide an easy way to reconfigure proxy every time a new service is deployed, or when a service is scaled. It uses docker service labels to define the metadata and rules for its dynamically-configured routing rules to send traffic from the PRoxy to real applications (regardless of the host they are within a Docker Swarm Cluster).

Docker Flow Proxy is composed on two parts :

The purpose of swarm-listener is to monitore swarm services (add, remove, scale..) and to send requests to the proxy whenever a service is created or destroyed. It must be running on a Swarm Manager and will queries Docker API in search for newly created services.

It uses docker service’s labels (com.df.*) to define the metadata and rules for dynamically configure routing rules of the Proxy.

First we will enable the Swarm mode

In this tutorial, we will only use a 2 node swarm cluster, but it will work exactly the same way with more nodes!

docker swarm init --advertise-addr=$(hostname -i)
docker swarm join-token manager

Copy the join command output and paste it in the other terminal to form a 2 node swarm cluster.

show members of the swarm

docker node ls

If you correctly execute, the above command, you must see 2 nodes:

$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
7p167ggf1wi3ox52z8ga2myu6 *  node1     Ready   Active        Leader
og1irjjh2fjtwt7dko7ht0qnq    node2     Ready   Active        Reachable

Create Docker Flow Proxy Docker Containers

We will start by creating a Docker Compose file named proxy.yml, which will defines our 2 services proxy and swarm-listener of our Docker Flow Proxy stack :

You can Click on the grey box to automatically copy the content on the terminal (don’t mess with the order of commands ;) )

cat <<EOF > proxy.yml
version: "3"

services:


  proxy:
    image: vfarcic/docker-flow-proxy
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    environment:
      - LISTENER_ADDRESS=swarm-listener
      - MODE=swarm
    networks:
      - public
    deploy:
      replicas: 2
      restart_policy:
        condition: on-failure


  swarm-listener:
    image: vfarcic/docker-flow-swarm-listener
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - public
    environment:
      - DF_NOTIFY_CREATE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/reconfigure
      - DF_NOTIFY_REMOVE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/remove
    deploy:
      replicas: 1
      placement:
        constraints: [node.role == manager]		                                                   
      restart_policy:
        condition: on-failure
      
networks:
  public:
    driver: overlay
    ipam:
      driver: default
      config:
      - subnet: 10.1.0.0/24

EOF

We are using version 3 of compose file (mandatory for docker stack deploy)
We are using image from vfarcic on docker Hub
Docker will create an overlay networks named public, on which will will add each container we want to publish
We uses constraints to deploy the swarm-listener service on a swarm manager (as it needs to listen to swarm events)
We gives the proxy service the address of the swarm-listener
We gives the swarm-listener 2 API endpoint to reconfigure the Proxy through environment variables.
DF_NOTIFY_* environments variables defines the url of the Proxy API for reconfiguration.

Launch the Docker Containers

docker stack deploy proxy --compose-file proxy.yml

The proxy container is configured to listen on port 80 and 443 for the standard HTTP traffic, and will listen privately on the internal network on port 8080 for the reconfiguration API requests.

Check docker networks

docker network ls

You should see that a network named proxy_public has been created with Driver overlay.

Later If we want others containers to be able to be accessible through the proxy load balancer we will need to attached them to this network.

See Your Docker Swarm Stack

List all your deployed stacks, and view detailed on a specific stack

docker stack ls
docker stack ps proxy

We must have 2 Proxy Running and 1 swarm-listener Running

Since we have set 2 replicas for the proxy service it will be deployed on both nodes while swarm-listener must be on one manager node

View logs of our Proxy service

docker service logs --tail=10 proxy_proxy

View logs of our swarm-listener service

docker service logs --tail=10 proxy_swarm-listener

Scaling the Proxy service

Normally, creating a new instance of the proxy service, means that it will starts without any state, as a result, the new instances would not have any knowledge of our already deployed services. Fortunately docker-flow provides an environment variable LISTEN_ADDRESS=swarm-listener which tells the proxy the adress of the swarm-listener to resend notifications for all the services. As a result, each proxy instance will soon have the same state as the other :)

Deploy our first service and connect it to the Docker Flow Proxy

The swarm-listener service is listening to docker swarm events informations, and will reconfigure the proxy service based on the service’s metadatas, we need to configure thoses metadata as docker service labels:

Configure service with routing based on URL Path

We can set a label to inform the proxy to route the traffic according to the target service URI Path using com.df.* rules labels:

cat <<EOF > http.yml
version: "3"

services:
  http:
    image: emilevauge/whoami
    networks:
      - proxy_public
    deploy:
      replicas: 3
      labels:
        - com.df.notify=true
        - com.df.distribute=true
        - com.df.servicePath=/http/
        - com.df.reqPathSearch=/http/
        - com.df.reqPathReplace=/renamed/
        - com.df.port=80

networks:
  proxy_public:
    external: true

EOF

!!Note: Because we are working with Docker Swarm Mode, labels must be set at the service level in the deploy section, instead of at container level!!

The notify label ask swarm-listener to re-configure Flow Proxy
The distribute label means that reconfiguration should be applied to all Proxy instances.

We are using Docker 1.13 networking features (routing mesh, and VIP) So that Docker takes care of load balancing on all instances of our service, and so that there is no need to reconfigure the proxy every time a new instance is deployed. (We configure our docker’s VIP service IP adress in the proxy, so 1 IP per service)

launch the container

docker stack deploy http --compose-file http.yml

The proxy container should have been attached the proxy_public network, which we can verify by inspecting the network:

docker network inspect proxy_public

check Service status

docker stack ps http

Request the service

We have defined that our service will be receive the request if an incoming request starts with the path /http. This was done using the rule in the service’s com.df.servicePath=/http label

We may now be able to reach our service from any host : from node1

curl http://localhost/http/

from node2

curl http://localhost/http/

We should see a response that was generated by the service. The Url on the service site may have been rewritten (see the /rewrited/ in the GET parameter)

You can request the service in your Browser:

Link to http service

You can request the logs of the Proxy Load Balancer:

docker service logs --tail=10 -f proxy_proxy

You can request the logs of the application

docker service logs --tail=10 http_http

Scaling Service

Swarm is continuously monitoring containers health. If one of them fails, it will redeployed to one of available nodes. If a whole node fails, or if we ask to drain all containers out of a node for maintenance, Swarm will recreate all the containers that were running on that node. In production we need to reach zero down time, and so to guarentee our nodes will be available, we need to scale our services, so that we have many instances of our service running on severals nodes. That way, while we are waiting for one instance to recuperate from a failure, others can take over the load.

We can use docker swarm to scale the services of our applications: Exemple, scale our http service to use 5 instances:

docker service scale http_http=5

Check that you have 5 instances of the service :

docker service ps http_http

You can make local calls to the http service and see the loadbalancing :

curl http://localhost/http/

On every request, it’s a different docker container that will respond!

Retrieve Proxy Configuration

If we have activated the admin port (8080), then we can request the proxy to retrieve the configuration Docker Flow Proxy is base on HAProxy so what we retrieve here is the HAProxy configuration

curl http://localhost:8080/v1/docker-flow-proxy/config

Deploy a Microservice Application

We have see how we can leverage Docker labels to dynamically customize our LoadBalancing routing rules, and how docker-compose can be used to create and link services together.

Now let’s try to launch a more complicated Microservice application.

We will uses Docker’s vote microservice application with custom labels to be used within our Docker Flow Proxy loadbalancer.

The voting application is composed of :

A Python webapp which lets you vote between two options
A Redis queue which collects new votes
A Java worker which consumes votes and stores them in…
A Postgres database backed by a Docker volume
A Node.js webapp which shows the results of the voting in real time

Run voting microservice application

First you need to Retrieve voting-app application

git clone https://github.com/allamand/example-voting-app.git

Go to the stack directory

cd example-voting-app

and launch the app using docker-compose file, you can view the docker-compose-flow-proxy.yml file

docker stack deploy cloud -c docker-compose-flow-proxy.yml

This command will build each part of the microservice from sources. It may take a little time to get all services up & running (time to download images..) You can take a coffee since this may take a little to finish ;)

Rewriting Paths

In this example, we need the incoming requests that starts with /vote/ or /result/ to be routed to the according services by the proxy. But each of our service needs traffic to be send on /, so we need the Proxy to rewrite the Path while sending the request.

For that we are using specific docker-flow labels reqPathSearch and reqPathReplace:

      labels:
        - com.df.notify=true
        - com.df.distribute=true
        - com.df.servicePath=/vote/
        - com.df.reqPathSearch=/vote/
        - com.df.reqPathReplace=/
        - com.df.port=80

To monitor the setup state, you can use:

docker stack ps cloud

Be carreful, the Output shows two state columns :

Desired State which represents what you are asking to swarm

Current State which is the current state of the container (which may be stuck in Preparing for a moment while downloading the images).

Once all containers are in the Running state, you can start test the application.

While the application is working you can take a look at the docker-compose file we are deploying :

cat docker-compose-flow-proxy.yml

We can view the updated configuration on the proxy API

curl http://localhost:8080/v1/docker-flow-proxy/config

You can now make your Vote!!

Link to vote service

And See the results of votes

Link to result service

You can see the logs of the services :

docker service logs --tail=10 cloud_vote

You are now able to deploy any stack on Docker Swarm Mode using docker-compose and Docker Flow Proxy!

Bonus

We can add a Swarm visualizer service :

cat <<EOF > visualizer.yml
version: "3"

services:
  visu:
    image: dockersamples/visualizer
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    networks:
      - proxy_public
    ports:
      - 81:8080
    deploy:
      replicas: 1
      placement:
        constraints: [node.role == manager]		                                                   
      restart_policy:
        condition: on-failure

networks:
  proxy_public:
    external: true

EOF

launch the container

docker stack deploy visu --compose-file visualizer.yml

docker service ps visu_visu

wait few second for the service to start

We can now target directly the port 81 of our swarm cluster and docker will direclty reach our Visualizer service

Link to Visualizer service

This should be something like :

Free resources

When you are finished with this tutorial, please free the unused resources:

docker stack rm cloud
docker stack rm visu
docker stack rm http
docker stack rm proxy

Multi-stage builds

Sun, 26 Mar 2017 00:00:00 +0000

Prerequisites

This tutorial will need the master build of Docker, which is available on Play With Docker.

Multi-staged builds

A common pipe-line for building applications in Docker involves adding SDKs and runtimes, followed by adding code and building it. The most efficient way to get a small image tends to be to use 2-3 Dockerfiles with different filenames where each one takes the output of the last. This is referred to as the Builder pattern in the Docker community.

This lab explores a new bleeding-edge feature called Multi-stage builds. It is not yet released into a Docker version, but when it is available on the Docker Hub/Cloud and for all the Docker editions it will mean we can use a single Dockerfile with multiple stages instead of the Builder pattern.

Let’s build a simple Golang application which counts internal/external facing anchor tags to help us come up with an SEO rating.

Let’s try out the href-counter Docker image from the hub, then look at how to re-build from the Github repository:

docker run -e url=https://news.ycombinator.com alexellis2/href-counter

{"internal":197,"external":32}

You get a JSON object returned giving the total amount of internal vs external links.

Let’s clone the source:

git clone https://github.com/alexellis/href-counter
cd href-counter

Cloning into 'href-counter'...
remote: Counting objects: 24, done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 24 (delta 7), reused 12 (delta 1), pack-reused 0
Unpacking objects: 100% (24/24), done.

The old way of doing things

Let’s build the Docker image with all the Golang toolchain and see how big the image comes out as:

docker build -t alexellis2/href-counter:sdk . -f Dockerfile.build

Now check the size of the image:

docker images |grep href-counter

docker images |grep href-counter
href-counter        sdk                 131c782e8c35        30 second
s ago      692MB

The docker history command will show you that the layers we added during the build are only a small part of the resulting image (about 20MB +/-):

docker history alexellis2/href-counter:sdk |head -n 4

IMAGE               CREATED             CREATED BY
                   SIZE                COMMENT
f8b1953fb9c7        1 second ago        /bin/sh -c CGO_ENABLED=0 GOOS=linux go bui...   5.64MB
5d24895500e8        9 seconds ago       /bin/sh -c #(nop) COPY file:d3eec1f1fefbec...   1.71kB
d83dc0785057        9 seconds ago       /bin/sh -c go get -d -v golang.org/x/net/html   13.6MB
c6f59b210906        11 seconds ago      /bin/sh -c #(nop) WORKDIR /go/src/github.c...   0B

The image is quite large, but this Golang package can be built into a very small binary with no external dependencies, then added to an Alpine Linux base image.

The builder pattern

Type in cat builder.sh so you can see how the builder pattern uses two separate Dockerfiles. This will help us get the context for the next step where we will use a single Dockerfile.

./build.sh

#!/bin/sh
echo Building alexellis2/href-counter:build

docker build -t alexellis2/href-counter:build . -f Dockerfile.build
docker create --name extract alexellis2/href-counter:build
docker cp extract:/go/src/github.com/alexellis/href-counter/app ./app
docker rm -f extract

echo Building alexellis2/href-counter:latest
docker build -t alexellis2/href-counter:latest .

As you can see there are quite a few intermediate steps required to create an optimized image using the Builder pattern.

Let’s see how big the Docker image came out as:

docker images |grep alexellis2/href-counter

This is much smaller than when we built our first image with the Golang SDK included.

Multi-stage build example

While the builder pattern helps us achieve a small image, it does require extra leg-work for every piece of software we want to package in Docker.

Here is where multi-stage builds help us out. Instead of using a shell script to orchestrate two separate Dockerfiles, we can just use one and define stages throughout.

To use the Dockerfile.multi file in the Github repository to do a multi-stage build, then check the size of the resulting image against that of the image created by the Golang SDK base and the builder pattern.

docker build -t alexellis2/href-counter:multi . -f Dockerfile.multi

docker images |grep href-counter
alexellis2/href-counter   multi               44852229a1cc        2 minutes ago       10.3MB
alexellis2/href-counter   latest              bb997f819fbb        3 minutes ago       10.3MB
alexellis2/href-counter   build               298d3b970412        4 minutes ago       692MB
alexellis2/href-counter   sdk                 298d3b970412        4 minutes ago       692MB
alexellis2/href-counter   <none>              b0a73b688243        13 days ago         11.6MB

Here is an example of building “Hello World” in Go.

Create a new folder:

mkdir hello
cd hello

Create the app.go file:

echo 'package main

import "fmt"

func main() {
    fmt.Println("Hello world!")
}
' | tee app.go

Create a Dockerfile with the following contents:

echo '
FROM golang:1.7.3
COPY app.go .
RUN go build -o app app.go

FROM scratch
COPY --from=0 /go/app .
CMD ["./app"]
' | tee Dockerfile

Now build and run the Dockerfile:

docker build -t hello-world-lab .
docker run hello-world-lab

The resulting size of hello-world is very small:

docker images |grep hello-world-lab

More about the lab

This lab was built from a blog post. by Alex Ellis

Blog: Builder pattern vs. Multi-stage builds in Docker.

Note: We do not recommend moving over to multi-stage builds until they are fully available on the Docker Hub/Cloud and all editions of Docker. The example in build.sh provides an interim solution for using separate Dockerfiles to build and ship code.

Orchestration, part 1: from Compose to Swarm

Mon, 13 Mar 2017 00:00:00 +0000

This lab is longer than the other ones. It features a demo app built around a microservices architecture, defined through a Compose file.

You will run this app on a single node. Then, to scale that app on a cluster, you will learn the concepts powering SwarmKit, Docker’s native orchestration system. Armed with this new knowledge, you will setup your own Swarm cluster across five nodes, and use it to deploy the demo app.

We recommend that you open two tabs or two windows for that lab: one for the materials, another for the Play-With-Docker environment.

Links:

Introduction (recommended reading)
Part 1
Play-With-Docker environment

Orchestration, part 2: securing and operating Swarm

Mon, 13 Mar 2017 00:00:00 +0000

This lab is longer than the other ones. It is designed to follow the part 1 (“from Compose to Swarm”) but can also be done independently (it provides jump-start instructions if you want to skip part 1).

You will learn about the security features of Swarm: secrets, cluster locking, network encryption, and more.

You will also see how to implement logging, metrics, and will given a galore of tips and tricks to operate Swarm.

We recommend that you open two tabs or two windows for that lab: one for the materials, another for the Play-With-Docker environment.

Links:

Introduction (recommended reading)
Part 2
Play-With-Docker environment

Security Lab: Capabilities

Mon, 06 Mar 2017 00:00:00 +0000

Lab: Capabilities

Difficulty: Advanced

Time: Approximately 30 minutes

In this lab you’ll learn the basics of capabilities in the Linux kernel. You’ll learn how they work with Docker, some basic commands to view and manage them, as well as how to add and remove capabilities in new containers.

You will complete the following steps as part of this lab.

Step 1 - Introduction to capabilities
Step 2 - Working with Docker and capabilities
Step 3 - Testing Docker capabilities
Step 4 - Extra for experts

Step 1: Introduction to capabilities

In this step you’ll learn the basics of capabilities.

The Linux kernel is able to break down the privileges of the root user into distinct units referred to as capabilities. For example, the CAP_CHOWN capability is what allows the root user to make arbitrary changes to file UIDs and GIDs. The CAP_DAC_OVERRIDE capability allows the root user to bypass kernel permission checks on file read, write and execute operations. Almost all of the special powers associated with the Linux root user are broken down into individual capabilities.

This breaking down of root privileges into granular capabilities allows you to:

Remove individual capabilities from the root user account, making it less powerful/dangerous.
Add privileges to non-root users at a very granular level.

Capabilities apply to both files and threads. File capabilities allow users to execute programs with higher privileges. This is similar to the way the setuid bit works. Thread capabilities keep track of the current state of capabilities in running programs.

The Linux kernel lets you set capability bounding sets that impose limits on the capabilities that a file/thread can gain.

Docker imposes certain limitations that make working with capabilities much simpler. For example, file capabilities are stored within a file’s extended attributes, and extended attributes are stripped out when Docker images are built. This means you will not normally have to concern yourself too much with file capabilities in containers.

It is of course possible to get file capabilities into containers at runtime, however this is not recommended.

In an environment without file based capabilities, it’s not possible for applications to escalate their privileges beyond the bounding set (a set beyond which capabilities cannot grow). Docker sets the bounding set before starting a container. You can use Docker commands to add or remove capabilities to or from the bounding set.

By default, Docker drops all capabilities except those needed, using a whitelist approach.

Step 2: Working with Docker and capabilities

In this step you’ll learn the basic approach to managing capabilities with Docker. You’ll also learn the Docker commands used to manage capabilities for a container’s root account.

As of Docker 1.12 you have 3 high level options for using capabilities:

Run containers as root with a large set of capabilities and try to manage capabilities within your container manually.
Run containers as root with limited capabilities and never change them within a container.
Run containers as an unprivileged user with no capabilities.

Option 2 as the most realistic as of Docker 1.12. Option 3 would be ideal but not realistic. Option 1 should be avoided wherever possible.

Note: Another option may be added in future versions of Docker that will allow you to run containers as a non-root user with added capabilities. The correct way of doing this requires ambient capabilities which was added to the Linux kernel in version 4.3. Whether it is possible for Docker to approximate this behavior in older kernels requires more research.

In the following commands, $CAP will be used to indicate one or more individual capabilities. We’ll test these out in the next section.

To drop capabilities from the root account of a container.
```
docker run --rm -it --cap-drop $CAP alpine sh
```
To add capabilities to the root account of a container.
```
docker run --rm -it --cap-add $CAP alpine sh
```
To drop all capabilities and then explicitly add individual capabilities to the root account of a container.
```
docker run --rm -it --cap-drop ALL --cap-add $CAP alpine sh
```

The Linux kernel prefixes all capability constants with “CAP_”. For example, CAP_CHOWN, CAP_NET_ADMIN, CAP_SETUID, CAP_SYSADMIN etc. Docker capability constants are not prefixed with “CAP_” but otherwise match the kernel’s constants.

For more information on capabilities, including a full list, see the capabilities man page

Step 3: Testing Docker capabilities

In this step you will start various new containers. Each time you will use the commands learned in the previous step to tweak the capabilities associated with the account used to run the container.

Start a new container and prove that the container’s root account can change the ownership of files.
```
 docker run --rm -it alpine chown nobody /
```
The command gives no return code indicating that the operation succeeded. The command works because the default behavior is for new containers to be started with a root user. This root user has the CAP_CHOWN capability by default.
Start another new container and drop all capabilities for the containers root account other than the CAP_CHOWN capability. Remember that Docker does not use the “CAP_” prefix when addressing capability constants.
```
 docker run --rm -it --cap-drop ALL --cap-add CHOWN alpine chown nobody /
```
This command also gives no return code, indicating a successful run. The operation succeeds because although you dropped all capabilities for the container’s root account, you added the chown capability back. The chown capability is all that is needed to change the ownership of a file.
Start another new container and drop only the CHOWN capability form its root account.
```
 docker run --rm -it --cap-drop CHOWN alpine chown nobody /
```
```
 chown: /: Operation not permitted
```
This time the command returns an error code indicating it failed. This is because the container’s root account does not have the CHOWN capability and therefore cannot change the ownership of a file or directory.
Create another new container and try adding the CHOWN capability to the non-root user called nobody. As part of the same command try and change the ownership of a file or folder.
```
 docker run --rm -it --cap-add chown -u nobody alpine chown nobody /
```
```
chown: /: Operation not permitted
```
The above command fails because Docker does not yet support adding capabilities to non-root users.

In this step you have added and removed capabilities to a range of new containers. You have seen that capabilities can be added and removed from the root user of a container at a very granular level. You also learned that Docker does not currently support adding capabilities to non-root users.

Step 4: Extra for experts

The remainder of this lab will show you additional tools for working with capabilities form the Linux shell.

There are two main sets of tools for managing capabilities:

libcap focuses on manipulating capabilities.
libcap-ng has some useful tools for auditing.

Below are some useful commands from both.

You may need to manually install the packages required for some of these commands.

libcap

capsh - lets you perform capability testing and limited debugging
setcap - set capability bits on a file
getcap - get the capability bits from a file

libcap-ng

pscap - list the capabilities of running processes
filecap - list the capabilities of files
captest - test capabilities as well as list capabilities for current process

The remainder of this step will show you some examples of libcap and libcap-ng.

Listing all capabilities

The following command will start a new container using Alpine Linux, install the libcap package and then list capabilities.

    docker run --rm -it alpine sh -c 'apk add -U libcap; capsh --print'

   (1/1) Installing libcap (2.25-r0)
   Executing busybox-1.24.2-r9.trigger
   OK: 5 MiB in 12 packages
   Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip
   Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
   Securebits: 00/0x0/1'b0
    secure-noroot: no (unlocked)
    secure-no-suid-fixup: no (unlocked)
    secure-keep-caps: no (unlocked)
   uid=0(root)
   gid=0(root)
   groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)

Current is multiple sets separated by spaces. Multiple capabilities within the same set are separated by commas ,. The letters following the + at the end of each set are as follows:

e is effective
i is inheritable
p is permitted

For information on what these mean, see the capabilities manpage.

Experimenting with capabilities

The capsh command can be useful for experimenting with capabilities. capsh --help shows how to use the command:

docker run --rm -it alpine sh -c 'apk add -U libcap;capsh --help'

fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.ta
r.gz
(1/1) Installing libcap (2.25-r1)
Executing busybox-1.25.1-r0.trigger
OK: 4 MiB in 12 packages
usage: capsh [args ...]
  --help         this message (or try 'man capsh')
  --print        display capability relevant state
  --decode=xxx   decode a hex string to a list of caps
  --supports=xxx exit 1 if capability xxx unsupported
  --drop=xxx     remove xxx,.. capabilities from bset
  --caps=xxx     set caps as per cap_from_text()
  --inh=xxx      set xxx,.. inheritiable set
  --secbits=<n>  write a new value for securebits
  --keep=<n>     set keep-capabability bit to <n>
  --uid=<n>      set uid to <n> (hint: id <username>)
  --gid=<n>      set gid to <n> (hint: id <username>)
  --groups=g,... set the supplemental groups
  --user=<name>  set uid,gid and groups to that of user
  --chroot=path  chroot(2) to this path
  --killit=<n>   send signal(n) to child
  --forkfor=<n>  fork and make child sleep for <n> sec
  ==             re-exec(capsh) with args as for --
  --             remaing arguments are for /bin/bash
                 (without -- [capsh] will simply exit(0))

Warning: --drop sounds like what you want to do, but it only affects the bounding set. This can be very confusing because it doesn’t actually take away the capability from the effective or inheritable set. You almost always want to use --caps.

Modifying capabilities

Libcap and libcap-ng can both be used to modify capabilities.

Use libcap to modify the capabilities on a file.

The command below shows how to set the CAP_NET_RAW capability as effective and permitted on the file represented by $file. The setcap command calls on libcap to do this.
```
setcap cap_net_raw=ep $file
```
Use libcap-ng to set the capabilities of a file.

The filecap command calls on libcap-ng.
```
filecap /absolute/path net_raw
```
Note: filecap requires absolute path names. Shortcuts like ./ are not permitted.

Auditing

There are multiple ways to read out the capabilities from a file.

Using libcap:
```
getcap $file

$file = cap_net_raw+ep
```

Using libcap-ng:

$ filecap /absolue/path/to/file

file                     capabilities
/absolute/path/to/file        net_raw

Using extended attributes (attr package):

getfattr -n security.capability $file

# file: $file
security.capability=0sAQAAAgAgAAAAAAAAAAAAAAAAAAA=

Tips

Docker images cannot have files with capability bits set. This reduces the risk of Docker containers using capabilities to escalate privileges. However, it is possible to mount volumes that contain files with capability bits set into containers. Therefore you should use caution if doing this.

You can audit directories for capability bits with the following commands:

# with libcap
getcap -r /

# with libcap-ng
filecap -a

To remove capability bits you can use.

# with libcap
setcap -r $file

# with libcap-ng
filecap /path/to/file none

Security Lab: Seccomp

Mon, 06 Mar 2017 00:00:00 +0000

Lab: Seccomp

Difficulty: Advanced

Time: Approximately 20 minutes

seccomp is a sandboxing facility in the Linux kernel that acts like a firewall for system calls (syscalls). It uses Berkeley Packet Filter (BPF) rules to filter syscalls and control how they are handled. These filters can significantly limit a containers access to the Docker Host’s Linux kernel - especially for simple containers/applications.

You will complete the following steps as part of this lab.

Step 1 - Clone the labs GitHub repo
Step 2 - Test a seccomp profile
Step 3 - Run a container with no seccomp profile
Step 4 - Selectively remove syscalls
Step 5 - Write a seccomp profile
Step 6 - A few Gotchas

Prerequisites

With this lab in Play With Docker you have all you need to complete the lab. If you are running this on another environment, you will need:

A Linux-based Docker Host with seccomp enabled
Docker 1.10 or higher (preferably 1.12 or higher)

The following commands show you how to check if seccomp is enabled in your system’s kernel:

Check from Docker 1.12 or higher

   $ docker info | grep seccomp
   Security Options: apparmor seccomp   

If the above output does not return a line with seccomp then your system does not have seccomp enabled in its kernel.

Check from the Linux command line

   $ grep SECCOMP /boot/config-$(uname -r)
   CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
   CONFIG_SECCOMP_FILTER=y
   CONFIG_SECCOMP=y

Seccomp and Docker

Docker has used seccomp since version 1.10 of the Docker Engine.

Docker uses seccomp in filter mode and has its own JSON-based DSL that allows you to define profiles that compile down to seccomp filters. When you run a container it gets the default seccomp profile unless you override this by passing the --security-opt flag to the docker run command.

The following example command starts an interactive container based off the Alpine image and starts a shell process. It also applies the seccomp profile described by <profile>.json to it.

   docker run -it --rm --security-opt seccomp=<profile>.json alpine sh ...

The above command sends the JSON file from the client to the daemon where it is compiled into a BPF program using a thin Go wrapper around libseccomp.

Docker seccomp profiles operate using a whitelist approach that specifies allowed syscalls. Only syscalls on the whitelist are permitted.

Docker supports many security related technologies. It is possible for other security related technologies to interfere with your testing of seccomp profiles. For this reason, the best way to test the effect of seccomp profiles is to add all capabilities and disable apparmor. This gives you the confidence the behavior you see in the following steps is solely due to seccomp changes.

The following docker run flags add all capabilities and disable apparmor: --cap-add ALL --security-opt apparmor=unconfined.

Step 1: Clone the labs GitHub repo

In this step you will clone the lab’s GitHub repo so that you have the seccomp profiles that you will use for the remainder of this lab.

Clone the labs GitHub repo.

git clone https://github.com/docker/labs

Change into the labs/security/seccomp directory.
```
cd labs/security/seccomp
```

The remaining steps in this lab will assume that you are running commands from this labs/security/seccomp directory. This will be important when referencing the seccomp profiles on the various docker run commands throughout the lab.

Step 2: Test a seccomp profile

In this step you will use the deny.json seccomp profile included the lab guides repo. This profile has an empty syscall whitelist meaning all syscalls will be blocked. As part of the demo you will add all capabilities and effectively disable apparmor so that you know that only your seccomp profile is preventing the syscalls.

Use the docker run command to try to start a new container with all capabilities added, apparmor unconfined, and the seccomp-profiles/deny.json seccomp profile applied.

docker run --rm -it --cap-add ALL --security-opt apparmor=unconfined --security-opt seccomp=seccomp-profiles/deny.json alpine sh

docker: Error response from daemon: cannot start a stopped process: unknown.

In this scenario, Docker doesn’t actually have enough syscalls to start the container!

Inspect the contents of the seccomp-profiles/deny.json profile.

cat seccomp-profiles/deny.json

{
     "defaultAction": "SCMP_ACT_ERRNO",
     "architectures": [
             "SCMP_ARCH_X86_64",
             "SCMP_ARCH_X86",
             "SCMP_ARCH_X32"
     ],
     "syscalls": [
     ]
}

Notice that there are no syscalls in the whitelist. This means that no syscalls will be allowed from containers started with this profile.

In this step you removed capabilities and apparmor from interfering, and started a new container with a seccomp profile that had no syscalls in its whitelist. You saw how this prevented all syscalls from within the container or to let it start in the first place.

Step 3: Run a container with no seccomp profile

Unless you specify a different profile, Docker will apply the default seccomp profile to all new containers. In this step you will see how to force a new container to run without a seccomp profile.

Start a new container with the --security-opt seccomp=unconfined flag so that no seccomp profile is applied to it.
```
docker run --rm -it --security-opt seccomp=unconfined debian:jessie sh
```
From the terminal of the container run a whoami command to confirm that the container works and can make syscalls back to the Docker Host.
```
whoami
```
```
root
```
To prove that we are not running with the default seccomp profile, try running a unshare command, which runs a shell process in a new namespace:
```
  unshare --map-root-user --user
  whoami
```
```
  root
```
Exit the new shell and the container.
```
exit
exit
```

Run the following strace command from your Docker Host to see a list of the syscalls used by the whoami program.

Your Docker Host will need the strace package installed.

apk add --update strace
strace -c -f -S name whoami 2>&1 1>/dev/null | tail -n +3 | head -n -2 | awk '{print $(NF)}'

access
arch_prctl
brk
close
connect
execve
<SNIP>
socket
write

You can also run the following simpler command and get a more verbose output.

   strace whoami

   execve("/usr/bin/whoami", ["whoami", "-qq"], [/* 21 vars */]) = 0
   brk(0)                                  = 0x1980000
   <SNIP>

You can substitute whoami for any other program.

In this step you started a new container with no seccomp profile and verified that the whoami program could execute. You also used the strace program to list the syscalls made by a particular run of the whoami program.

Step 4: Selectively remove syscalls

In this step you will see how applying changes to the default.json profile can be a good way to fine-tune which syscalls are available to containers.

The default-no-chmod.json profile is a modification of the default.json profile with the chmod(), fchmod(), and chmodat() syscalls removed from its whitelist.

Start a new container with the default-no-chmod.json profile and attempt to run the chmod 777 / -v command.

docker run --rm -it --security-opt seccomp=./seccomp-profiles/default-no-chmod.json alpine sh

and then from inside the container:

chmod 777 / -v

chmod: /: Operation not permitted

The command fails because the chmod 777 / -v command uses some of the chmod(), fchmod(), and chmodat() syscalls that have been removed from the whitelist of the default-no-chmod.json profile.

Exit the container.
```
exit
```

Start another new container with the default.json profile and run the same chmod 777 / -v.

docker run --rm -it --security-opt seccomp=./seccomp-profiles/default.json alpine sh

and then from inside the container:

chmod 777 / -v

mode of '/' changed to 0777 (rwxrwxrwx)

The command succeeds this time because the default.json profile has the chmod(), fchmod(), and chmodat syscalls included in its whitelist.

Exit the container.
```
exit
```
Check both profiles for the presence of the chmod(), fchmod(), and chmodat() syscalls.

Be sure to perform these commands from the command line of your Docker Host and not from inside of the container created in the previous step.
```
cat ./seccomp-profiles/default.json | grep chmod
```
```
"name": "chmod",
"name": "fchmod",
"name": "fchmodat",
```
```
cat ./seccomp-profiles/default-no-chmod.json | grep chmod
```
The output above shows that the default-no-chmod.json profile contains no chmod related syscalls in the whitelist.

In this step you saw how removing particular syscalls from the default.json profile can be a powerful way to start fine tuning the security of your containers.

Step 5: Write a seccomp profile

It is possible to write Docker seccomp profiles from scratch. You can also edit existing profiles. In this step you will learn about the syntax and behavior of Docker seccomp profiles.

The layout of a Docker seccomp profile looks like the following:

{
    "defaultAction": "SCMP_ACT_ERRNO",
    "architectures": [
        "SCMP_ARCH_X86_64",
        "SCMP_ARCH_X86",
        "SCMP_ARCH_X32"
    ],
    "syscalls": [
        {
            "name": "accept",
            "action": "SCMP_ACT_ALLOW",
            "args": []
        },
        {
            "name": "accept4",
            "action": "SCMP_ACT_ALLOW",
            "args": []
        },
        ...
    ]
}

The most authoritative source for how to write Docker seccomp profiles is the structs used to deserialize the JSON.

The table below lists the possible actions in order of precedence. Higher actions overrule lower actions.

Action	Description
SCMP_ACT_KILL	Kill with a exit status of `0x80 + 31 (SIGSYS) = 159`
SCMP_ACT_TRAP	Send a `SIGSYS` signal without executing the system call
SCMP_ACT_ERRNO	Set `errno` without executing the system call
SCMP_ACT_TRACE	Invoke a ptracer to make a decision or set `errno` to `-ENOSYS`
SCMP_ACT_ALLOW	Allow

The most important actions for Docker users are SCMP_ACT_ERRNO and SCMP_ACT_ALLOW.

Profiles can contain more granular filters based on the value of the arguments to the system call.

{
    ...
    "syscalls": [
        {
            "name": "accept",
            "action": "SCMP_ACT_ALLOW",
            "args": [
                {
                    "index": 0,
                    "op": "SCMP_CMP_MASKED_EQ",
                    "value": 2080505856,
                    "valueTwo": 0
                }
            ]
        }
    ]
}

index is the index of the system call argument
op is the operation to perform on the argument. It can be one of:
- SCMP_CMP_NE - not equal
- SCMP_CMP_LT - less than
- SCMP_CMP_LE - less than or equal to
- SCMP_CMP_EQ - equal to
- SCMP_CMP_GE - greater or equal to
- SCMP_CMP_GT - greater than
- SCMP_CMP_MASKED_EQ - masked equal: true if (value & arg == valueTwo)
value is a parameter for the operation
valueTwo is used only for SCMP_CMP_MASKED_EQ

The rule only matches if all args match. Add multiple rules to achieve the effect of an OR.

strace can be used to get a list of all system calls made by a program. It’s a very good starting point for writing seccomp policies. Here’s an example of how we can list all system calls made by ls:

strace -c -f -S name ls 2>&1 1>/dev/null | tail -n +3 | head -n -2 | awk '{print $(NF)}'

access
arch_prctl
brk
close
execve
<SNIP>
statfs
write

The output above shows the syscalls that will need to be enabled for a container running the ls program to work, in addition to the syscalls required to start a container.

In this step you learned the format and syntax of Docker seccomp profiles. You also learned the order of preference for actions, as well as how to determine the syscalls needed by an individual program.

Step 6: A few gotchas

The remainder of this lab will walk you through a few things that are easy to miss when using seccomp with Docker.

Timing of a seccomp profile application

In versions of Docker prior to 1.12, seccomp polices tended to be applied very early in the container creation process. This resulted in you needing to add syscalls to your profile that were required for the container creation process but not required by your container. This was not ideal. See:

A good way to avoid this issue in Docker 1.12+ can be to use the --security-opt no-new-privileges flag when starting your container. However, this will also prevent you from gaining privileges through setuid binaries.

Truncation

When writing a seccomp filter, there may be unused or randomly set bits on 32-bit arguments when using a 64-bit operating system after the filter has run.

When checking values from args against a blacklist, keep in mind that arguments are often silently truncated before being processed, but after the seccomp check. For example, this happens if the i386 ABI is used on an x86-64 kernel: although the kernel will normally not look beyond the 32 lowest bits of the arguments, the values of the full 64-bit registers will be present in the seccomp data. A less surprising example is that if the x86-64 ABI is used to perform a system call that takes an argument of type int, the more-significant half of the argument register is ignored by the system call, but visible in the seccomp data.

https://www.kernel.org/doc/Documentation/prctl/seccomp_filter.txt

seccomp escapes

Syscall numbers are architecture dependent. This limits the portability of BPF filters. Fortunately Docker profiles abstract this issue away, so you don’t need to worry about it if using Docker seccomp profiles.

ptrace is disabled by default and you should avoid enabling it. This is because it allows bypassing of seccomp. You can use this script to test for seccomp escapes through ptrace.

Differences between Docker versions

Seccomp is supported as of Docker 1.10.
Using the --privileged flag when creating a container with docker run disables seccomp in all versions of docker - even if you explicitly specify a seccomp profile. In general you should avoid using the --privileged flag as it does too many things. You can achieve the same goal with --cap-add ALL --security-opt apparmor=unconfined --security-opt seccomp=unconfined. If you need access to devices use -ice.
In docker 1.10-1.12 docker exec --privileged does not bypass seccomp. This may change in future versions (see https://github.com/docker/docker/issues/21984).
In docker 1.12 and later, adding a capability may enable some appropriate system calls in the default seccomp profile. However, it does not disable apparmor.

Using multiple filters

The only way to use multiple seccomp filters, as of Docker 1.12, is to load additional filters within your program at runtime. The kernel supports layering filters.

When using multiple layered filters, all filters are always executed starting with the most recently added. The highest precedence action returned is taken. See the man page for all the details: http://man7.org/linux/man-pages/man2/seccomp.2.html.

Misc

There is no easy way to use seccomp in a mode that reports errors without crashing the program. However, there are several round-about ways to accomplish this. One such way is to use SCMP_ACT_TRAP and write your code to handle SIGSYS and report the errors in a useful way. Here is some information on how Firefox handles seccomp violations.

Docker registry for Linux Part 1

Tue, 28 Feb 2017 00:00:00 +0000

Introduction

A registry is a service for storing and accessing Docker images. Docker Hub and Docker Store are the best-known hosted registries, which you can use to store public and private images. You can also run your own registry using the open-source Docker Registry, which is a Go application in a Alpine Linux container.

What You Will Learn

You’ll learn how to:

run a local registry in a container and configure your Docker engine to use the registry;
generate SSL certificates (using Docker!) and run a secure local registry with a friendly domain name;
generate encrypted passwords (using Docker!) and run an authenticated, secure local registry over HTTPS with basic auth.

Note. The open-source registry does not have a Web UI, so there’s no interface like Docker Hub or Docker Store. Instead there is a REST API you can use to query the registry. For a local registry which has a Web UI and role-based access control, Docker, Inc. has the Trusted Registry product.

You’ll need Docker running on in this tutorial, or on a Linux machine and be familiar with the key Docker concepts, and with Docker volumes:

Docker concepts
Docker volumes
Part 1 - Running a Registry Container in Linux

There are several ways to run a registry container. The simplest is to run an insecure registry over HTTP, but for that we need to configure Docker to explicitly allow insecure access to the registry.

Docker expects all registries to run on HTTPS. The next section of this lab will introduce a secure version of our registry container, but for this part of the tutorial we will run a version on HTTP. When registering a image, Docker returns an error message like this:

http: server gave HTTP response to HTTPS client

The Docker Engine needs to be explicitly setup to use HTTP for the insecure registry. For this sample it has already been done, 127.0.0.1:5000 has already been added to the daemon.

** Running on your own Linux machine instead of in this browser window ** Edit or create /etc/docker/docker file:

vi /etc/docker/docker

# add this line
DOCKER_OPTS="--insecure-registry 127.0.0.1:5000"

Close and save the file, then restart the docker daemon.

pkill dockerd
dockerd > /dev/null 2>&1 &

** If you’re running on your own Mac or Windows machine instead of in this browser window ** In Docker for Mac, the Preferences menu lets you set the address for an insecure registry under the Daemon panel:

In Docker for Windows, the Settings menu lets you set the address for an insecure registry under the Daemon panel:

Testing the Registry Image

First we’ll test that the registry image is working correctly, by running it without any special configuration:

docker run -d -p 5000:5000 --name registry registry:2

Understanding Image Names

Typically we work with images from Docker Store, which is the default registry for Docker. Commands using just the image repository name work fine, like this:

docker pull hello-world

hello-world is the repository name, which we are using as a short form of the full image name. The full name is docker.io/hello-world:latest. That breaks down into three parts:

docker.io - the hostname of the registry which stores the image;
hello-world - the repository name, in this case in {imageName} format;
latest - the image tag.

If a tag isn’t specified, then the default latest is used. If a registry hostname isn’t specified then the default docker.io for Docker Store is used. If you want to use images with any other registry, you need to explicitly specify the hostname - the default is always Docker Store.

With a local registry, the hostname and the custom port used by the registry is the full registry address, e.g. 127.0.0.1:5000. In this sample we’ll just be using 127.0.0.1:5000 as that’s already been added to the daemon.

Pushing and Pulling from the Local Registry

Docker uses the hostname from the full image name to determine which registry to use. We can build images and include the local registry hostname in the image tag, or use the docker tag command to add a new tag to an existing image.

These commands pull a public image from Docker Store, tag it for use in the private registry with the full name 127.0.0.1:5000/hello-world, and then push it to the registry:

docker tag hello-world 127.0.0.1:5000/hello-world
docker push 127.0.0.1:5000/hello-world

When you push the image to your local registry, you’ll see similar output to when you push a public image to the Hub:

The push refers to a repository [127.0.0.1:5000/hello-world]
a55ad2cda2bf: Pushed
cfbe7916c207: Pushed
fe4c16cbf7a4: Pushed
latest: digest: sha256:79e028398829da5ce98799e733bf04ac2ee39979b238e4b358e321ec549da5d6 size: 948

On your machine, you can remove the new image tag and the original image, and pull it again from the local registry to verify it was correctly stored:

docker rmi 127.0.0.1:5000/hello-world
docker rmi hello-world
docker pull 127.0.0.1:5000/hello-world

That exercise shows the registry works correctly, but at the moment it’s not very useful because all the image data is stored in the container’s writable storage area, which will be lost when the container is removed. To store the data outside of the container, we need to mount a host directory when we start the container.

Running a Registry Container with External Storage

Remove the existing registry container by removing the container which holds the storage layer. Any images pushed will be deleted:

docker kill registry
docker rm registry

In this example, the new container will use a host-mounted Docker volume. When the registry server in the container writes image layer data, it appears to be writing to a local directory in the container but it will be writing to a directory on the host.

Create the registry:

mkdir registry-data
docker run -d -p 5000:5000 --name registry -v $(pwd)/registry-data:/var/lib/registry registry

Tag and push the container with the new IP address of the registry.

docker pull hello-world
docker tag hello-world 127.0.0.1:5000/hello-world
docker push 127.0.0.1:5000/hello-world

Repeating the previous docker push command uploads an image to the registry container, and the layers will be stored in the container’s /var/lib/registry directory, which is actually mapped to the $(pwd)/registry-data directory on your machine. Storing data outside of the container means we can build a new version of the registry image and replace the old container with a new one using the same host mapping - so the new registry container has all the images stored by the previous container.

Using an insecure registry isn’t practical in multi-user scenarios. Effectively there’s no security so anyone can push and pull images if they know the registry hostname. The registry server supports authentication, but only over a secure SSL connection. We’ll run a secure version of the registry server in a container next.

Docker registry for Linux Parts 2 & 3

Mon, 27 Feb 2017 00:00:00 +0000

Part 2 - Running a Secured Registry Container in Linux

We saw how to run a simple registry container in Part 1, using the official Docker registry image. The registry server can be configured to serve HTTPS traffic on a known domain, so it’s straightforward to run a secure registry for private use with a self-signed SSL certificate.

Generating the SSL Certificate in Linux

The Docker docs explain how to generate a self-signed certificate on Linux using OpenSSL:

mkdir -p certs 
openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt

Generating a 4096 bit RSA private key
........++
............................................................++
writing new private key to 'certs/domain.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Docker
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:127.0.0.1
Email Address []:

If you are running the registry locally, be sure to use your host name as the CN.

To get the docker daemon to trust the certificate, copy the domain.crt file.

mkdir /etc/docker/certs.d
mkdir /etc/docker/certs.d/127.0.0.1:5000 
cp $(pwd)/certs/domain.crt /etc/docker/certs.d/127.0.0.1:5000/ca.crt

Make sure to restart the docker daemon.

pkill dockerd
dockerd > /dev/null 2>&1 &

The /dev/null part is to avoid the output logs from docker daemon.

Now we have an SSL certificate and can run a secure registry.

Running the Registry Securely

The registry server supports several configuration switches as environment variables, including the details for running securely. We can use the same image we’ve already used, but configured for HTTPS.

For the secure registry, we need to run a container which has the SSL certificate and key files available, which we’ll do with an additional volume mount (so we have one volume for registry data, and one for certs). We also need to specify the location of the certificate files, which we’ll do with environment variables:

mkdir registry-data
docker run -d -p 5000:5000 --name registry \
  --restart unless-stopped \
  -v $(pwd)/registry-data:/var/lib/registry -v $(pwd)/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  registry

The new parts to this command are:

--restart unless-stopped - restart the container when it exits, unless it has been explicitly stopped. When the host restarts, Docker will start the registry container, so it’s always available.
-v $pwd\certs:c:\certs - mount the local certs folder into the container, so the registry server can access the certificate and key files;
-e REGISTRY_HTTP_TLS_CERTIFICATE - specify the location of the SSL certificate file;
-e REGISTRY_HTTP_TLS_KEY - specify the location of the SSL key file.

We’ll let Docker assign a random IP address to this container, because we’ll be accessing it by host name. The registry is running securely now, but we’ve used a self-signed certificate for an internal domain name.

Accessing the Secure Registry

We’re ready to push an image into our secure registry.

docker pull hello-world
docker tag hello-world 127.0.0.1:5000/hello-world
docker push 127.0.0.1:5000/hello-world
docker pull 127.0.0.1:5000/hello-world

We can go one step further with the open-source registry server, and add basic authentication - so we can require users to securely log in to push and pull images.

** We have added Part 3 to the end of this section to allow you to continue to use the set-up we have above **

Part 3 - Using Basic Authentication with a Secured Registry in Linux

From Part 2 we have a registry running in a Docker container, which we can securely access over HTTPS from any machine in our network. We used a self-signed certificate, which has security implications, but you could buy an SSL from a CA instead, and use that for your registry. With secure communication in place, we can set up user authentication.

Usernames and Passwords

The registry server and the Docker client support basic authentication over HTTPS. The server uses a file with a collection of usernames and encrypted passwords. The file uses Apache’s htpasswd.

Create the password file with an entry for user “moby” with password “gordon”;

mkdir auth
docker run --entrypoint htpasswd registry:latest -Bbn moby gordon > auth/htpasswd

The options are:

–entrypoint Overwrite the default ENTRYPOINT of the image
-B Use bcrypt encryption (required)
-b run in batch mode
-n display results

We can verify the entries have been written by checking the file contents - which shows the user names in plain text and a cipher text password:

cat auth/htpasswd

moby:$2y$05$Geu2Z4LN0QDpUJBHvP5JVOsKOLH/XPoJBqISv1D8Aeh6LVGvjWWVC

Running an Authenticated Secure Registry

Adding authentication to the registry is a similar process to adding SSL - we need to run the registry with access to the htpasswd file on the host, and configure authentication using environment variables.

As before, we’ll remove the existing container and run a new one with authentication configured:

docker kill registry
docker rm registry
docker run -d -p 5000:5000 --name registry \
  --restart unless-stopped \
  -v $(pwd)/registry-data:/var/lib/registry \
  -v $(pwd)/certs:/certs \
  -v $(pwd)/auth:/auth \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  -e REGISTRY_AUTH=htpasswd \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e "REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd" \
  registry

The options for this container are:

-v $(pwd)/auth:/auth - mount the local auth folder into the container, so the registry server can access htpasswd file;
-e REGISTRY_AUTH=htpasswd - use the registry’s htpasswd authentication method;
-e REGISTRY_AUTH_HTPASSWD_REALM='Registry Realm' - specify the authentication realm;
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd - specify the location of the htpasswd file.

Now the registry is using secure transport and user authentication.

Authenticating with the Registry

With basic authentication, users cannot push or pull from the registry unless they are authenticated. If you try and pull an image without authenticating, you will get an error:

docker pull 127.0.0.1:5000/hello-world

Using default tag: latest
Error response from daemon: Get https://127.0.0.1:5000/v2/hello-world/manifests/latest: no basic auth credentials

The result is the same for valid and invalid image names, so you can’t even check a repository exists without authenticating. Logging in to the registry is the same docker login command you use for Docker Store, specifying the registry hostname:

docker login 127.0.0.1:5000

Username: moby
Password:
Login Succeeded

If you use the wrong password or a username that doesn’t exist, you get a 401 error message:

Error response from daemon: login attempt to https://registry.local:5000/v2/ failed with status: 401 Unauthorized

Now you’re authenticated, you can push and pull as before:

docker pull 127.0.0.1:5000/hello-world

Using default tag: latest
latest: Pulling from hello-world
Digest: sha256:961497c5ca49dc217a6275d4d64b5e4681dd3b2712d94974b8ce4762675720b4
Status: Image is up to date for registry.local:5000/hello-world:latest

Note. The open-source registry does not support the same authorization model as Docker Store or Docker Trusted Registry. Once you are logged in to the registry, you can push and pull from any repository, there is no restriction to limit specific users to specific repositories.

Conclusion

Docker Registry is a free, open-source application for storing and accessing Docker images. You can run the registry in a container on your own network, or in a virtual network in the cloud, to host private images with secure access. For Linux hosts, there is an official registry image on Docker Hub.

We’ve covered all the options, from running an insecure registry, through adding SSL to encrypt traffic, and finally adding basic authentication to restrict access. By now you know how to set up a usable registry in your own environment, and you’ve also used some key Docker patterns - using containers as build agents and to run basic commands, without having to install software on your host machines.

There is still more you can do with Docker Registry - using a different storage driver so the image data is saved to reliable share storage, and setting up your registry as a caching proxy for Docker Store are good next steps.

Go + Docker = ♥

Thu, 23 Feb 2017 00:00:00 +0000

This is a short collection of tips and tricks showing how Docker can be useful when working with Go code. For instance, I’ll show you how to compile Go code with different versions of the Go toolchain, how to cross-compile to a different platform (and test the result!), or how to produce really small container images.

The following article assumes that you have Docker installed on your system. It doesn’t have to be a recent version (we’re not going to use any fancy feature here).

Go without `go`

… And by that, we mean “Go without installing go”.

If you write Go code, or if you have even the slightest interest into the Go language, you certainly have the Go compiler and toolchain installed, so you might be wondering “what’s the point?”; but there are a few scenarios where you want to compile Go without installing Go.

You still have this old Go 1.2 on your machine (that you can’t or won’t upgrade), and you have to work on this codebase that requires a newer version of the toolchain.
You want to play with cross compilation features of Go 1.5 (for instance, to make sure that you can create OS X binaries from a Linux system).
You want to have multiple versions of Go side-by-side, but don’t want to completely litter your system.
You want to be 100% sure that your project and all its dependencies download, build, and run fine on a clean system.

If any of this is relevant to you, then let’s call Docker to the rescue!

Compiling a program in a container

When you have installed Go, you can do go get -v github.com/user/repo to download, build, and install a library. (The -v flag is just here for verbosity, you can remove it if you prefer your toolchain to be swift and silent!)

You can also do go get github.com/user/repo/... (yes, that’s three dots) to download, build, and install all the things in that repo (including libraries and binaries).

We can do that in a container!

Try this:

docker container run golang go get -v github.com/golang/example/hello/...

This will pull the golang image (unless you have it already; then it will start right away), and create a container based on that image. In that container, go will download a little “hello world” example, build it, and install it. But it will install it in the container … So how do we run that program now?

Running our program in a container

One solution is to commit the container that we just built, i.e. “freeze” it into a new image:

docker container commit $(docker container ls -lq) awesomeness

Note: docker container ls -lq outputs the ID (and only the ID!) of the last container that was executed. If you are the only user on your machine, and if you haven’t created another container since the previous command, that container should be the one in which we just built the “hello world” example.

Now, we can run our program in a container based on the image that we just built:

docker container run awesomeness hello

The output should be Hello, Go examples!.

Bonus points

When creating the image with docker container commit, you can use the --change flag to specify arbitrary Dockerfile commands. For instance, you could use a CMD or ENTRYPOINT command so that docker run awesomeness automatically executes hello.

Running in a throwaway container

What if we don’t want to create an extra image just to run this Go program?

We got you covered:

docker container run --rm golang sh -c \
    "go get github.com/golang/example/hello/... && exec hello"

Wait a minute, what are all those bells and whistles?

--rm tells to the Docker CLI to automatically issue a docker container rm command once the container exits. That way, we don’t leave anything behind ourselves.
We chain together the build step (go get) and the execution step (exec hello) using the shell logical operator &&. If you’re not a shell aficionado, && means “and”. It will run the first part go get..., and if (and only if!) that part is successful, it will run the second part (exec hello). If you wonder why this is like that: it works like a lazy and evaluator, which needs to evaluate the right hand side only if the left hand side evaluates to true.
We pass our commands to sh -c, because if we were to simply do docker container run golang "go get ... && hello", Docker would try to execute the program named go SPACE get SPACE etc. and that wouldn’t work. So instead, we start a shell and instruct the shell to execute the command sequence.
We use exec hello instead of hello: this will replace the current process (the shell that we started) with the hello program. This ensures that hello will be PID 1 in the container, instead of having the shell as PID 1 and hello as a child process. This is totally useless for this tiny example, but when we will run more useful programs, this will allow them to receive external signals properly, since external signals are delivered to PID 1 of the container. What kind of signal, you might be wondering? A good example is docker container stop, which sends SIGTERM to PID 1 in the container.

Using a different version of Go

When you use the golang image, Docker expands that to golang:latest, which (as you might guess) will map to the latest version available on the Docker Hub.

If you want to use a specific version of Go, that’s very easy: specify that version as a tag after the image name.

For instance, to use Go 1.5, change the example above to replace golang with golang:1.5:

docker container run --rm golang:1.5 sh -c \
    "go get github.com/golang/example/hello/... && exec hello"

You can see all the versions (and variants) available on the Golang image page on the Docker Hub.

Installing on our system

OK, so what if we want to run the compiled program on our system, instead of in a container?

We could copy the compiled binary out of the container. Note, however, that this will work only if our container architecture matches our host architecture; in other words, if we run Docker on Linux. (I’m leaving out people who might be running Windows Containers!)

The easiest way to get the binary out of the container is to map the $GOPATH/bin directory to a local directory. In the golang container, $GOPATH is /go. So we can do the following:

docker container run -v /tmp/bin:/go/bin \
  golang go get github.com/golang/example/hello/...
/tmp/bin/hello

If you are on Linux, you should see the Hello, Go examples! message. But if you are, for instance, on a Mac, you will probably see:

/tmp/test/hello: cannot execute binary file

What can we do about it?

Cross-compilation

Go 1.5 comes with outstanding out-of-the-box cross-compilation abilities, so if your container operating system and/or architecture doesn’t match your system’s, it’s no problem at all!

To enable cross-compilation, you need to set GOOS and/or GOARCH.

For instance, assuming that you are on a 64 bits Mac:

docker container run -e GOOS=darwin -e GOARCH=amd64 -v /tmp/crosstest:/go/bin \
  golang go get github.com/golang/example/hello/...

The output of cross-compilation is not directly in $GOPATH/bin, but in $GOPATH/bin/$GOOS_$GOARCH. In other words, to run the program, you have to execute /tmp/crosstest/darwin_amd64/hello.

Installing straight to the `$PATH`

If you are on Linux, you can even install directly to your system bin directories:

docker container run -v /usr/local/bin:/go/bin \
  golang go get github.com/golang/example/hello/...

However, on a Mac, trying to use /usr as a volume will not mount your Mac’s filesystem to the container. It will mount the /usr directory of the Moby VM (the small Linux VM hidden behind the Docker whale icon in your toolbar).

You can, however, use /tmp or something in your home directory, and then copy it from there.

Building lean images

The Go binaries that we produced with this technique are statically linked. This means that they embed all the code that they need to run, including all dependencies. This contrasts with dynamically linked programs, which don’t contain some basic libraries (like the “libc”) and use a system-wide copy which is resolved at run time.

This means that we can drop our Go compiled program in a container, without anything else, and it should work.

Let’s try this!

The `scratch` image

There is a special image in the Docker ecosystem: scratch. This is an empty image. It doesn’t need to be created or downloaded, since by definition, it is empty.

Let’s create a new, empty directory for our new Go lean image.

In this new directory, create the following Dockerfile:

FROM scratch
COPY ./hello /hello
ENTRYPOINT ["/hello"]

This means:

start from scratch (an empty image),
add the hello file to the root of the image,
define this hello program to be the default thing to execute when starting this container.

Then, produce our hello binary as follows:

docker container run -v $(pwd):/go/bin --rm \
  golang go get github.com/golang/example/hello/...

Note: we don’t need to set GOOS and GOARCH here, because precisely, we want a binary that will run in a Docker container, not on our host system. So leave those variables alone!

Then, we can build the image:

docker image build -t hello .

And test it:

docker container run hello

(This should display Hello, Go examples!.)

Last but not least, check the image’s size:

docker image ls hello

If we did everything right, this image should be about 2 MB. Not bad!

Building something without pushing to GitHub

Of course, if we had to push to GitHub each time we wanted to compile, we would waste a lot of time.

When you want to work on a piece of code and build it within a container, you can mount a local directory to /go in the golang container, so that the $GOPATH is persisted across invocations: docker run -v $HOME/go:/go golang ....

But you can also mount local directories to specific paths, to “override” some packages (the ones that you have edited locally). Here is a complete example:

# Adapt the two following environment variables if you are not running on a Mac
export GOOS=darwin GOARCH=amd64
mkdir go-and-docker-is-love
cd go-and-docker-is-love
git clone git://github.com/golang/example
cat example/hello/hello.go
sed -i .bak s/olleH/eyB/ example/hello/hello.go
docker container run --rm \
  -v $(pwd)/example:/go/src/github.com/golang/example \
  -v $(pwd):/go/bin/${GOOS}_${GOARCH} \
  -e GOOS -e GOARCH \
  golang go get github.com/golang/example/hello/...
./hello
# Should display "Bye, Go examples!"

The special case of the `net` package and CGo

Before diving into real-world Go code, we have to confess something: we lied a little bit about the static binaries. If you are using CGo, or if you are using the net package, the Go linker will generate a dynamic binary. In the case of the net package (which a lot of useful Go programs out there are using indeed!), the main culprit is the DNS resolver. Most systems out there have a fancy, modular name resolution system (like the Name Service Switch) which relies on plugins which are, technically, dynamic libraries. By default, Go will try to use that; and to do so, it will produce dynamic libraries.

How do we work around that?

Re-using another distro’s libc

One solution is to use a base image that has the essential libraries needed by those Go programs to function. Almost any “regular” Linux distro based on the GNU libc will do the trick. So instead of FROM scratch, you would use FROM debian or FROM fedora, for instance. The resulting image will be much bigger now; but at least, the bigger bits will be shared with other images on your system.

Note: you cannot use Alpine in that case, since Alpine is using the musl library instead of the GNU libc.

Bring your own libc

Another solution is to surgically extract the files needed, and place them in your container with COPY. The resulting container will be small. However, this extraction process leaves the author with the uneasy impression of a really dirty job, and they would rather not go into more details.

If you want to see for yourself, look around ldd and the Name Service Switch plugins mentioned earlier.

Producing static binaries with `netgo`

We can also instruct Go to not use the system’s libc, and substitute Go’s netgo library, which comes with a native DNS resolver.

To use it, just add -tags netgo -installsuffix netgo to the go get options.

-tags netgo instructs the toolchain to use netgo.
-installsuffix netgo will make sure that the resulting libraries (if any) are placed in a different, non-default directory. This will avoid conflicts between code built with and without netgo, if you do multiple go get (or go build) invocations. If you build in containers like we have shown so far, this is not strictly necessary, since there will be no other Go code compiled in this container, ever; but it’s a good idea to get used to it, or at least know that this flag exists.

The special case of SSL certificates

There is one more thing that you have to worry about if your code has to validate SSL certificates; for instance if it will connect to external APIs over HTTPS. In that case, you need to put the root certificates in your container too, because Go won’t bundle those into your binary.

Installing the SSL certificates

Three again, there are multiple options available, but the easiest one is to use a package from an existing distribution.

Alpine is a good candidate here because it’s so tiny. The following Dockerfile will give you a base image that is small, but has an up-to-date bundle of root certificates:

FROM alpine:3.4
RUN apk add --no-cache ca-certificates apache2-utils

Check it out; the resulting image is only 6 MB!

Note: the --no-cache option tells apk (the Alpine package manager) to get the list of available packages from Alpine’s distribution mirrors, without saving it to disk. You might have seen Dockerfiles doing something like apt-get update && apt-get install ... && rm -rf /var/cache/apt/*; this achieves something equivalent (i.e. not leave package caches in the final image) with a single flag.

As an added bonus, putting your application in a container based on the Alpine image gives you access to a ton of really useful tools: now you can drop a shell into your container and poke around while it’s running, if you need to!

Wrapping it up

We saw how Docker can help us to compile Go code in a clean, isolated environment; how to use different versions of the Go toolchain; and how to cross-compile between different operating systems and platforms.

We also saw how Go can help us to build small, lean container images for Docker, and described a number of associated subtleties linked (no pun intended) to static libraries and network dependencies.

Beyond the fact that Go is really good fit for a project that Docker, we hope that we showed you how Go and Docker can benefit from each other and work really well together!

Acknowledgements

This was initially presented during the hack day at GopherCon 2016.

I would like to thank all the people who proofread this material and gave ideas and suggestions to make it better; including but not limited to:

Aaron Lehmann
Stephen Day
AJ Bowen

All mistakes and typos are my own; all the good stuff is theirs! ☺

Live Debugging Node.js with Docker

Wed, 22 Feb 2017 00:00:00 +0000

Note: This tutorial requires you to run your app locally on your own computer

Pre-requisites

Getting Started

The first thing to notice is that you don’t actually need to have Node.js installed on your machine. You can just use Docker and your IDE. In this case we’re going to show you how to use Visual Studio Code.

We’ve created a simple application which includes an error. You can see the app in the app/ directory of this repository. You can either clone this repository, or create the files yourself. Make sure they’re all in the same directory. You will need the following files:

app.js
package.json
index.html
Dockerfile
docker-compose.yml

app.js defines a simple node app. It serves up index.html, refreshing every 2 seconds with quote from an array. Here’s what it looks like:

Let’s take a look at the Dockerfile:

FROM node:5.11.0-slim

WORKDIR /code

RUN npm install -g nodemon

COPY package.json /code/package.json
RUN npm install && npm ls
RUN mv /code/node_modules /node_modules

COPY . /code

CMD ["npm", "start"]

As you can see it installs nodemon, a utility that will monitor for any changes in your source and automatically restart your server.

You’ll start the app with the docker-compose.yml

version: "3"

services:
  web:
    build: .
    command: nodemon --debug=5858
    volumes:
      - .:/code
    ports:
      - "8000:8000"
      - "5858:5858"

A few things are going on here:

It defines a service called “web”, which uses the image built from the Dockerfile in the current directory.
It overrides the command specified in the Dockerfile to enable the remote debugging feature built into Node.js. We do that here because when you ship this application’s container image to production, you don’t want the debugger enabled – it’s a development-only override.
It overwrites the application code in the container by mounting the current directory as a volume. This means that the code inside the running container will update whenever you update the local files on your hard drive. This is very useful, as it means you don’t have to rebuild the image every time you make a change to the application.
It maps port 8000 inside the container to port 8000 on localhost, so you can actually visit the application.
Finally, it maps port 5858 inside the container to the same port on localhost, so you can connect to the remote debugger.

Run the app

Using your terminal, navigate to the directory with the app files and start up the app:

$ docker-compose up

Docker Compose will build the image and start a container for the app. You should see this output:

Creating network "nodeexample_default" with the default driver
Creating nodeexample_web_1
Attaching to nodeexample_web_1
web_1  | [nodemon] 1.9.2
web_1  | [nodemon] to restart at any time, enter `rs`
web_1  | [nodemon] watching: *.*
web_1  | [nodemon] starting `node --debug=5858 app.js`
web_1  | Debugger listening on port 5858
web_1  | HTTP server listening on port 80

The app is now running. Open up http://localhost:8000/ to see it in action, and take a moment to appreciate the poetry.

It’s undoubtedly beautiful, but the problem is obvious: we’re outputting a blank message at the end before cycling back to the first line. It’s time to debug.

Start debugging

Open up the app directory in VSCode. Head over to the debugger by clicking the bug icon in the left-hand sidebar.

Create a boilerplate debugger config by clicking the gear icon and selecting “Node.js” in the dropdown.

A JSON file will be created and displayed. Replace its contents with the following:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Attach",
            "type": "node",
            "request": "attach",
            "port": 5858,
            "address": "localhost",
            "restart": true,
            "sourceMaps": false,
            "outDir": null,
            "localRoot": "${workspaceRoot}",
            "remoteRoot": "/code"
        }
    ]
}

There are three important changes here:

The whole “Launch” config has been deleted – you’re using Compose to launch the app, not VSCode, so it’s unnecessary.
restart is set to true, so that the debugger re-attaches when the app restarts.
remoteRoot is set to the path of the code directory inside the container, because it’s almost certainly different than the path to the code on your machine.

With the “Attach” config selected, click the “play” button to start the debugger.

Now go back to app.js and find the line that reads lineIndex += 1 line, just after we initialize the message variable. Set a breakpoint by clicking in the gutter, just to the left of the line number.

If your browser window is still open and refreshing, in a second or two you should see it hit the breakpoint. If not, go back and refresh it – VSCode will pop back to the front as soon as the debugger hits it.

Hit the Play button at the top to resume code execution. It’ll hit the breakpoint every time the browser refreshes, which is every 2 seconds. You can see it cycling through the lines, and then the bug shows up – right after the last line, message gets set to undefined.

The reason becomes clear if you open up the “Closure” section under “VARIABLES”: lineIndex has incremented to 4 – the length of the LINES array – when it should have been reset after getting to 3. We’ve got an off-by-one error.

Fix the bug

Replace the > with >= in the conditional on the next line:

Now save the file. A second or two later, you should see the debugger detach and then reattach (the yellow line highlighting the breakpoint will disappear and reappear). This is because several things have just happened:

Upon saving the file, Docker detected the filesystem change event and proxied it through to the container.
nodemon detected the event and restarted the application. You can confirm this by looking at your terminal: there should be a line that reads “restarting due to changes…”
Finally, VSCode detected that the remote debugger had gone away and reattached.

The debugger is now attached again. However, your browser tab might have errored out – go refresh it if so.

You can now step through the debugger once again and see that the lines cycle properly – no more undefined.

Remove the breakpoint and detach the debugger by clicking the stop button. Go back to the browser window and enjoy the updated experience.

And that’s it, you’re done!

True or false: You have to restart a container after you make changes to the code or they won’t be reflected in the application

( ) True
(x) False

True or false: Debugging a NodeJS app running in a container requires a special plugin for the IDE

( ) True
(x) False

In-container Java Development: Netbeans

Wed, 22 Feb 2017 00:00:00 +0000

Note: This tutorial requires you to run your app locally on your own computer

Pre-requisites

Getting Started

Using your git client clone the repository.

git clone https://github.com/docker/labs
cd labs/developer-tools/java-debugging

Open NetBeans IDE, Click on Open Project...

Select app and click on Open Project.

Building the application

The application is a basic Spring MVC application that receives user input from a form, writes the data to a database, and queries the database.

The application is built using Maven. To build the application click on Run > Build Project.

The results of the build will be displayed in the console.

Running the application

Open a terminal and go to the application directory. Start the application with docker-compose

> docker-compose up

Docker will build the images for Apache Tomcat and MySQL and start the containers. It will also mount the application directory (./app/target/UserSignup) as a data volume on the host system to the Tomcat webapps directory in the web server container.

Open a browser window and go to: ‘localhost:8080’; you should see the Tomcat home page

When the Tomcat image was built, the user roles were also configured. Click on the Manager App button to see the deployed applications. When prompted for username and password, enter system and manager respectively to log into the Tomcat Web Application Manager page.

You can use the Manager page to Start, Stop, Reload or Undeploy web applications.

To go to the application, Click on /UserSignup link.

Debugging the Application

In the application, click on Signup to create a new user. Fill out the registration form and click Submit

Click Yes to confirm.

Test out the login.

Oh no!

Configure Remote Debugging

In the menu click on Debug > Attach Debugger...

Make sure that the port is set to 8000, click on OK.

Finding the Error

Since the problem is with the password, lets see how the password is set in the User class. In the User class, the setter for password is scrambled using rot13 before it is saved to the database.

Since we enabled remote debugging earlier, you should see the Daemon Threads for Tomcat in the Debugging window. Set a breakpoint for in the User class where the password is set.

NetBeans will display the code at the breakpoint and the value of password in the variables window. Note that the value is m0by

Click on Continue icon or press F5 to let the code run.

Next, set a breakpoint on the getPassword in the User class to see the value returned for password. You can also toggle off the breakpoint for setPassword. Try to log into the application. Look at the value for password in the NetBeans variables window, note that it is z0ol which is m0by using ROT13.

In this MVC application the UserController uses the findByLogin method in the UserServiceImpl class which uses the findByUsername method to retrieve the information from the database. It then checks to see if the password from the form matches the user password. Since the password from the login form is not scrambled using ROT13, it does not match the user password and you cannot log into the application.

To fix this, apply ROT13 to the password by adding

import com.docker.UserSignup.utit.Rot13

String passwd = Rot13.rot13(password);

Set a breakpoint in UserServiceImpl on the findByLogin method. Press F11 or click on Run > Build Project to update the deployed code. Log in again and look at the values for the breakpoint. The ‘passwd’ variable is z0ol which matches the password for the user moby.

Continue (F5) and you should successfully log in.

True or false: You have to restart a container after you make changes to the code or they won’t be reflected in the application

( ) True
(x) False

True or false: Debugging a Java app running in a container requires a special plugin for the IDE

( ) True
(x) False

In-container Java Development: Intellij

Wed, 22 Feb 2017 00:00:00 +0000

Note: This tutorial requires you to run your app locally on your own computer

Pre-requisites

Getting Started

In IntelliJ, clone the repository. Click on Check out from Version Control > Github

If this the first time to use IntelliJ with Github, log into your Github account.

On the command line clone the docker/labs repository

Click on Import project from external model, select Maven. Click Next

Check Search for projects recursively. Click Next

Select the project and click Next

Select the JDK(set the JDK home path) and click Next

Click Finish

Click on Project View to open the project.

Building the application

The application is a basic Spring MVC application that receives user input from a form, writes the data to a database, and queries the database.

The application is built using Maven. To build the application click on icon on the bottom left of the IntelliJ window and select Maven Projects.

The Maven Projects window will open on the right side. Maven goals of clean and install need to be set to build the application.

To set the clean goal, click on Lifecycle to display the tree of goals. Right click on clean and select Create 'UserSignup [clean]'...

Click OK in the Create Run/Debug Configuration window.

Configure the install goal similarly. Click on install in the Lifecycle tree. Select Create 'UserSignup[install]'...

Click OK in the Create Run/Debug Configuration window.

To build the application run clean

Then run install

When the application builds, you will see a success message in the log window.

Running the application

Open a terminal and go to the application directory. Start the application with docker-compose

> docker-compose up

Docker will build the images for Apache Tomcat and MySQL then start the containers. It will also mount the application directory (./app/target/UserSignup) as a data volume on the host system to the Tomcat webapps directory in the web server container.

Open a browser window and go to: ‘localhost:8080’; you should see the Tomcat home page

You can use the Manager page to Start, Stop, Reload or Undeploy web applications.

To go to the application, Click on /UserSignup link.

Debugging the Application

In the application, click on Signup to create a new user. Fill out the registration form and click Submit

Click Yes to confirm.

Test out the login.

Oh no!

Configure Remote Debugging

Tomcat supports remote debugging the Java Platform Debugger Architecture (JPDA). Remote debugging was enabled when the tomcat image (registration-webserver) was built.

To configure remote debugging in IntelliJ, click on Run > Edit Configuration ...

Add a new remote configuration.

In the Run\Debug Configurations window, set the Name of the configuration as docker tomcat and in Settings set the port to ‘8000’ as the default Tomcat JPDA debuging port. Click on OK to save the configuration.

Finding the Error

Since the problem is with the password, let’s see how the password is set in the User class. In the User class, the setter for password is scrambled using rot13 before it is saved to the database.

Try registering a new user using the debugger. In the menu click on Run > Debug...

Choose the remote Tomcat debug configuration. The Debugger console will be displayed at the bottom of the IntelliJ window.

Set a breakpoint in the User class where the password is set.

IntelliJ will display the code at the breakpoint and the value of password in the variables window. Note that the value is m0by

Click on Resume Program to let the code run or press F8 to step over the breakpoint.

Next, set a breakpoint on the getPassword in the User class to see the value returned for password. You can also toggle off the breakpoint for setPassword.

Try to log into the application. Look at the value for password in the debugging console, note that it is z0ol which is m0by using ROT13.

To fix this, apply ROT13 to the password by adding

import com.docker.UserSignup.utit.Rot13

String passwd = Rot13.rot13(password);

Set a breakpoint in UserServiceImpl on the findByLogin method. Log in again and look at the values for the breakpoint. The ‘passwd’ variable is z0ol which matches the password for the user moby.

Continue (F8) and you should successfully log in.

True or false: You have to restart a container after you make changes to the code or they won’t be reflected in the application

( ) True
(x) False

True or false: Debugging a Java app running in a container requires a special plugin for the IDE

( ) True
(x) False

In-container Java Development: Eclipse

Wed, 22 Feb 2017 00:00:00 +0000

Note: This tutorial requires you to run your app locally on your own computer

Pre-requisites

Docker for OSX or Docker for Windows
Eclipse (install Eclipse IDE for Java EE Developers)
Java Development Kit
Maven for Eclipse (see instructions for adding the Maven plug-in to Eclipse)

Getting Started

On the command line clone the registration-docker repository

git clone https://github.com/docker/labs
cd labs/developer-tools/java-debugging

In Eclipse, import the app directory of that project as an existing maven project

File> Import Select Maven> Existing Maven Projects> Next

Select the app subdirectory of the directory where you cloned the project.

Select the pom.xml from the app directory, click Finish.

Building the application

The application is a basic Spring MVC application that receives user input from a form, writes the data to a database, and queries the database.

The application is built using Maven. To build the application click on Run > Run configurations

Select Maven build > New

Enter a Name for the configuration.

Set the base direct of the application <path>/registration-docker/app.

Set the Goals to clean install.

Click Apply

Click Run

The results of the build will be displayed in the console.

Running the application

Open a terminal and go to the application directory. Start the application with docker-compose

docker-compose up

Open a browser window and go to: ‘localhost:8080’; you should see the Tomcat home page

You can use the Manager page to Start, Stop, Reload or Undeploy web applications.

To go to the application, Click on /UserSignup link.

Debugging the Application

In the application, click on Signup to create a new user. Fill out the registration form and click Submit

Click Yes to confirm.

Test out the login.

Oh no!

Configure Remote Debugging

Tomcat supports remote debugging the Java Platform Debugger Architecture (JPDA). Remote debugging was enabled when the tomcat image (registration-webserver) was built.

To configure remote debugging in Eclipse, click on Run > Debug Configurations ...

Select Remote Java Application and click on Launch New Configuration icon

Enter a Name for the configuration. Select the project using the browse button. Click on Apply to save the configuration and click on Debug to start the debugging connection between Tomcat and Eclipse.

Finding the Error

Since the problem is with the password, lets see how the password is set in the User class. In the User class, the setter for password is scrambled using rot13 before it is saved to the database.

Try registering a new user using the debugger. In Eclipse, change the view or Perspective to the debugger by clicking on Window > Perspective > Open Perspective > Debug

Eclipse will switch to the debug perspective. Since we enable remote debugging earlier, you should see the Daemon Threads for Tomcat in the debug window. Set a breakpoint for in the User class where the password is set.

Eclipse will display the code at the breakpoint and the value of password in the variables window. Note that the value is m0by

Click on resume or press F8 to let the code run.

Next, set a breakpoint on the getPassword in the User class to see the value returned for password. You can also toggle off the breakpoint for setPassword.

Try to log into the application. Look at the value for password in the Eclipse variables window, note that it is z0ol which is m0by using ROT13.

To fix this, apply ROT13 to the password by adding

import com.docker.UserSignup.utit.Rot13

String passwd = Rot13.rot13(password);

Continue (F8) and you should successfully log in.

True or false: You have to restart a container after you make changes to the code or they won’t be reflected in the application

( ) True
(x) False

True or false: Debugging a Java app running in a container requires a special plugin for the IDE

( ) True
(x) False

Swarm stack introduction

Sat, 28 Jan 2017 00:00:00 +0000

Let’s deploy the voting app stack on a swarm.

Purpose

The purpose of this lab is to illustrate how to deploy a stack (multi services application) against a Swarm using a docker compose file.

The application

The voting app is a very handy multi containers application often used for demo purposes during meetup and conferences.

It basically allow users to vote between cat and dog (but could be “space” or “tab” too if you feel like it).

This application is available on Github and updated very frequently when new features are developed.

Init your swarm

There should be two terminals displayed. The first accesses the swarm manager node and the second accesses the swarm worker node once the swarm is created.

Let’s create a Docker Swarm first. Copy the below command into the first terminal.

docker swarm init --advertise-addr $(hostname -i)

From the output above, copy the join command (watch out for newlines) and paste it in the other terminal.

Show members of swarm

From the first terminal, check the number of nodes in the swarm (running this command from the second terminal worker will fail as swarm related commands need to be issued against a swarm manager).

docker node ls

The above command should output 2 nodes, the first one being the manager, and the second one a worker.

Clone the voting-app

Let’s retrieve the voting app code from Github and go into the application folder.

Ensure you are in the first terminal and do the below:

git clone https://github.com/docker/example-voting-app
cd example-voting-app

Deploy a stack

A stack is a group of services that are deployed together. The docker-stack.yml in the current folder will be used to deploy the voting app as a stack.

Ensure you are in the first terminal and do the below:

docker stack deploy --compose-file=docker-stack.yml voting_stack

Note: being able to create a stack from a docker compose file is a great feature added in Docker 1.13.

Check the stack deployed from the first terminal

docker stack ls

The output should be the following one. It indicates the 6 services of the voting app’s stack (named voting_stack) have been deployed.

NAME          SERVICES
voting_stack  6

Let’s check the service within the stack

docker stack services voting_stack

The output should be like the following one (your ID should be different though).

ID            NAME                     MODE        REPLICAS  IMAGE
10rt1wczotze  voting_stack_visualizer  replicated  1/1       dockersample
s/visualizer:stable
8lqj31k3q5ek  voting_stack_redis       replicated  2/2       redis:alpine
nhb4igkkyg4y  voting_stack_result      replicated  2/2       dockersample
s/examplevotingapp_result:before
nv8d2z2qhlx4  voting_stack_db          replicated  1/1       postgres:9.4
ou47zdyf6cd0  voting_stack_vote        replicated  2/2       dockersample
s/examplevotingapp_vote:before
rpnxwmoipagq  voting_stack_worker      replicated  1/1       dockersample
s/examplevotingapp_worker:latest

Let’s list the tasks of the vote service.

docker service ps voting_stack_vote

You should get an output like the following one where the 2 tasks (replicas) of the service are listed.

ID            NAME                 IMAGE
      NODE   DESIRED STATE  CURRENT STATE           ERROR  PORTS
my7jqgze7pgg  voting_stack_vote.1  dockersamples/examplevotingapp_vote:be
fore  node1  Running        Running 56 seconds ago
3jzgk39dyr6d  voting_stack_vote.2  dockersamples/examplevotingapp_vote:be
fore  node2  Running        Running 58 seconds ago

From the NODE column, we can see one task is running on each node.

Finally, we can check that our APP is running, the RESULT page is also available as well as SWARM VISUALIZER

Conclusion

Using only a couple of commands enables to deploy a stack of services on a Docker Swarm using the really great Docker Compose file format.

What is a stack?

(x) a multi-service app running on a Swarm
( ) a way of creating multiple nodes
( ) a method of using multiple compose files to run an app

A stack can:

be deployed from the commandline
can use the compose file format to deploy
can run a Dockerfile
be used to manage your hosts
can be used to manage services over multiple nodes

Docker images deeper dive

Fri, 27 Jan 2017 00:00:00 +0000

Let’s have fun with Docker images

In this lab we will see how to create an image from a container. Even if this is not used very often it’s interesting to try it at least once. Then we will focus on the image creation using a Dockerfile. We will then see how to get the details of an image through the inspection and explore the filesystem to have a better understanding of what happens behind the hood. We will end this lab with the image API.

Image creation from a container

Let’s start by running an interactive shell in a ubuntu container.

docker container run -ti ubuntu bash

As we’ve done in the previous lab, we will install the figlet package in this container.

apt-get update
apt-get install -y figlet

We then exit from this container

exit

Get the ID of this container using the ls command (do not forget the -a option as the non running container are not returned by the ls command).

docker container ls -a

Run the following command, using the ID retreived, in order to commit the container and create an image out of it.

docker container commit CONTAINER_ID

Once it has been commited, we can see the newly created image in the list of available images.

docker image ls

From the previous command, get the ID of the newly created image and tag it so it’s named ourfiglet.

docker image tag IMAGE_ID ourfiglet

Now we will run a container based on the newly created image named ourfiglet, and specify the command to be ran such as it uses the figlet package.

docker container run ourfiglet figlet hello

As figlet is present in our ourfiglet image, the command ran returns the following output.

 _          _ _
| |__   ___| | | ___
| '_ \ / _ \ | |/ _ \
| | | |  __/ | | (_) |
|_| |_|\___|_|_|\___/

This example shows that we can create a container, add all the libraries and binaries in it and then commit this one in order to create an image. We can then use that image as we would do for any other images. This approach is not the recommended one as it is not very portable.

In the following we will see how images are usually created, using a Dockerfile, which is a text file that contains all the instructions to build an image.

Image creation using a Dockerfile

We will use a simple example in this section and build a hello world application in Node.js. We will start by creating a file in which we retrieve the hostname and display it.

Copy the following content into index.js file.

var os = require("os");
var hostname = os.hostname();
console.log("hello from " + hostname);

We will dockerrize this application and start by creating a Dockerfile for this purpose. We will use alpine as the base image, add a Node.js runtime and then copy our source code. We will also specify the default command to be ran upon container creation.

Create a file named Dockerfile and copy the following content into it.

FROM alpine
RUN apk update && apk add nodejs
COPY . /app
WORKDIR /app
CMD ["node","index.js"]

Let’s build our first image out of this Dockerfile, we will name it hello:v0.1

docker image build -t hello:v0.1 .

We then create a container to check it is running fine.

docker container run hello:v0.1

You should then have an output similar to the following one (the ID will be different though).

hello from 92d79b6de29f

There are always several ways to write a Dockerfile, we can start from a Linux distribution and then install a runtime (as we did above) or use images where this has already been done for us.

To illustrate that, we will now create a new Dockerfile but we will use the mhart/alpine-node:6.9.4 image. This is not an official image but it’s a very well known and used one.

Create a new Dockerfile named Dockerfile-v2 and make sure it has the following content.

FROM mhart/alpine-node:6.9.4
COPY . /app
WORKDIR /app
CMD ["node","index.js"]

Basically, it is not that different from the previous one, it just uses a base image that embeds alpine and a Node.js runtime so we do not have to install it ourself. In this example, installing Node.js is not a big deal, but it is really helpful to use image where a runtime (or else) is already packages when using more complex environments.

We will now create a new image using this Dockerfile.

docker image build -f Dockerfile-v2 -t hello:v0.2 .

Note: as we do not use the default name for our Dockerfile, we use the -f option to point towards the one we need to use.

We now run a container from this image.

docker container run hello:v0.2

Once again, the output will look like the following.

hello from 4094ff6bffbd

ENTRYPOINT vs COMMAND

In the 2 previous Dockerfile, we used CMD to define the command to be ran when a container is launched. As we have seen, there are several ways to define the command, using ENTRYPOINT and/or CMD. We will illustrate this on a new Dockerfile, named Dockerfile-v3, that as the following content.

FROM alpine
ENTRYPOINT ["ping"]
CMD ["localhost"]

Here, we define the ping command as the ENTRYPOINT and the localhost as the CMD, the command that will be ran by default is the concatenation of ENTRYPOINT and CMD: ping localhost. This command can be seen as a wrapper around the ping utility to which we can change the address we provide as a parameter.

Let’s create an image based on this new file.

docker image build -f Dockerfile-v3 -t ping:v0.1 .

We can run this image without specifying any command:

docker container run ping:v0.1

That should give a result like the following one.

PING localhost (127.0.0.1): 56 data bytes
bytes from 127.0.0.1: seq=0 ttl=64 time=0.046 ms
bytes from 127.0.0.1: seq=1 ttl=64 time=0.046 ms
bytes from 127.0.0.1: seq=2 ttl=64 time=0.046 ms
bytes from 127.0.0.1: seq=3 ttl=64 time=0.046 ms
bytes from 127.0.0.1: seq=4 ttl=64 time=0.047 ms

You can also override the default CMD indicating another IP address. We will use 8.8.8.8 which is the IP of a Google’s DNS.

docker container run ping:v0.1 8.8.8.8

That should return the following.

PING 8.8.8.8 (8.8.8.8): 56 data bytes
bytes from 8.8.8.8: seq=0 ttl=38 time=9.235 ms
bytes from 8.8.8.8: seq=1 ttl=38 time=8.590 ms
bytes from 8.8.8.8: seq=2 ttl=38 time=8.585 ms

Image Inspection

As we have already seen with containers, and as we will see with other Docker’s components (volume, network, …), the inspect command is available for the image API and it returns all the information of the image provided.

The alpine image should already be present locally, if it’s not, run the following command to pull it.

docker image pull alpine

Once we are sure it is there let’s inspect it.

docker image inspect alpine

There is a lot of information in there:

the layers the image is composed of
the driver used to store the layers
the architecture / os it has been created for
metadata of the image
…

We will not go into all the details now but it’s interesing to see an example of the Go template notation that enables to extract the part of information we need in just a simple command.

Let’s get the list of layers (only one for alpine)

docker image inspect --format "{{ json .RootFS.Layers }}" alpine | python -m json.tool

[
    "sha256:60ab55d3379d47c1ba6b6225d59d10e1f52096ee9d5c816e42c635ccc57a5a2b"
]

Let’s try another example to query only the Architecture information

docker image inspect --format "{{ .Architecture }}" alpine

This should return amd64.

Feel free to play with the Go template format and get familiar with it as it’s really handy.

Filesystem exploration

We first stop and remove all containers from your host (you might not be able to remove images if containers are using some of the layers).

docker container stop $(docker container ls -aq)
docker container rm $(docker container ls -aq)

Note: containers can also be removed in a non graceful way:

docker container rm -f $(docker container ls -aq)

We also remove all the images

docker image rm $(docker image ls -q)

We will now have a look inside the /var/lib/docker/overlay2 folder where the image and container layers are stored.

ls /var/lib/docker/overlay2

As we do not have any images yet, there should not be anything in this folder.

Let’s pull an nginx image

docker image pull nginx

You should get something like the following where we can see that 3 layers are pulled.

Using default tag: latest
latest: Pulling from library/nginx
5040bd298390: Pull complete
d7a91cdb22f0: Pull complete
9cac4850e5df: Pull complete
Digest: sha256:33ff28a2763feccc1e1071a97960b7fef714d6e17e2d0ff573b74825d0049303
Status: Downloaded newer image for nginx:latest

If we have a look to the changes that occurs in the /var/lib/docker/overlay2 folder

ls /var/lib/docker/overlay2

we can see the following:

261fed39e3aca63326758681c96cad5bfe7eeeabafda23408bee0f5ae365d3fd
28f7998921ca5e4b28231b59b619394ba73571b5127a9c28cc9bacb3db706d2a
backingFsBlockDev
c1ae1be1c1c62dbaacf26bb9a5cde02e30d5364e06a437d0626f31c55af82a58
l

Some folders, with names that looks like hash, were created. Those are the layers which, merged together, build the image filesystem.

Let’s run a container based on nginx.

docker container run -d nginx

We can now see 2 additional folders (ID, ID-init), those ones correspond to the read-write layer of the running container.

11995e6da1dc5acab33aceacea3656d3795a4fb136c3a65b37d40b97747b5f84
11995e6da1dc5acab33aceacea3656d3795a4fb136c3a65b37d40b97747b5f84-init
261fed39e3aca63326758681c96cad5bfe7eeeabafda23408bee0f5ae365d3fd
28f7998921ca5e4b28231b59b619394ba73571b5127a9c28cc9bacb3db706d2a
backingFsBlockDev
c1ae1be1c1c62dbaacf26bb9a5cde02e30d5364e06a437d0626f31c55af82a58
l

Feel free to go into those folder and explore their filesystems.

Docker Volumes

Wed, 25 Jan 2017 00:00:00 +0000

Understanding what is this volume thing

In this lab, we will illustrate the concept of volume. We will see how to use volume

in a Dockerfile
at runtime with the -v option
using the volume API

We will also see what is bind-mounting on a simple example.

Data persistency without a volume ?

We will first illustrate how data is not persisted outside of a container by default.

Let’s run an interactive shell within an alpine container named c1.

docker container run --name c1 -ti alpine sh

We will create the /data folder and a dummy hello.txt file in it.

mkdir /data && cd /data && touch hello.txt

We will then check how the read-write layer (container layer) is accessible from the host.

Let exit the container first.

exit

Let’s inspect our container in order to get the location of the container’s layer. We can use the inspect command and then scroll into the output until the GraphDriver key, like the following.

docker container inspect c1

Or we can directly use the Go template notation and get the content of the GraphDriver keys right away.

docker container inspect -f "{{ json .GraphDriver }}" c1 | jq

You should then get an output like the following (the ID will not be the same though)

{
    "Data": {
        "LowerDir": "/var/lib/docker/overlay2/55922a6b646ba6681c5eca253a19e90270e3872329a239a82877b2f8c505c9a2-init/diff:/var/lib/docker/overlay2/30474f5fc34277d1d9e5ed5b48e2fb979eee9805a61a0b2c4bf33b766ba65a16/diff",
        "MergedDir": "/var/lib/docker/overlay2/55922a6b646ba6681c5eca253a19e90270e3872329a239a82877b2f8c505c9a2/merged",
        "UpperDir": "/var/lib/docker/overlay2/55922a6b646ba6681c5eca253a19e90270e3872329a239a82877b2f8c505c9a2/diff",
        "WorkDir": "/var/lib/docker/overlay2/55922a6b646ba6681c5eca253a19e90270e3872329a239a82877b2f8c505c9a2/work"
    },
    "Name": "overlay2"
}

From our host, if we inspect the folder which path is specified in UpperDir, we can see our /data and the hello.txt file we created are there.

Try the below command, to see the contents of the /data folder:

ls /var/lib/docker/overlay2/[YOUR_ID]/diff/data

What happen if we remove our c1 container now ? Let’s try.

docker container rm c1

It seems the folder defined in the UpperDir above does not exist anymore. Do you confirm that ? Try running the ls command again and see the results.

This shows that data created in a container is not persisted. It’s removed with the container’s layer when the container is deleted.

Defining a volume in a Dockerfile

We will now see how volumes come into the picture to handle the data persistency.

We will start by creating a Dockerfile based on alpine and define the /data as a volume. This means that anything written by a container in /data will be persisted outside of the Union filesystem.

Create a Dockerfile with the following content

FROM alpine
VOLUME ["/data"]
ENTRYPOINT ["/bin/sh"]

Note: we specify /bin/sh as the ENTRYPOINT so that if no command is provided in interactive mode we will end up in a shell inside our container.

Let’s build an image from this Dockerfile.

docker image build -t img1 .

We will then create a container in interactive mode (using -ti flags) from this image and name it c2.

docker container run --name c2 -ti img1

We should then end up in a shell within the container. From there, we will go into /data and create a hello.txt file.

cd /data
touch hello.txt
ls

Let’s exit the container making sure it remains running: use the Control-P / Control-Q combination for this. Use the following command to make sure it’s still running.

docker container ls

Note: the container, named c2, should be listed there.

We will now inspect this container in order to get the location of the volume (defined on /data) on the host. We can use the inspect command and then scroll into the output until we find the Mounts key…

docker container inspect c2

Or we can directly use the Go template notation and get the content of the Mounts keys right away.

docker container inspect -f "{{ json .Mounts }}"  c2 | jq

You should then get an output like the following (the ID will not be the same though)

[
    {
        "Destination": "/data",
        "Driver": "local",
        "Mode": "",
        "Name": "2f5b7c6b77494934293fc7a09198dd3c20406f05272121728632a4aab545401c",
        "Propagation": "",
        "RW": true,
        "Source": "/var/lib/docker/volumes/2f5b7c6b77494934293fc7a09198dd3c20406f05272121728632a4aab545401c/_data",
        "Type": "volume"
    }
]

This output shows that the volume defined in /data is stored in /var/lib/docker/volumes/2f5…01c/_data on the host (removing part of the ID for a better readability).

Copy your own path (the one under the Source key) and make sure the hello.txt file we created (from within the container) is there.

We now remove the c2 container.

docker container stop c2 && docker container rm c2

Check that the folder defined under the Source key is still there and contains hello.txt file.

From the above, we can see that a volume bypasses the union filesystem and is not dependent on a container’s lifecycle.

Defining a volume at runtime

We have seen volume defined in a Dockerfile, we will see they can also be defined at runtime using the -v flag of the docker container run command.

Let’s create a container from the alpine image, we’ll use the -d option so it runs in background and also define a volume on /data as we’ve done previously. In order the PID 1 process remains active, we use the following command that pings Google DNS and log the output in a file within the /data folder.

ping 8.8.8.8 > /data/ping.txt

The container is ran that way:

docker container run --name c3 -d -v /data alpine sh -c 'ping 8.8.8.8 > /data/ping.txt'

Let’s inspect the container and get the Mounts key using the Go template notation.

docker container inspect -f "{{ json .Mounts }}" c3 | jq

We have pretty much the same output as we had when we defined the volume in the Dockerfile.

[
  {
    "Type": "volume",
    "Name": "af621cde2717307e5bf91be850c5a00474d58b8cdc8d6e37f2e373631c2f1331",
    "Source": "/var/lib/docker/volumes/af621cde2717307e5bf91be850c5a00474d58b8cdc8d6e37f2e373631c2f1331/_data",
    "Destination": "/data",
    "Driver": "local",
    "Mode": "",
    "RW": true,
    "Propagation": ""
  }
]

If we use the folder defined in the Source key, and check the content of the ping.txt within the /data folder, we get something similar to the following.

tail -f /var/lib/docker/volumes/OUR_ID/_data/ping.txt
bytes from 8.8.8.8: seq=34 ttl=37 time=0.462 ms
bytes from 8.8.8.8: seq=35 ttl=37 time=0.436 ms
bytes from 8.8.8.8: seq=36 ttl=37 time=0.512 ms
bytes from 8.8.8.8: seq=37 ttl=37 time=0.487 ms
bytes from 8.8.8.8: seq=38 ttl=37 time=0.409 ms
bytes from 8.8.8.8: seq=39 ttl=37 time=0.438 ms
bytes from 8.8.8.8: seq=40 ttl=37 time=0.477 ms
...

The ping.txt file is updated regularly by the command running in the c3 container.

Stopping and removing the container will obviously stop the ping command but the /data/ping.txt file will still be there. Give it a try :)

Usage of the Volume API

The volume API introduced in Docker 1.9 enables to perform operations on volume very easily.

First have a look at the commands available in the volume API.

docker volume --help

We will start with the create command, and create a volume named html.

docker volume create --name html

If we list the existing volume, our html volume should be the only one.

docker volume ls

The output should be something like

DRIVER              VOLUME NAME
[other previously created volumes]
local               html

In the volume API, like for almost all the other Docker’s API, there is an inspect command. Let’s use it against the html volume.

docker volume inspect html

The output should be the following one.

[
    {
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/html/_data",
        "Name": "html",
        "Options": {},
        "Scope": "local"
    }
]

The Mountpoint defined here is the path on the Docker host where the volume can be accessed. We can note that this path uses the name of the volume instead of the auto-generated ID we saw in the example above.

We can now use this volume and mount it on a specific path of a container. We will use a Nginx image and mount the html volume onto /usr/share/nginx/html folder within the container.

Note: /usr/share/nginx/html is the default folder served by nginx. It contains 2 files: index.html and 50x.html

docker container run --name www -d -p 8080:80 -v html:/usr/share/nginx/html nginx

Note: we use the -p option to map the nginx default port (80) to a port on the host (8080). We will come back to this in the lesson dedicated to the networking.

From the host, let’s have a look at the content of the volume.

ls /var/lib/docker/volumes/html/_data

The content of the /usr/share/nginx/html folder of the www container has been copied into the /var/lib/docker/volumes/html/_data folder on the host.

Let’s have a look at the nginx’s welcome page

From our host, we can now modify the index.html file and verify the changes are taken into account within the container.

cat<<END >/var/lib/docker/volumes/html/_data/index.html
SOMEONE HERE ?
END

Let’s have a look at the nginx’s welcome page. We can see the changes we have done in the index.html.

Note: please reload the page if you cannot see the changes.

Mount host’s folder into a container

The last item we will talk about is named bind-mount and consist of mounting a host’s folder into a container’s folder. This is done using the -v option of the docker container run command. Instead of specifying one single path (as we did when defining volumes) we will specified 2 paths separated by a column.

docker container run -v HOST_PATH:CONTAINER_PATH [OPTIONS] IMAGE [CMD]

Note: HOST_PATH and CONTAINER_PATH can be a folder or file. None of the Paths have to exist before starting the Container as they will be created automatically during the start.

1st case

Let’s run an alpine container bind mounting the local /tmp folder inside the container /data folder.

docker container run -ti -v /tmp:/data alpine sh

We end up in a shell inside our container. By default, there is no /data folder in an alpine distribution. What is the impact of the bind-mount ?

ls /data

The /data folder has been created inside the container and it contains the content of the /tmp folder of the host. We can now, from the container, change files on the host and the other way round.

2nd case

Let’s run a nginx container bind mounting the local /tmp folder inside the /usr/share/nginx/html folder of the container.

docker container run -ti -v /tmp:/usr/share/nginx/html nginx bash

Are the default index.html and 50x.html files still there in the container’s /usr/share/nginx/html folder ?

ls /usr/share/nginx/html

No ! The content of the container’s folder has been overridden with the content of the host folder.

Bind-mounting is very usefull in development as it enables, for instance, to share source code on the host with the container.

Swarm mode introduction

Tue, 24 Jan 2017 00:00:00 +0000

This tutorial will show you how to setup a swarm and deploy your first services.

Init your swarm

docker swarm init --advertise-addr $(hostname -i)

Copy the join command (watch out for newlines) output and paste it in the other terminal.

Show members of swarm

Type the below command in the first terminal:

docker node ls

That last line will show you a list of all the nodes, something like this:

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGE
R STATUS
kytp4gq5mrvmdbb0qpifdxeiv *  node1     Ready   Active        Leader
lz1j4d6290j8lityk4w0cxls5    node2     Ready   Active

If you try to execute an administrative command in a non-leader node worker, you’ll get an error. Try it here:

docker node ls

Creating services

The next step is to create a service and list out the services. This creates a single service called web that runs the latest nginx, type the below commands in the first terminal:

docker service create -p 80:80 --name web nginx:latest
docker service ls

You can check that nginx is running by executing the following command:

curl http://localhost:80

Scaling up

We will be performing these actions in the first terminal. Next let’s inspect the service:

docker service inspect web

That’s lots of info! Now, let’s scale the service:

docker service scale web=15

Docker has spread the 15 services evenly over all of the nodes

docker service ps web

Updating nodes

You can also drain a particular node, that is remove all services from that node. The services will automatically be rescheduled on other nodes.

docker node update --availability drain node2

docker service ps web

You can check out the nodes and see that node2 is still active but drained.

docker node ls

Scaling down

You can also scale down the service

docker service scale web=10

Lets check our service status

docker service ps web

Now bring node2 back online and show it’s new availability

docker node update --availability active node2

docker node inspect node2 --pretty

Which of these can you do with Docker Swarm Mode?

Docker compose with swarm secrets

Mon, 23 Jan 2017 00:00:00 +0000

Start securing your swarm services using the latest compose reference that allows to specify secrets in your application stack

Getting started

Make sure your daemon is in swarm mode or initialize it as follows:

docker swarm init --advertise-addr $(hostname -i)

Automatic provision

In this example we’ll let compose automatically create our secrets and provision them through compose with the defined secret file

Create a new secret and store it in a file:

echo "shh, this is a secret" > mysecret.txt

Use your secret file in your compose stack as follows:

echo 'version: '\'3.1\''
services:
    test:
        image: '\'alpine\''
        command: '\'cat /run/secrets/my_secret \''
        secrets: 
            - my_secret

secrets:
    my_secret:
        file: ./mysecret.txt
' > docker-compose.yml

Deploy your stack service:

docker stack deploy -c docker-compose.yml secret

Results in the below output:

Creating network secret_default
Creating secret secret_my_secret
Creating service secret_test

After your stack is deployed you can check your service output:

docker service logs -f secret_test

Results in the below output (below values after secret_test.1. may vary):

secret_test.1.lcygnppmzfdp@node1    | shhh, this is a secret
secret_test.1.mg1420w2i3x4@node1    | shhh, this is a secret
secret_test.1.8osraz8yxjrb@node1    | shhh, this is a secret
secret_test.1.byh5b9uik6db@node1    | shhh, this is a secret
.
.
.

Using existing secrets

Create a new secret using the docker CLI:

echo "some other secret" | docker secret create manual_secret -

Define your secret as external in your compose file so it’s not created while deploying the stack

echo 'version: '\'3.1\''
services:
    test:
        image: '\'alpine\''
        command: '\'cat /run/secrets/manual_secret \''
        secrets: 
            - manual_secret

secrets:
    manual_secret:
        external: true
' > external-compose.yml

Deploy your stack as you did in the automatic section:

docker stack deploy -c external-compose.yml external_secret

Validate your secret is there by checking the service logs

docker service logs -f external_secret_test

Docker swarm config files

Sun, 22 Jan 2017 00:00:00 +0000

This tutorials showcases the config swarm feature that allow config objects to be attached to services. Config files can be mounted inside services’ containers, avoiding the need to bake configuration into images.

Configuration files are similar to secrets, and in fact the CLI and API show few differences between the two. The principal differences so far are:

Secrets are always redacted at the API level, so the payload cannot be obtained through an API call after they are created.
Secrets are restricted to the /run/secrets directory inside the container, as a design choice. Config files can be mounted anywhere. Start securing your swarm services using the latest compose reference that allows to specify secrets in your application stack

Requirements

Docker 17.06+

Getting started

Initialize your swarm:

docker swarm init --advertise-addr $(hostname -i)

Let’s peak the config options:

docker config --help

As you can see the API is very similar to the docker secrets. Let’s create our first config object

echo "this is some crazy config stuff" | docker config create my_config -

As stated before, unlike secrets, you can actually see the content of the config objects directly from the CLI. Let’s check this:

docker config inspect my_config

Wait, what?, where’s my config?. Docker hides the config information by default to prevent unnecessary large outputs; in order to display the config value the --pretty flag needs to added

docker config inspect --pretty my_config

Finally, let’s deploy a service using our recently created config

docker service create --name test_cfg --config my_config alpine cat /my_config

You can check your service logs to see your configuration.

docker service logs test_cfg

As you can see, as we didn’t specify any destination mountpoint, by default configs will be located at the root path. However, with configs you can place them wherever you need.

docker service create --name test_cfg_mount --config source=my_config,target=/tmp/cfg alpine cat /tmp/cfg

Same as before, check your service logs to see the expected configuration:

docker service logs test_cfg_mount

Windows Containers Setup

Sun, 22 Jan 2017 00:00:00 +0000

Setup

There are four environments you can use to set-up Windows Containers. We have provided separate guides for each:

Windows 10 with the Anniversary Update
Windows Server 2016 on Azure
Windows Server 2016 on AWS
Windows Server 2016 on bare metal or in VM

Setup - Windows 10

This chapter explores setting up a Windows environment to properly use Windows containers on Windows 10.

Windows 10 with Anniversary Update

For developers, Windows 10 is a great place to run Docker Windows containers and containerization support was added to the Windows 10 kernel with the Anniversary Update (note that container images can only be based on Windows Server Core and Nanoserver, not Windows 10). All that’s missing is the Windows-native Docker Engine and some image base layers.

The simplest way to get a Windows Docker Engine is by installing the Docker for Windows public beta (direct download link). Docker for Windows used to only setup a Linux-based Docker development environment, but the public beta version now sets up both Linux and Windows Docker development environments, and we’re working on improving Windows container support and Linux/Windows container interoperability.

With the public beta installed, the Docker for Windows tray icon has an option to switch between Linux and Windows container development.

You can skip the rest of this lab.

Setup - Azure

This chapter explores setting up a Windows environment to properly use Windows containers on Microsoft Azure.

Windows Server 2016 on Azure

Microsoft Azure has a pre-baked VM image with Docker engine and base images pre-loaded. To get started (requires Azure account):

Create a Windows Server 2016 Datacenter - with Containers VM. This VM image has Docker pre-installed and the Windows base layers pre-loaded.
Select “Classic” deployment model and hit “Create”
Input setup parameters
- Default settings are good
- Creating a new resource group is fine
- DS2_V2 instance type has good performance for development and testing
Check the Summary and hit “OK”. Setup will take a couple of minutes
Once the VM is running, select “Connect” to open a remote desktop connection. If using macOS, get the free Remote Desktop app in the Mac App Store
Login with the username and password configured during setup
Start PowerShell
Start-Service docker
Check that Docker is running with docker version

You can skip the rest of this lab.

Setup - AWS

This chapter explores setting up a Windows environment to properly use Windows containers on Amazon Web Services (AWS).

Windows Server 2016 on AWS

AWS has a pre-baked AMI with Docker Engine already installed. To start an instance, do the following (requires AWS account):

Open the EC2 launch-instance wizard
Select the “Microsoft Windows Server 2016 Base with Containers” AMI
Input setup parameters
- c4.large has good performance for development and testing
- The default AWS security group settings will let you connect with Remote Desktop
Select “Review and Launch”
Once the VM is up, hit “Connect”. If using macOS, get the free Remote Desktop app in the Mac App Store
See details on getting the initial Windows Administrator password for your AWS instance
Start PowerShell
Check that Docker is running with docker version

You can skip the rest of this lab.

Setup - Windows Server 2016

This chapter explores setting up a Windows environment to properly use Windows containers on Windows Server 2016, on bare metal or in VM.

Windows Server 2016 on bare metal or in VM

Windows Server 2016 is where Docker Windows containers should be deployed for production. For developers planning to do lots of Docker Windows container development, it may also be worth setting up a Windows Server 2016 dev system (in a VM, for example), at least until Windows 10 and Docker for Windows support for Windows containers matures. Running a VM with Windows Server 2016 is also a great way to do Docker Windows container development on macOS and older Windows versions.

Once Windows Server 2016 is running, log in, run Windows Update (use sconfig on Windows Server Core) to ensure all the latest updates are installed and install the Windows-native Docker Engine (that is, don’t use “Docker for Windows”). There are two options: Install using a Powershell Package (recommended) or with DSC.

PowerShell Package Provider (recommended)

Microsoft maintains a PowerShell package provider that lets easily install Docker on Windows Server 2016.

Run the following in an Administrative PowerShell prompt:

Install-Module -Name DockerMsftProvider -Force
Install-Package -Name docker -ProviderName DockerMsftProvider -Force
Restart-Computer -Force

PowerShell Desired State Configuration

If interested in experimenting with Windows PowerShell Desired State Configuration, Daniel Scott-Raynsford has built a prototype script that uses DSC to install Docker Engine.

Here’s how to use it:

Install-Script -Name Install-DockerOnWS2016UsingDSC
Install-DockerOnWS2016UsingDSC.ps1

See Daniel’s blog post for details on installing Docker with DSC.

Whether using the PowerShell Package Provider or DSC, Docker Engine is now running as a Windows service, listening on the default Docker named pipe.

For development VMs running (for example) in a Hyper-V VM on Windows 10, it might be advantageous to make the Docker Engine running in the Windows Server 2016 VM available to the Windows 10 host:

# Open firewall port 2375
netsh advfirewall firewall add rule name="docker engine" dir=in action=allow protocol=TCP localport=2375

# Configure Docker daemon to listen on both pipe and TCP (replaces docker --register-service invocation above)
Stop-Service docker
dockerd --unregister-service
dockerd -H npipe:// -H 0.0.0.0:2375 --register-service
Start-Service docker

The Windows Server 2016 Docker engine can now be used from the VM host by setting DOCKER_HOST: $env:DOCKER_HOST = "<ip-address-of-vm>:2375"

Windows Containers Basics

Sat, 21 Jan 2017 00:00:00 +0000

Getting Started with Windows Containers

This chapter will cover the basics of using Windows Containers with Docker.

##Running Windows containers

First, make sure the Docker installation is working:

> docker version
Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 05:27:08 2016
 OS/Arch:      windows/amd64
 Experimental: true

Server:
 Version:      1.12.2-cs2-ws-beta
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   050b611
 Built:        Tue Oct 11 02:35:40 2016
 OS/Arch:      windows/amd64

Next, pull a base image that’s compatible with the evaluation build, re-tag it and do a test-run:

docker pull microsoft/windowsservercore:10.0.14393.321
docker tag microsoft/windowsservercore:10.0.14393.321 microsoft/windowsservercore
docker run microsoft/windowsservercore hostname
69c7de26ea48

Building and pushing Windows container images

Pushing images to Docker Cloud requires a free Docker ID. Storing images on Docker Cloud is a great way to save build artifacts for later user, to share base images with co-workers or to create build-pipelines that move apps from development to production with Docker.

Docker images are typically built with docker build from a Dockerfile recipe, but for this example, we’re going to just create an image on the fly in PowerShell.

"FROM microsoft/windowsservercore `n CMD echo Hello World!" | docker build -t <docker-id>/windows-test-image -

Test the image:

docker run <docker-id>/windows-test-image
Hello World!

docker push <docker-id>/windows-test-image

Images stored on Docker Cloud are available in the web interface and public images can be pulled by other Docker users in the Docker Store.

Docker Orchestration Hands-on Lab

Fri, 20 Jan 2017 00:00:00 +0000

In this lab you will play around with the container orchestration features of Docker. You will deploy a simple application to a single host and learn how that works. Then, you will configure Docker Swarm Mode, and learn to deploy the same simple application across multiple hosts. You will then see how to scale the application and move the workload across different hosts easily.

Difficulty: Beginner

Time: Approximately 30 minutes

Tasks:

Section #1 - What is Orchestration

Section #2 - Configure Swarm Mode

Section #3 - Deploy applications across multiple hosts

Section #4 - Scale the application

Section #5 - Drain a node and reschedule the containers

Cleaning Up

Section 1: What is Orchestration

So, what is Orchestration anyways? Well, Orchestration is probably best described using an example. Let’s say that you have an application that has high traffic along with high-availability requirements. Due to these requirements, you typically want to deploy across at least 3+ machines, so that in the event a host fails, your application will still be accessible from at least two others. Obviously, this is just an example and your use-case will likely have its own requirements, but you get the idea.

Deploying your application without Orchestration is typically very time consuming and error prone, because you would have to manually SSH into each machine, start up your application, and then continually keep tabs on things to make sure it is running as you expect.

But, with Orchestration tooling, you can typically off-load much of this manual work and let automation do the heavy lifting. One cool feature of Orchestration with Docker Swarm, is that you can deploy an application across many hosts with only a single command (once Swarm mode is enabled). Plus, if one of the supporting nodes dies in your Docker Swarm, other nodes will automatically pick up load, and your application will continue to hum along as usual.

If you are typically only using docker run to deploy your applications, then you could likely really benefit from using Docker Compose, Docker Swarm mode, or both Docker Compose and Swarm.

Section 2: Configure Swarm Mode

Real-world applications are typically deployed across multiple hosts as discussed earlier. This improves application performance and availability, as well as allowing individual application components to scale independently. Docker has powerful native tools to help you do this.

An example of running things manually and on a single host would be to create a new container on node1 by running docker run -dt ubuntu sleep infinity.

docker run -dt ubuntu sleep infinity

Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
d54efb8db41d: Pull complete
f8b845f45a87: Pull complete
e8db7bf7c39f: Pull complete
9654c40e9079: Pull complete
6d9ef359eaaa: Pull complete
Digest: sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535
Status: Downloaded newer image for ubuntu:latest
846af8479944d406843c90a39cba68373c619d1feaa932719260a5f5afddbf71

This command will create a new container based on the ubuntu:latest image and will run the sleep command to keep the container running in the background. You can verify our example container is up by running docker ps on node1.

docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
044bea1c2277        ubuntu              "sleep infinity"    2 seconds ago       Up 1 second                             distracted_mayer

But, this is only on a single node. What happens if this node goes down? Well, our application just dies and it is never restarted. To restore service, we would have to manually log into this machine, and start tweaking things to get it back up and running. So, it would be helpful if we had some type of system that would allow us to run this “sleep” application/service across many machines.

In this section you will configure Swarm Mode. This is a new optional mode in which multiple Docker hosts form a self-orchestrating group of engines called a swarm. Swarm mode enables new features such as services and bundles that help you deploy and manage multi-container apps across multiple Docker hosts.

You will complete the following:

Configure Swarm mode
Run the app
Scale the app
Drain nodes for maintenance and reschedule containers

For the remainder of this lab we will refer to Docker native clustering as Swarm mode. The collection of Docker engines configured for Swarm mode will be referred to as the swarm.

A swarm comprises one or more Manager Nodes and one or more Worker Nodes. The manager nodes maintain the state of swarm and schedule application containers. The worker nodes run the application containers. As of Docker 1.12, no external backend, or 3rd party components, are required for a fully functioning swarm - everything is built-in!

In this part of the demo you will use all three of the nodes in your lab. node1 will be the Swarm manager, while node2 and node3 will be worker nodes. Swarm mode supports highly available redundant manager nodes, but for the purposes of this lab you will only deploy a single manager node.

Step 2.1 - Create a Manager node

In this step you’ll initialize a new Swarm, join a single worker node, and verify the operations worked.

Run docker swarm init on node1.

docker swarm init --advertise-addr $(hostname -i)

Swarm initialized: current node (6dlewb50pj2y66q4zi3egnwbi) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-1wxyoueqgpcrc4xk2t3ec7n1poy75g4kowmwz64p7ulqx611ih-68pazn0mj8p4p4lnuf4ctp8xy \
    10.0.0.5:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

You can run the docker info command to verify that node1 was successfully configured as a swarm manager node.

docker info

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 2
Server Version: 17.03.1-ee-3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 13
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: rwezvezez3bg1kqg0y0f4ju22
 Is Manager: true
 ClusterID: qccn5eanox0uctyj6xtfvesy2
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.0.0.5
 Manager Addresses:
  10.0.0.5:2377
<Snip>

The swarm is now initialized with node1 as the only Manager node. In the next section you will add node2 and node3 as Worker nodes.

Step 2.2 - Join Worker nodes to the Swarm

You will perform the following procedure on node2 and node3. Towards the end of the procedure you will switch back to node1.

Now, take the entire docker swarm join ... command we copied earlier from node1 where it was displayed as terminal output. We need to paste the copied command into the terminal of node2 and node3.

It should look something like this for node2. By the way, if the docker swarm join ... command scrolled off your screen already, you can run the docker swarm join-token worker command on the Manager node to get it again.

Remember, the tokens displayed here are not the actual tokens you will use. Copy the command from the output on node1. On node2 and node3 it should look like this:

docker swarm join \
    --token SWMTKN-1-1wxyoueqgpcrc4xk2t3ec7n1poy75g4kowmwz64p7ulqx611ih-68pazn0mj8p4p4lnuf4ctp8xy \
    10.0.0.5:2377

docker swarm join \
    --token SWMTKN-1-1wxyoueqgpcrc4xk2t3ec7n1poy75g4kowmwz64p7ulqx611ih-68pazn0mj8p4p4lnuf4ctp8xy \
    10.0.0.5:2377

Once you have run this on node2 and node3, switch back to node1, and run a docker node ls to verify that both nodes are part of the Swarm. You should see three nodes, node1 as the Manager node and node2 and node3 both as Worker nodes.

docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
6dlewb50pj2y66q4zi3egnwbi *  node1   Ready   Active        Leader
ym6sdzrcm08s6ohqmjx9mk3dv    node3   Ready   Active
yu3hbegvwsdpy9esh9t2lr431    node2   Ready   Active

The docker node ls command shows you all of the nodes that are in the swarm as well as their roles in the swarm. The * identifies the node that you are issuing the command from.

Congratulations! You have configured a swarm with one manager node and two worker nodes.

Section 3: Deploy applications across multiple hosts

Now that you have a swarm up and running, it is time to deploy our really simple sleep application.

You will perform the following procedure from node1.

Step 3.1 - Deploy the application components as Docker services

Our sleep application is becoming very popular on the internet (due to hitting Reddit and HN). People just love it. So, you are going to have to scale your application to meet peak demand. You will have to do this across multiple hosts for high availability too. We will use the concept of Services to scale our application easily and manage many containers as a single entity.

Services were a new concept in Docker 1.12. They work with swarms and are intended for long-running containers.

You will perform this procedure from node1.

Let’s deploy sleep as a Service across our Docker Swarm.

docker service create --name sleep-app ubuntu sleep infinity

of5rxsxsmm3asx53dqcq0o29c

Verify that the service create has been received by the Swarm manager.

docker service ls

ID            NAME       MODE        REPLICAS  IMAGE
of5rxsxsmm3a  sleep-app  replicated  1/1       ubuntu:latest

The state of the service may change a couple times until it is running. The image is being downloaded from Docker Store to the other engines in the Swarm. Once the image is downloaded the container goes into a running state on one of the three nodes.

At this point it may not seem that we have done anything very differently than just running a docker run .... We have again deployed a single container on a single host. The difference here is that the container has been scheduled on a swarm cluster.

Well done. You have deployed the sleep-app to your new Swarm using Docker services.

Section 4: Scale the application

Demand is crazy! Everybody loves your sleep app! It’s time to scale out.

One of the great things about services is that you can scale them up and down to meet demand. In this step you’ll scale the service up and then back down.

You will perform the following procedure from node1.

Scale the number of containers in the sleep-app service to 7 with the docker service update --replicas 7 sleep-app command. replicas is the term we use to describe identical containers providing the same service.

docker service update --replicas 7 sleep-app

The Swarm manager schedules so that there are 7 sleep-app containers in the cluster. These will be scheduled evenly across the Swarm members.

We are going to use the docker service ps sleep-app command. If you do this quick enough after using the --replicas option you can see the containers come up in real time.

docker service ps sleep-app

ID            NAME         IMAGE          NODE     DESIRED STATE  CURRENT STATE          ERROR  PORTS
7k0flfh2wpt1  sleep-app.1  ubuntu:latest  node1  Running        Running 9 minutes ago
wol6bzq7xf0v  sleep-app.2  ubuntu:latest  node3  Running        Running 2 minutes ago
id50tzzk1qbm  sleep-app.3  ubuntu:latest  node2  Running        Running 2 minutes ago
ozj2itmio16q  sleep-app.4  ubuntu:latest  node3  Running        Running 2 minutes ago
o4rk5aiely2o  sleep-app.5  ubuntu:latest  node2  Running        Running 2 minutes ago
35t0eamu0rue  sleep-app.6  ubuntu:latest  node2  Running        Running 2 minutes ago
44s8d59vr4a8  sleep-app.7  ubuntu:latest  node1  Running        Running 2 minutes ago

Notice that there are now 7 containers listed. It may take a few seconds for the new containers in the service to all show as RUNNING. The NODE column tells us on which node a container is running.

Scale the service back down to just four containers with the docker service update --replicas 4 sleep-app command.

docker service update --replicas 4 sleep-app

Verify that the number of containers has been reduced to 4 using the docker service ps sleep-app command.

docker service ps sleep-app

ID            NAME         IMAGE          NODE     DESIRED STATE  CURRENT STATE           ERROR  PORTS
7k0flfh2wpt1  sleep-app.1  ubuntu:latest  node1  Running        Running 13 minutes ago
wol6bzq7xf0v  sleep-app.2  ubuntu:latest  node3  Running        Running 5 minutes ago
35t0eamu0rue  sleep-app.6  ubuntu:latest  node2  Running        Running 5 minutes ago
44s8d59vr4a8  sleep-app.7  ubuntu:latest  node1  Running        Running 5 minutes ago

You have successfully scaled a swarm service up and down.

Section 5: Drain a node and reschedule the containers

Your sleep-app has been doing amazing after hitting Reddit and HN. It’s now number 1 on the App Store! You have scaled up during the holidays and down during the slow season. Now you are doing maintenance on one of your servers so you will need to gracefully take a server out of the swarm without interrupting service to your customers.

Take a look at the status of your nodes again by running docker node ls on node1.

docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
6dlewb50pj2y66q4zi3egnwbi *  node1   Ready   Active        Leader
ym6sdzrcm08s6ohqmjx9mk3dv    node3   Ready   Active
yu3hbegvwsdpy9esh9t2lr431    node2   Ready   Active

You will be taking node2 out of service for maintenance.

Let’s see the containers that you have running on node2.

docker ps

CONTAINER ID        IMAGE                                                                            COMMAND             CREATED             STATUS              PORTS               NAMES
4e7ea1154ea4        ubuntu@sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535   "sleep infinity"    9 minutes ago       Up 9 minutes                            sleep-app.6.35t0eamu0rueeozz0pj2xaesi

You can see that we have one of the slepp-app containers running here (your output might look different though).

Now let’s jump back to node1 (the Swarm manager) and take node2 out of service. To do that, let’s run docker node ls again.

docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
6dlewb50pj2y66q4zi3egnwbi *  node1   Ready   Active        Leader
ym6sdzrcm08s6ohqmjx9mk3dv    node3   Ready   Active
yu3hbegvwsdpy9esh9t2lr431    node2   Ready   Active

We are going to take the ID for node2 and run docker node update --availability drain yournodeid. We are using the node2 host ID as input into our drain command. Replace yournodeid with the id of node2.

docker node update --availability drain yournodeid

Check the status of the nodes

docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
6dlewb50pj2y66q4zi3egnwbi *  node1   Ready   Active        Leader
ym6sdzrcm08s6ohqmjx9mk3dv    node3   Ready   Active
yu3hbegvwsdpy9esh9t2lr431    node2   Ready   Drain

Node node2 is now in the Drain state.

Switch back to node2 and see what is running there by running docker ps.

docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

node2 does not have any containers running on it.

Lastly, check the service again on node1 to make sure that the container were rescheduled. You should see all four containers running on the remaining two nodes.

docker service ps sleep-app

ID            NAME             IMAGE          NODE     DESIRED STATE  CURRENT STATE           ERROR  PORTS
7k0flfh2wpt1  sleep-app.1      ubuntu:latest  node1  Running        Running 25 minutes ago
wol6bzq7xf0v  sleep-app.2      ubuntu:latest  node3  Running        Running 18 minutes ago
s3548wki7rlk  sleep-app.6      ubuntu:latest  node3  Running        Running 3 minutes ago
35t0eamu0rue   \_ sleep-app.6  ubuntu:latest  node2  Shutdown       Shutdown 3 minutes ago
44s8d59vr4a8  sleep-app.7      ubuntu:latest  node1  Running        Running 18 minutes ago

Cleaning Up

Execute the docker service rm sleep-app command on node1 to remove the service called myservice.

docker service rm sleep-app

Execute the docker ps command on node1 to get a list of running containers.

docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
044bea1c2277        ubuntu              "sleep infinity"    17 minutes ago      17 minutes ag                           distracted_mayer

You can use the docker kill <CONTAINER ID> command on node1 to kill the sleep container we started at the beginning.

docker kill yourcontainerid

Finally, let’s remove node1, node2, and node3 from the Swarm. We can use the docker swarm leave --force command to do that.

Lets run docker swarm leave --force on node1.

docker swarm leave --force

Then, run docker swarm leave --force on node2.

docker swarm leave --force

Finally, run docker swarm leave --force on node3.

docker swarm leave --force

Congratulations! You’ve completed this lab. You now know how to build a swarm, deploy applications as collections of services, and scale individual services up and down.

Docker Networking Hands-on Lab

Fri, 20 Jan 2017 00:00:00 +0000

In this lab you will learn about key Docker Networking concepts. You will get your hands dirty by going through examples of a few basic networking concepts, learn about Bridge networking, and finally Overlay networking.

Difficulty: Beginner to Intermediate

Time: Approximately 45 minutes

Tasks:

Section #1 - Networking Basics

Section #2 - Bridge Networking

Section #3 - Overlay Networking

Cleaning Up

Section #1 - Networking Basics

Step 1: The Docker Network Command

The docker network command is the main command for configuring and managing container networks. Run the docker network command from the first terminal.

docker network

Usage:	docker network COMMAND

Manage networks

Options:
      --help   Print usage

Commands:
  connect     Connect a container to a network
  create      Create a network
  disconnect  Disconnect a container from a network
  inspect     Display detailed information on one or more networks
  ls          List networks
  prune       Remove all unused networks
  rm          Remove one or more networks

Run 'docker network COMMAND --help' for more information on a command.

The command output shows how to use the command as well as all of the docker network sub-commands. As you can see from the output, the docker network command allows you to create new networks, list existing networks, inspect networks, and remove networks. It also allows you to connect and disconnect containers from networks.

Step 2: List networks

Run a docker network ls command to view existing container networks on the current Docker host.

docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
3430ad6f20bf        bridge              bridge              local
a7449465c379        host                host                local
06c349b9cc77        none                null                local

The output above shows the container networks that are created as part of a standard installation of Docker.

New networks that you create will also show up in the output of the docker network ls command.

You can see that each network gets a unique ID and NAME. Each network is also associated with a single driver. Notice that the “bridge” network and the “host” network have the same name as their respective drivers.

Step 3: Inspect a network

The docker network inspect command is used to view network configuration details. These details include; name, ID, driver, IPAM driver, subnet info, connected containers, and more.

Use docker network inspect <network> to view configuration details of the container networks on your Docker host. The command below shows the details of the network called bridge.

docker network inspect bridge

[
    {
        "Name": "bridge",
        "Id": "3430ad6f20bf1486df2e5f64ddc93cc4ff95d81f59b6baea8a510ad500df2e57",
        "Created": "2017-04-03T16:49:58.6536278Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

NOTE: The syntax of the docker network inspect command is docker network inspect <network>, where <network> can be either network name or network ID. In the example above we are showing the configuration details for the network called “bridge”. Do not confuse this with the “bridge” driver.

Step 4: List network driver plugins

The docker info command shows a lot of interesting information about a Docker installation.

Run the docker info command and locate the list of network plugins.

docker info

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.03.1-ee-3
Storage Driver: aufs
<Snip>
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
<Snip>

The output above shows the bridge, host,macvlan, null, and overlay drivers.

Section #2 - Bridge Networking

Step 1: The Basics

Every clean installation of Docker comes with a pre-built network called bridge. Verify this with the docker network ls.

docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
3430ad6f20bf        bridge              bridge              local
a7449465c379        host                host                local
06c349b9cc77        none                null                local

The output above shows that the bridge network is associated with the bridge driver. It’s important to note that the network and the driver are connected, but they are not the same. In this example the network and the driver have the same name - but they are not the same thing!

The output above also shows that the bridge network is scoped locally. This means that the network only exists on this Docker host. This is true of all networks using the bridge driver - the bridge driver provides single-host networking.

All networks created with the bridge driver are based on a Linux bridge (a.k.a. a virtual switch).

Install the brctl command and use it to list the Linux bridges on your Docker host. You can do this by running sudo apt-get install bridge-utils.

apk update
apk add bridge

Then, list the bridges on your Docker host, by running brctl show.

brctl show

bridge name	bridge id		STP enabled	interfaces
docker0		8000.024252ed52f7	no

The output above shows a single Linux bridge called docker0. This is the bridge that was automatically created for the bridge network. You can see that it has no interfaces currently connected to it.

You can also use the ip a command to view details of the docker0 bridge.

ip a

<Snip>
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:52:ed:52:f7 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever

Step 2: Connect a container

The bridge network is the default network for new containers. This means that unless you specify a different network, all new containers will be connected to the bridge network.

Create a new container by running docker run -dt ubuntu sleep infinity.

docker run -dt ubuntu sleep infinity

Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
d54efb8db41d: Pull complete
f8b845f45a87: Pull complete
e8db7bf7c39f: Pull complete
9654c40e9079: Pull complete
6d9ef359eaaa: Pull complete
Digest: sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535
Status: Downloaded newer image for ubuntu:latest
846af8479944d406843c90a39cba68373c619d1feaa932719260a5f5afddbf71

docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
846af8479944        ubuntu              "sleep infinity"    55 seconds ago      Up 54 seconds                           heuristic_boyd

As no network was specified on the docker run command, the container will be added to the bridge network.

Run the brctl show command again.

brctl show

bridge name	bridge id		STP enabled	interfaces
docker0		8000.024252ed52f7	no		vethd630437

Notice how the docker0 bridge now has an interface connected. This interface connects the docker0 bridge to the new container just created.

You can inspect the bridge network again, by running docker network inspect bridge, to see the new container attached to it.

docker network inspect bridge

<Snip>
        "Containers": {
            "846af8479944d406843c90a39cba68373c619d1feaa932719260a5f5afddbf71": {
                "Name": "heuristic_boyd",
                "EndpointID": "1265c418f0b812004d80336bafdc4437eda976f166c11dbcc97d365b2bfa91e5",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
<Snip>

Step 3: Test network connectivity

The output to the previous docker network inspect command shows the IP address of the new container. In the previous example it is “172.17.0.2” but yours might be different.

Ping the IP address of the container from the shell prompt of your Docker host by running ping -c5 <IPv4 Address>. Remember to use the IP of the container in your environment.

ping -c5 172.17.0.2
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.055 ms
64 bytes from 172.17.0.2: icmp_seq=2 ttl=64 time=0.031 ms
64 bytes from 172.17.0.2: icmp_seq=3 ttl=64 time=0.034 ms
64 bytes from 172.17.0.2: icmp_seq=4 ttl=64 time=0.041 ms
64 bytes from 172.17.0.2: icmp_seq=5 ttl=64 time=0.047 ms

--- 172.17.0.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4075ms
rtt min/avg/max/mdev = 0.031/0.041/0.055/0.011 ms

The replies above show that the Docker host can ping the container over the bridge network. But, we can also verify the container can connect to the outside world too. Lets log into the container, install the ping program, and then ping www.github.com.

First, we need to get the ID of the container started in the previous step. You can run docker ps to get that.

docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
846af8479944        ubuntu              "sleep infinity"    7 minutes ago       Up 7 minutes                            heuristic_boyd

Next, lets run a shell inside that ubuntu container, by running docker exec -it <CONTAINER ID> /bin/bash.

docker exec -it yourcontainerid /bin/bash
root@846af8479944:/#

Next, we need to install the ping program. So, lets run apt-get update && apt-get install -y iputils-ping.

apt-get update && apt-get install -y iputils-ping

Lets ping www.github.com by running ping -c5 www.github.com

  ping -c5 www.github.com

PING www.docker.com (104.239.220.248) 56(84) bytes of data.
64 bytes from 104.239.220.248: icmp_seq=1 ttl=45 time=38.1 ms
64 bytes from 104.239.220.248: icmp_seq=2 ttl=45 time=37.3 ms
64 bytes from 104.239.220.248: icmp_seq=3 ttl=45 time=37.5 ms
64 bytes from 104.239.220.248: icmp_seq=4 ttl=45 time=37.5 ms
64 bytes from 104.239.220.248: icmp_seq=5 ttl=45 time=37.5 ms

--- www.docker.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4003ms
rtt min/avg/max/mdev = 37.372/37.641/38.143/0.314 ms

Finally, lets disconnect our shell from the container, by running exit.

exit

We should also stop this container so we clean things up from this test, by running docker stop <CONTAINER ID>.

docker stop yourcontainerid

This shows that the new container can ping the internet and therefore has a valid and working network configuration.

Step 4: Configure NAT for external connectivity

In this step we’ll start a new NGINX container and map port 8080 on the Docker host to port 80 inside of the container. This means that traffic that hits the Docker host on port 8080 will be passed on to port 80 inside the container.

NOTE: If you start a new container from the official NGINX image without specifying a command to run, the container will run a basic web server on port 80.

Start a new container based off the official NGINX image by running docker run --name web1 -d -p 8080:80 nginx.

docker run --name web1 -d -p 8080:80 nginx

Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
6d827a3ef358: Pull complete
b556b18c7952: Pull complete
03558b976e24: Pull complete
9abee7e1ef9d: Pull complete
Digest: sha256:52f84ace6ea43f2f58937e5f9fc562e99ad6876e82b99d171916c1ece587c188
Status: Downloaded newer image for nginx:latest
4e0da45b0f169f18b0e1ee9bf779500cb0f756402c0a0821d55565f162741b3e

Review the container status and port mappings by running docker ps.

docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                           NAMES
4e0da45b0f16        nginx               "nginx -g 'daemon ..."   2 minutes ago       Up 2 minutes        443/tcp, 0.0.0.0:8080->80/tcp   web1

The top line shows the new web1 container running NGINX. Take note of the command the container is running as well as the port mapping - 0.0.0.0:8080->80/tcp maps port 8080 on all host interfaces to port 80 inside the web1 container. This port mapping is what effectively makes the containers web service accessible from external sources (via the Docker hosts IP address on port 8080).

Now that the container is running and mapped to a port on a host interface you can test connectivity to the NGINX web server.

To complete the following task you will need the IP address of your Docker host. This will need to be an IP address that you can reach (e.g. your lab is hosted in Azure so this will be the instance’s Public IP - the one you SSH’d into). Just point your web browser to the IP and port 8080 of your Docker host. Also, if you try connecting to the same IP address on a different port number it will fail.

If for some reason you cannot open a session from a web broswer, you can connect from your Docker host using the curl 127.0.0.1:8080 command.

curl 127.0.0.1:8080

<!DOCTYPE html>
<html>
<Snip>
<head>
<title>Welcome to nginx!</title>
    <Snip>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

If you try and curl the IP address on a different port number it will fail.

NOTE: The port mapping is actually port address translation (PAT).

Section #3 - Overlay Networking

Step 1: The Basics

In this step you’ll initialize a new Swarm, join a single worker node, and verify the operations worked.

Run docker swarm init --advertise-addr $(hostname -i).

docker swarm init --advertise-addr $(hostname -i)

Swarm initialized: current node (rzyy572arjko2w0j82zvjkc6u) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-69b2x1u2wtjdmot0oqxjw1r2d27f0lbmhfxhvj83chln1l6es5-37ykdpul0vylenefe2439cqpf \
    10.0.0.5:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

In the first terminal copy the entire docker swarm join ... command that is displayed as part of the output from your terminal output. Then, paste the copied command into the second terminal.

docker swarm join \
>     --token SWMTKN-1-69b2x1u2wtjdmot0oqxjw1r2d27f0lbmhfxhvj83chln1l6es5-37ykdpul0vylenefe2439cqpf \
>     10.0.0.5:2377
This node joined a swarm as a worker.

Run a docker node ls to verify that both nodes are part of the Swarm.

docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
ijjmqthkdya65h9rjzyngdn48    node2   Ready   Active
rzyy572arjko2w0j82zvjkc6u *  node1   Ready   Active        Leader

The ID and HOSTNAME values may be different in your lab. The important thing to check is that both nodes have joined the Swarm and are ready and active.

Step 2: Create an overlay network

Now that you have a Swarm initialized it’s time to create an overlay network.

Create a new overlay network called “overnet” by running docker network create -d overlay overnet.

docker network create -d overlay overnet

wlqnvajmmzskn84bqbdi1ytuy

Use the docker network ls command to verify the network was created successfully.

docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
3430ad6f20bf        bridge              bridge              local
a4d584350f09        docker_gwbridge     bridge              local
a7449465c379        host                host                local
8hq1n8nak54x        ingress             overlay             swarm
06c349b9cc77        none                null                local
wlqnvajmmzsk        overnet             overlay             swarm

The new “overnet” network is shown on the last line of the output above. Notice how it is associated with the overlay driver and is scoped to the entire Swarm.

NOTE: The other new networks (ingress and docker_gwbridge) were created automatically when the Swarm cluster was created.

Run the same docker network ls command from the second terminal.

docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
55f10b3fb8ed        bridge              bridge              local
b7b30433a639        docker_gwbridge     bridge              local
a7449465c379        host                host                local
8hq1n8nak54x        ingress             overlay             swarm
06c349b9cc77        none                null                local

Notice that the “overnet” network does not appear in the list. This is because Docker only extends overlay networks to hosts when they are needed. This is usually when a host runs a task from a service that is created on the network. We will see this shortly.

Use the docker network inspect <network> command to view more detailed information about the “overnet” network. You will need to run this command from the first terminal.

docker network inspect overnet

[
    {
        "Name": "overnet",
        "Id": "wlqnvajmmzskn84bqbdi1ytuy",
        "Created": "0001-01-01T00:00:00Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": null
    }
]

Step 3: Create a service

Now that we have a Swarm initialized and an overlay network, it’s time to create a service that uses the network.

Execute the following command from the first terminal to create a new service called myservice on the overnet network with two tasks/replicas.

docker service create --name myservice \
--network overnet \
--replicas 2 \
ubuntu sleep infinity

ov30itv6t2n7axy2goqbfqt5e

Verify that the service is created and both replicas are up by running docker service ls.

docker service ls

ID            NAME       MODE        REPLICAS  IMAGE
ov30itv6t2n7  myservice  replicated  2/2       ubuntu:latest

The 2/2 in the REPLICAS column shows that both tasks in the service are up and running.

Verify that a single task (replica) is running on each of the two nodes in the Swarm by running docker service ps myservice.

docker service ps myservice

ID            NAME         IMAGE          NODE     DESIRED STATE  CURRENT STATE               ERROR  PORTS
riicggj5tuta  myservice.1  ubuntu:latest  node2  Running        Running about a minute ago
nlozn82wsttv  myservice.2  ubuntu:latest  node1  Running        Running about a minute ago

The ID and NODE values might be different in your output. The important thing to note is that each task/replica is running on a different node.

Now that the second node is running a task on the “overnet” network it will be able to see the “overnet” network. Lets run docker network ls from the second terminal to verify this.

docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
55f10b3fb8ed        bridge              bridge              local
b7b30433a639        docker_gwbridge     bridge              local
a7449465c379        host                host                local
8hq1n8nak54x        ingress             overlay             swarm
06c349b9cc77        none                null                local
wlqnvajmmzsk        overnet             overlay             swarm

We can also run docker network inspect overnet on the second terminal to get more detailed information about the “overnet” network and obtain the IP address of the task running on the second terminal.

docker network inspect overnet

[
    {
        "Name": "overnet",
        "Id": "wlqnvajmmzskn84bqbdi1ytuy",
        "Created": "2017-04-04T09:35:47.526642642Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "fbc8bb0834429a68b2ccef25d3c90135dbda41e08b300f07845cb7f082bcdf01": {
                "Name": "myservice.1.riicggj5tutar7h7sgsvqg72r",
                "EndpointID": "8edf83ebce77aed6d0193295c80c6aa7a5b76a08880a166002ecda3a2099bb6c",
                "MacAddress": "02:42:0a:00:00:03",
                "IPv4Address": "10.0.0.3/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "node1-f6a6f8e18a9d",
                "IP": "10.0.0.5"
            },
            {
                "Name": "node2-507a763bed93",
                "IP": "10.0.0.6"
            }
        ]
    }
]

You should note that as of Docker 1.12, docker network inspect only shows containers/tasks running on the local node. This means that 10.0.0.3 is the IPv4 address of the container running on the second node. Make a note of this IP address for the next step (the IP address in your lab might be different than the one shown here in the lab guide).

Step 4: Test the network

To complete this step you will need the IP address of the service task running on node2 that you saw in the previous step (10.0.0.3).

Execute the following commands from the first terminal.

docker network inspect overnet

[
    {
        "Name": "overnet",
        "Id": "wlqnvajmmzskn84bqbdi1ytuy",
        "Created": "2017-04-04T09:35:47.362263887Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "d676496d18f76c34d3b79fbf6573a5672a81d725d7c8704b49b4f797f6426454": {
                "Name": "myservice.2.nlozn82wsttv75cs9vs3ju7vs",
                "EndpointID": "36638a55fcf4ada2989650d0dde193bc2f81e0e9e3c153d3e9d1d85e89a642e6",
                "MacAddress": "02:42:0a:00:00:04",
                "IPv4Address": "10.0.0.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "node1-f6a6f8e18a9d",
                "IP": "10.0.0.5"
            },
            {
                "Name": "node2-507a763bed93",
                "IP": "10.0.0.6"
            }
        ]
    }
]

Notice that the IP address listed for the service task (container) running is different to the IP address for the service task running on the second node. Note also that they are on the same “overnet” network.

Run a docker ps command to get the ID of the service task so that you can log in to it in the next step.

docker ps

CONTAINER ID        IMAGE                                                                            COMMAND                  CREATED             STATUS              PORTS                           NAMES
d676496d18f7        ubuntu@sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535   "sleep infinity"         10 minutes ago      Up 10 minutes                                       myservice.2.nlozn82wsttv75cs9vs3ju7vs
<Snip>

Log on to the service task. Be sure to use the container ID from your environment as it will be different from the example shown below. We can do this by running docker exec -it <CONTAINER ID> /bin/bash.

docker exec -it yourcontainerid /bin/bash
root@d676496d18f7:/#

Install the ping command and ping the service task running on the second node where it had a IP address of 10.0.0.3 from the docker network inspect overnet command.

apt-get update && apt-get install -y iputils-ping

Now, lets ping 10.0.0.3.

root@d676496d18f7:/# ping -c5 10.0.0.3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
^C
--- 10.0.0.3 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 2998ms

The output above shows that both tasks from the myservice service are on the same overlay network spanning both nodes and that they can use this network to communicate.

Step 5: Test service discovery

Now that you have a working service using an overlay network, let’s test service discovery.

If you are not still inside of the container, log back into it with the docker exec -it <CONTAINER ID> /bin/bash command.

Run cat /etc/resolv.conf form inside of the container.

docker exec -it yourcontainerid /bin/bash

cat /etc/resolv.conf

search ivaf2i2atqouppoxund0tvddsa.jx.internal.cloudapp.net
nameserver 127.0.0.11
options ndots:0

The value that we are interested in is the nameserver 127.0.0.11. This value sends all DNS queries from the container to an embedded DNS resolver running inside the container listening on 127.0.0.11:53. All Docker container run an embedded DNS server at this address.

NOTE: Some of the other values in your file may be different to those shown in this guide.

Try and ping the “myservice” name from within the container by running ping -c5 myservice.

root@d676496d18f7:/# ping -c5 myservice
PING myservice (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.020 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.052 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=0.044 ms
64 bytes from 10.0.0.2: icmp_seq=4 ttl=64 time=0.042 ms
64 bytes from 10.0.0.2: icmp_seq=5 ttl=64 time=0.056 ms

--- myservice ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.020/0.042/0.056/0.015 ms

The output clearly shows that the container can ping the myservice service by name. Notice that the IP address returned is 10.0.0.2. In the next few steps we’ll verify that this address is the virtual IP (VIP) assigned to the myservice service.

Type the exit command to leave the exec container session and return to the shell prompt of your Docker host.

root@d676496d18f7:/# exit

Inspect the configuration of the “myservice” service by running docker service inspect myservice. Lets verify that the VIP value matches the value returned by the previous ping -c5 myservice command.

docker service inspect myservice

[
    {
        "ID": "ov30itv6t2n7axy2goqbfqt5e",
        "Version": {
            "Index": 19
        },
        "CreatedAt": "2017-04-04T09:35:47.009730798Z",
        "UpdatedAt": "2017-04-04T09:35:47.05475096Z",
        "Spec": {
            "Name": "myservice",
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "ubuntu:latest@sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535",
                    "Args": [
                        "sleep",
                        "infinity"
                    ],
<Snip>
        "Endpoint": {
            "Spec": {
                "Mode": "vip"
            },
            "VirtualIPs": [
                {
                    "NetworkID": "wlqnvajmmzskn84bqbdi1ytuy",
                    "Addr": "10.0.0.2/24"
                }
            ]
        },
<Snip>

Towards the bottom of the output you will see the VIP of the service listed. The VIP in the output above is 10.0.0.2 but the value may be different in your setup. The important point to note is that the VIP listed here matches the value returned by the ping -c5 myservice command.

Feel free to create a new docker exec session to the service task (container) running on node2 and perform the same ping -c5 service command. You will get a response form the same VIP.

Cleaning Up

Hopefully you were able to learn a little about how Docker Networking works during this lab. Lets clean up the service we created, the containers we started, and finally disable Swarm mode.

Execute the docker service rm myservice command to remove the service called myservice.

docker service rm myservice

Execute the docker ps command to get a list of running containers.

docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                           NAMES
846af8479944        ubuntu              "sleep infinity"         17 minutes ago      Up 17 minutes                                       heuristic_boyd
4e0da45b0f16        nginx               "nginx -g 'daemon ..."   12 minutes ago      Up 12 minutes       443/tcp, 0.0.0.0:8080->80/tcp   web1

You can use the docker kill <CONTAINER ID ...> command to kill the ubunut and nginx containers we started at the beginning.

docker kill yourcontainerid1 yourcontainerid2

Finally, lets remove node1 and node2 from the Swarm. We can use the docker swarm leave --force command to do that.

Lets run docker swarm leave --force on node1.

docker swarm leave --force

Lets also run docker swarm leave --force on node2.

docker swarm leave --force

Congratulations! You’ve completed this lab!

Continuous Integration With Docker Cloud

Fri, 20 Jan 2017 00:00:00 +0000

Difficulty: Beginner

Time: Approximately 20 minutes

In this lab, you will learn how to configure a continuous integration (CI) pipeline for a web application using Docker Cloud’s automated build features. You will complete the following tasks as part of the lab:

Task 0: Configure the prerequisites

Task 1: Configure Docker Cloud to Automatically Build Docker Images

Task 1.1: Configure Docker Cloud Autobuilds

Task 1.2: Trigger an Autobuild

What is Docker Cloud?

Docker Cloud is Docker’s cloud platform for individual developers and teams to build, ship, and run containerized applications. Docker Cloud enables teams to come together to collaborate on their projects and automate complex continuous delivery flows, which enables you to focus on improving your app, and leave the rest up to Docker Cloud.

Document conventions

When you encounter a phrase in between < and > you are meant to substitute in a different value.

For instance if you see ssh <username>@<hostname> you would actually type something like ssh ubuntu@node0-gvs0mgc0216.southcentralus.cloudapp.azure.com

You may be asked to SSH into various nodes. These nodes are referred to as node0, node1 etc. These tags correspond to the very beginning of the hostnames you will find in your welcome email.

Task 0: Prerequisites

In order to complete this lab, you will need the following:

A Docker ID
A management host (for this lab you’ll need a Linux named node0)
A GitHub account
Git installed
Docker installed at the command-line

Obtain a Docker ID

If you do not already have a Docker ID, you will need to create one now. Creating a Docker ID is free, and allows you to use Docker Cloud.

If you already have a Docker ID, skip to the next prerequisite.

To create a Docker ID:

Use your web browser to visit https://cloud.docker.com.
On the right hand side of the screen, enter your preferred Docker ID, supply your email address, and choose a password.
Click Sign up.
Check your email (including your spam folder) for a confirmation email.
Follow the steps outlined in the email.
You should be redirected back to https://cloud.docker.com.

You now have a Docker ID. Remember to keep the password safe and secure.

GitHub account

In order to complete the CI portions of this lab, you will need an account on GitHub. If you do not already have one, you can create one for free at GitHub.

Continue with the lab as soon as you have completed the prerequisites.

Task 1: Configure Docker Cloud to Automatically Build Docker Images

One of the most powerful features of Docker Cloud is the ability to define end-to-end CI/CD pipelines. In this part of the lab, you’re going to link your GitHub account to Docker Cloud to facilitate seamless application delivery.

Let’s start by linking your Docker Cloud account to your GitHub account:

Using your web browser, go to https://cloud.docker.com and sign in with your Docker ID.
Click the Cloud Settings link in the menu on the left hand side of the Docker Cloud web UI.

Note: If you cannot see menu on the left, un-select Swarm Mode at the top of the screen.

Scroll down to the Source providers section. Click the power plug icon next to GitHub, and follow the procedure to link your GitHub account.

Now that you’ve got Docker Cloud linked to your GitHub account, we’ll start by forking a demo repo.

In your web browser, navigate to https://github.com/Cloud-Demo-Team/voting-demo.git.
Click the Fork button in the upper right hand corner to create your own copy of the source repository.

Next, we’ll clone the repository into our local Docker environment. The following commands will be executed in the terminal or command window from Linux node0

At the command line, change into the directory where you want to clone code.

Clone the repository (you will need to have git installed and the git binary present in your PATH).

 $ git clone https://github.com/<your github user name>/voting-demo.git

 Cloning into 'voting-demo'...
 remote: Counting objects: 481, done.
 remote: Total 481 (delta 0), reused 0 (delta 0), pack-reused 481
 Receiving objects: 100% (481/481), 105.01 KiB | 0 bytes/s, done.
 Resolving deltas: 100% (246/246), done.
 Checking connectivity... done.

This will create a copy of the forked repo in a directory called voting-demo within your home directory.

Task 1.1: Configure Autobuilds

Docker Cloud can automatically build new images when updates are pushed to a repository on GitHub.

In this step, you’re going to build two GitHub repositories - one for the voting part of the app and one for the results part. You’ll configure two new repositories in Docker Cloud so that each time a change is pushed to the source repo an updated Docker image will be built.

In your web browser, return to Docker Cloud and click the Repositories link on the left hand side.
Click the + icon near the top right of the page and select Repository.
Enter the following information in the Create Repository section:
- Name: results
- Description: Results service for the Docker voting app
- Visibilty: public
In the Build Settings section, you should see that your GitHub account is connected. Click the GitHub icon.
Make sure the your appropriate GitHub organization is populated from the drop down list, and select voting-demo for repository.
Select “Click here to customize the build settings” to configure the build rules.
Click Create at the bottom of the page.

You will be taken to the repository page.

Right now Docker Cloud doesn’t let you specify the build context when you create a repository, so you need to update the settings

Navigate to the Builds page and click ‘Configure Automated Builds’. Scroll down to Build Rule, and set the Build Context to /results:

Create a second repository

Repeat steps 1-8 with the following modifications:

Create Repo (Step 3)

Name: voting
Description: Voting service for the Docker voting app

Specify the Dockerfile path (in Step 7):

Enter /voting/Dockerfile for the Dockerfile Path

Check to make sure the repositories were created

If you click the Repositories menu on the left you should see both the voting and results respositories were created.

Well done! You’ve created two new repos and configured them to autobuild whenever new changes are pushed to the associated GitHub repos.

Task 1.2: Trigger an Autobuild

Switch back the command line of your VM.

If you have not already log into your Azure VM. For example (be sure to use the actual node name supplied in your email):

ssh ubuntu@node0-gvs0mgc0216.southcentralus.cloudapp.azure.com
Change to the directory containing the voting app.
```
 $ cd ~/voting-demo/voting
```
Use vi or your favorite text editor to open app.py.
- To use vi on Linux: $ vi app.py
Scroll down to find the lines containing optionA and optionB, and change Dev and Ops to Futbol and Soccer.
```
 optionA = "Futbol"
 optionB = "Soccer"
```
Save your changes.

Commit changes to the repository and push to GitHub using git add, git commit, and git push.

 $ git add *
 $ git commit -m "changing the voting options"
 [master 2ab640a] changing the voting options
 		1 file changed, 3 insertions(+), 2 deletions(-)
 		$ git push origin master
 		Counting objects: 4, done.
 Delta compression using up to 8 threads.
 Compressing objects: 100% (4/4), done.
 Writing objects: 100% (4/4), 380 bytes | 0 bytes/s, done.
 Total 4 (delta 3), reused 0 (delta 0)
 To https://github.com/<your github repo>/voting-demo.git
	c1788a1..2ab640a  master -> master

Note: You may be prompted to set your email and name when you attempt to commit your changes. If this is the case, simply follow the instructions provided on your screen.

Note: If you have two factor authentication (2FA) configured on your GitHub account you will need to enter your personal access token (PAT) instead of your password when prompted.

In the Docker Cloud web UI, navigate back to the voting repo and notice that the status is BUILDING.

Note: It can take several minutes for a build job to complete.
Click the Timeline tab near the top of the screen.
Click Build in master:/voting.

Here you can see the status of the build process:

Congratulations! You have successfully configured Docker Cloud to automatically build a new Docker image each time you push a change to your application’s repository on GitHub.

Learn More

To learn more about Docker Cloud’s continuous integration (CI) capabilities and how to bring more automation and collaboration to your application pipeline, check out:

Video Automated Builds with Docker Cloud
Docs: Automated Builds

Windows Containers Multi-Container Applications

Fri, 20 Jan 2017 00:00:00 +0000

Multi-Container Applications

This tutorial will walk you through using the sample Music Store application with Windows containers. The Music Store application is a standard .NET sample application, available in the aspnet GitHub repository. We’ve forked it to use Windows Containers.

Using docker-compose on Windows

Docker Compose is a great way develop complex multi-container consisting of databases, queues and web frontends.

To develop with Docker Compose on a Windows Server 2016 system, install compose too (this is not required on Windows 10 with Docker for Windows installed):

Invoke-WebRequest https://github.com/docker/compose/releases/download/1.11.1/docker-compose-Windows-x86_64.exe -UseBasicParsing -OutFile $env:ProgramFiles\docker\docker-compose.exe

To try out Compose on Windows, clone a variant of the ASP.NET Core MVC MusicStore app, backed by a SQL Server Express 2016 database.

git clone https://github.com/dockersamples/dotnet-musicstore.git
...
cd dotnet-musicstore
docker-compose -f .\docker-compose.windows.yml build
...
docker-compose -f .\docker-compose.windows.yml up
...

To access the running app from the host running the containers (for example when running on Windows 10 or if opening browser on Windows Server 2016 system running Docker engine) use the container IP and port 5000. localhost will not work:

docker inspect -f {{ .NetworkSettings.IPAddress }} dotnet-musicstore_web_1
172.21.124.54

If using Windows Server 2016 and accessing from outside the VM or host, simply use the VM or host IP and port 5000.

What’s happening here?

Take a closer look at the docker-compose.windows.yml file.

version: '2.1'
services:
  db:
    image: microsoft/mssql-server-windows-express
    environment:
      sa_password: "Password1"
    ports:
      - "1433:1433" # for debug. Remove this for production

  web:
    build:
      context: .
      dockerfile: Dockerfile.windows
    environment:
      - "Data:DefaultConnection:ConnectionString=Server=db,1433;Database=MusicStore;User Id=sa;Password=Password1;MultipleActiveResultSets=True"
    depends_on:
      - "db"
    ports:
      - "5000:5000"

networks:
  default:
    external:
      name: nat

You can find more details in the Docker Compose documentation, but basically here’s what is happening.

Two services are defined, db and web.
db is a Microsoft SQL Express image official image from Microsoft.
- The password for db is set to Password1 (obviously only for a developer environment).
- Port 1433 on the host is mapped to the exposed port 1433 in the container which is used for debugging.
web is built from Dockerfile.windows. - Compose passes along an environment variable which defines where the database is and how to connect to it. Notice that we can just refer to the database as db and Compose will allow web to discover the service there. - The port 5000 is mapped to the exposed port 5000 in the container.
The two services are added to an existing Docker network, named nat.

Let’s look at Dockerfile.windows to understand it a bit better.

FROM microsoft/dotnet:1.1-sdk-msbuild-nanoserver

This pulls in the official microsoft .NET Core image based on Nano Server

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop';"]

This sets the shell to powershell.

RUN set-itemproperty -path 'HKLM:\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters' -Name ServerPriorityTimeLimit -Value 0 -Type DWord

This disables the Windows DNS cache. The cache is too aggressive, and turning it off ensures the web container always queries the DNS server running in Docker to get the address of the database container.

RUN New-Item -Path \MusicStore\samples\MusicStore.Standalone -Type Directory
WORKDIR MusicStore

This creates a new directory in the container and makes it the working directory. Everything else that happens after this point will use MusicStore as the base directory.

COPY samples/MusicStore.Standalone/project.json samples/MusicStore.Standalone/project.json
COPY NuGet.config .

The command adds the project file (detailing NuGet package dependencies) and the NuGet config file.

RUN dotnet restore --no-cache .\samples\MusicStore.Standalone

This pulls in the right dependencies to the project.

COPY samples samples
RUN dotnet build .\samples\MusicStore.Standalone

This adds the rest of the app source code to the container, and compiles the project.

EXPOSE 5000
ENV ASPNETCORE_URLS http://0.0.0.0:5000
CMD dotnet run -p .\samples\MusicStore.Standalone

These last three commands expose correct ports and set the default run command for the container to run the MusicStore app.

Docker for IT Pros and System Administrators Stage 3

Fri, 20 Jan 2017 00:00:00 +0000

This final stage will help you

Implement a full proof of concept application
Develop a strategy for integrating Docker into your existing production environment
Become recognized as a leader in your organization on implementing Docker

In this stage we have a series of design and reference architectures that help you understand the best ways to structure your apps with Docker. And a good video overview of the process at the end.

Design and Reference Architectures

Docker EE DDC Implementation Considerations

Service Discovery and Load Balancing with Docker Universal Control Plane

An Introduction to Storage Solutions for Docker CaaS

Docker Reference Architecture: Designing Scalable, Portable Docker Container Networks

Docker Reference Architecture: Securing Docker EE and Security Best Practices

Docker Reference Architecture: Docker EE Best Practices and Design Considerations

Docker Reference Architecture: Development Pipeline Best Practices Using Docker EE

Next Steps

Once you’ve finished the three stages, you can

Docker for IT Pros and System Administrators Stage 2

Fri, 20 Jan 2017 00:00:00 +0000

This stage will help you

Understand the architecture of Docker, and the core features
Understand how to integrate Docker into your existing application infrastructure
Develop a proof of concept application deployment

Hands-on Learning

Dig deeper into some hands-on learning in the browser

And for a bonus, you can work through a comprehensive Orchestration Workshop

Part 1

Part 2

Bonus materials

As a bonus, check out these videos. The first video is from DockerCon 2017 in Austin and features Brandon Royal showing you how to port traditional monolithic applications to modern Docker based applications.

Next we have Nico Kabar discussing design considerations, tools and best practices to address a globally distributed hybrid cloud infrastructure with hundreds or even thousands of apps using Docker.

And in this playlist, learn from Docker Customers how they use Docker at their companies.

Docker for IT Pros and System Administrators Stage 1

Fri, 20 Jan 2017 00:00:00 +0000

This stage will

Get you familiar with the core concepts of Docker
Help you understand the fundamental value proposition for Docker
Help you see how Docker can help your organization

For an introduction to Docker specifically for sys admins and IT Pros, check out Mike Coleman’s talk Docker?!? But I’m a SYSADMIN!

Hands-on Learning

Give Docker a try with these two beginning tutorials:

Docker for Beginners - Linux

Deploy a multi-service application

Docker for Developers Stage 3

Fri, 20 Jan 2017 00:00:00 +0000

This final stage will help you

Deploy an application to a staging environment
Manage your staging environment with Docker Swarm Mode
Learn how to build a secure application

In this stage you’ll learn from the experts as we have a variety of videos with in depth content.

Next Steps

Once you’ve finished the three stages, you can

Docker for Developers Stage 2

Fri, 20 Jan 2017 00:00:00 +0000

This stage show you how to

This stage show you how to incorporate Docker into your entire developer workflow.

Self-guided on desktop labs

In-container development - learn how to integrate Docker into your IDE
Windows Containers
- Windows Container Setup Setup up your environment to work with Windows Containers
- Windows Container Basics
- Windows Containers Multi-Container Applications

Videos

In the following video, Docker Captain John Zaccone describes how Docker can help with the entire developer workflow, including setting up a continuous integration and staging environment

In this video, Abby Fuller goes deep into creating effective Docker Images

Docker for Developers Stage 1

Fri, 20 Jan 2017 00:00:00 +0000

This stage will

Get you familiar with the core concepts of Docker
Show you how to build and deploy basic applications

Self-guided in-browser tutorials

Modernizing Traditional Java Applications

Sun, 01 Jan 2017 00:00:00 +0000

In this Hands-on Lab, we will show you how to take a traditional Java EE app, and compile it to run in a lightweight Java container. The app we’ve chosen is an old Java EE demo called Movie Plex 7 that was originally written to run in Wildfly 3. You can find our fork of the repo in our javaee-demo repository on GitHub. In fact, if you were to take that repository, and lift the movieplex7-1.0-SNAPSHOT file, you could run that in a Java Application Server, such as WebLogic, Wildfly, or WebSphere.

If you aren’t yet familiar with the basic concepts of Docker, you may wish to start with the Docker 101 lab before running this lab.

Task 0: The Play with Docker Lab Environment
Task 1: Building the App in a Container
Task 2: Adding a New Front-end
Task 3: Moving from Wildfly to Tomcat EE

Step 0: The Play with Docker Lab Environment

The terminal window on the right hand side of this web page is a Linux-based Docker host running in the Play with Docker (PWD) lab environment. This is where you will run all of the commands in this lab.

There are two ways of running commands in the terminal:

Single-clicking any command will paste it into the terminal and execute it
You can manually type each command into the terminal

Step 1: Building the App in a Container

In this section of the lab, you are going to take a Java EE web app running on Wildfly, move it to Tomcat EE, and run it in a Docker container.

The easiest way to move an enterprise Java app into Docker is create a WAR file and deploy it to the app server.

Clone the lab repo into the PWD lab. Remember, you can click the command or manually type it into the terminal window.

git clone https://github.com/dockersamples/javaee-demo
cd javaee-demo

You will build the application using Maven and deploy it to container to see how it looks. Building and deploying the application can be done without having either Maven or an application server on your computer.

First, look at how the original application works. You can do this by building the application in Docker using a Wildfly container which was the app server the project first used. Although the GitHub repo contains an EAR file, you will build the application from source and deploy it using a multi-stage build Dockerfile. Here’s the text of the two-stage Dockerfile.wildfly file that you can find in the movieplex7 directory.

FROM maven:3-jdk-7 AS app
WORKDIR /usr/src/movieplex7
COPY pom.xml .
RUN mvn -B -f pom.xml -s /usr/share/maven/ref/settings-docker.xml dependency:resolve
COPY . .
RUN mvn -B -s /usr/share/maven/ref/settings-docker.xml package -DskipTests

FROM jboss/wildfly
RUN /opt/jboss/wildfly/bin/add-user.sh admin Admin --silent
COPY --from=app /usr/src/movieplex7/target/movieplex7-1.0-SNAPSHOT.war .
EXPOSE 8080 9990
CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]

The first part of the Dockerfile uses a maven container to build the war file. Instead of building the application locally, it is built inside the container with the tool chain needed to compile the application. In the second part of the Dockerfile, you use a Wildfly container and copy the war file built in the previous container to the app server for deployment.

There are a few additional commands to expose the ports and start Wildfly.

Here’s how to build the app:

cd movieplex7
docker image build -t movieplex7 -f Dockerfile.wildfly .

This creates a Docker image with the application deployed on Wildfly.

To run the application:

docker container run -d -p 8080:8080 --name movieplex7 movieplex7

Click here to see it running

Your app is working just fine in Docker now, with no code changes and running in the original app server. Feel free to play around with the functionality of the site to understand the app better.

Step 2: Adding a New Front End

One of the big advantages of containers is that they can ease the process of updating older applications by replacing parts of the application as needed. Java EE also makes it easy to consume the output of your application in different ways. This allows you to easily add a different client to an application.

React Client

In this section, you will update the Movie Plex 7 application by adding more attributes to the movie entity and also updating the presentation layer by writing a new client using React. Don’t worry, all the code is provided, you don’t need to know React.

As you saw earlier, the JavaServer Faces client was rather sparse and old-fashioned looking. You want to make the movie listing to include movie posters as well as more information about each movie. To do that, you will add a few attributes to the Movie entity, and include a path to a movie poster, and more information about the cast and the movie rating.

The Movie Plex 7 application includes a REST interface for querying movies. This simplifies writing a new client because you can use the API and get data via REST. The javaScript client is a bit more descriptive than the original JavaServer Faces client.

To get a separation of the new client from the existing client and server, you will deploy the React client in a separate container.

To get ready, stop and remove the running container, and then change the directory from movieplex7 to the root of the project. Due to a quirk in how React is built, and the base url of the Play with Docker lab, before you build the React client you need to run a script to inject the host URL into the Dockerfile:

docker container stop movieplex7
docker container rm movieplex7
cd ..
./add_ee_pwd_host.sh
more react-client/Dockerfile

FROM node:latest
ENV API_HOST=pwd10-0-16-3-8080.host4.labs.play-with-docker.com
WORKDIR /usr/src/react-client
COPY . .
RUN npm install
RUN npm run build
EXPOSE 3000
CMD ["npm", "run", "start"]

React uses Node.js to build a static site for the interface. The Dockerfile is much simpler than the one for the movieplex7 app. It uses a single-stage build, which passes the environment variable API_HOST to the builder. When it runs, it will start a simple Node server that serves the React pages, and exposes them on port 3000. We’ll deploy it in the next section.

Step 3: Moving from Wildfly to Tomcat EE

In Step 1, you compiled and deployed the Movie Plex 7 application from source by building it in a Maven container and then deploying it in a Wildfly container. The JavaServer Faces front-end was integrated into the the same application image.

In this step you will make three changes. First, as described in the previous step, you will use a React front-end deployed in a second container. Second, you will deploy the app in a Tomcat EE application server instead of Wildfly. Last, you will deploy both at the same time using a Docker Compose file.

The new front-end is described in the last section, so take a look at the new Dockerfile that deploys your app using the Tomcat Server.

more movieplex7/Dockerfile

FROM maven:latest AS app
WORKDIR /usr/src/movieplex7
COPY pom.xml .
RUN mvn -B -f pom.xml -s /usr/share/maven/ref/settings-docker.xml dependency:resolve
COPY . .
RUN mvn -B -s /usr/share/maven/ref/settings-docker.xml package -DskipTest

FROM tomee:8-jre-7.0.2-plume

# tomcat-users.xml sets up user accounts for the Tomcat manager GUI
# and script access for Maven deployments
WORKDIR /usr/local/tomee/
ADD tomcat/tomcat-users.xml conf/
ADD tomcat/web.xml conf/
# copy and deploy application
WORKDIR /usr/local/tomee/webapps/
COPY --from=app /usr/src/movieplex7/target/movieplex7-1.0-SNAPSHOT.war .
# start tomcat7
EXPOSE 8080
CMD ["catalina.sh", "run"]

You’ll see that the first stage is very similar to Dockerfile.wildfly. Only instead of using maven:3-jdk-7 as the base image, it uses maven:latest. There’s still no code change, you are still compiling the code into a WAR file. The second stage is a bit different, because it uses a different application server. But the end result is the same application running on both.

You now have two Dockerfiles, one to build the Java Movie Plex 7 application and another to build the React javaScript client. Running a build for each will result in two images. You can individually run each image with the appropriate flags, but a simpler and more consistent solution is to use Docker Compose to start both images as containers that are connected by a network defined in Docker. Another nice thing about Docker Compose is that you can also build the images without having to do a separate docker image build command for each image. This is the docker-compose.yml file in the project root:

version: "3.3"

services:
  movieplex7:
    build:
      context: ./movieplex7
    image: movieplex7-tomee
    ports:
      - "8080:8080"
    networks:
      - www

  react-client:
    build:
      context: ./react-client
    image: react-client
    ports:
      - "80:3000"
    networks:
      - www

networks:
    www:

The Compose file defines each service and the ports that they are mapped to. Of note is the react-client, which maps the public port 80 to the private in-container port of 3000.

To run and build the images use:

docker-compose up --build -d

Once it’s completed building, which the first time can take awhile, we can check on the status of the containers:

docker container ls

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
02c1ee2be5ff        movieplex7-tomee    "catalina.sh run"   26 minutes ago      Up 26 minutes       0.0.0.0:8080->8080/tcp   javaeedemo_movieplex7_1
2a65e6f08819        react-client        "npm run start"     26 minutes ago      Up 26 minutes       0.0.0.0:80->3000/tcp     javaeedemo_react-client_1

This shows that the containers are running. Since there is overhead to start and deploy the war file you can look at the log files of the movieplex7 container to check if it’s started

docker logs javaeedemo_movieplex7_1

...
02-Oct-2017 21:01:22.069 INFO [main] sun.reflect.DelegatingMethodAccessorImpl.invoke Starting ProtocolHandler [http-nio-8080]
02-Oct-2017 21:01:22.084 INFO [main] sun.reflect.DelegatingMethodAccessorImpl.invoke Starting ProtocolHandler [ajp-nio-8009]
02-Oct-2017 21:01:22.088 INFO [main] sun.reflect.DelegatingMethodAccessorImpl.invoke Server startup in 15073 ms

Now that the application server has started, you can click here and try out the new interface.

And the old interface is still there, you can see it by clicking on here.

Play with Docker classroom

Medical physics with Geant4 + Docker = ❤️

Simulating the building blocks of Reality

Particle Physics in Medicine

Geant4 and Medical Physics

Download the simulation code from Github

First simulation

More simulations

Antimatter to the rescue!

Light seems to zigzag!

Superman (o Supergirl)-approved shielding!

Beware of radiation!

Ionization

Simulation limits and photoelectric absorption

Energy is a kind of matter

Final words

.NET Conf 2019

Task 1: Grab the code

Clone the source code from GitHub

Task 2: Build the application image

Task 3: Run a .NET Core container!

You’re done!

Docker for Beginners - Linux

Task 0: Prerequisites

Clone the Lab’s GitHub Repo

Make sure you have a DockerID

Task 1: Run some simple Docker containers

Run a single task in an Alpine Linux container

Run an interactive Ubuntu container

Run a background MySQL container

Task 2: Package and run a custom app using Docker

Build a simple website image

Task 3: Modify a running website

Start our web app with a bind mount

Modify the running website

Update the image

Test the new version

Push your images to Docker Hub

Next Step

Application Containerization and Microservice Orchestration

Stage Setup

Step 0: Basic Link Extractor Script

Step 1: Containerized Link Extractor Script

Step 2: Link Extractor Module with Full URI and Anchor Text

Step 3: Link Extractor API Service

Step 4: Link Extractor API and Web Front End Services

Step 5: Redis Service for Caching

Step 6: Swap Python API Service with Ruby

Conclusions

Node.js with SQL Server on Docker

Task 0: Prerequisites

Clone the source code from GitHub

Save your Docker ID

Task 1: Run v1 of the app in a container

Task 2: Add a SQL Server database container for storage

Task 3: Switch to high availability in swarm mode

Task 4: Add a reverse proxy to improve performance

Task 5: Add monitoring and an application dashboard

Cleanup

Docker for IT Pros and System Administrators Stage 1

Hands-on Learning

More Reading: Ebook

Doing More With Docker Images

Image creation from a container

Image creation using a Dockerfile

Image layers

Image Inspection

Terminology

Footnotes

First Alpine Linux Containers

1.0 Running your first container

1.1 Docker Images

1.1 Docker Container Run

1.2 Container Isolation

1.3 Terminology

Swarm Mode Introduction for IT Pros

The application

Initialize Your Swarm

Show Swarm Members

Clone the Voting App