DEV Community: Nicola Apicella

Golang error updates

Nicola Apicella — Fri, 29 Sep 2023 15:12:09 +0000

Since go 1.20, golang has improved the ergonomics of error handling. Noteworthy improvements include:

errors.Is and errors.As have been updated to work on tree of errors
fmt.Errorf accepts multiple %w format verbs, allowing to join multiple errors
The Unwrap []error function allows traversing through tree of errors

Follow some tests demonstrating the practical application of these enhancements (also available in the golang playground):

package main

import (
    "errors"
    "fmt"
    "testing"
)

func TestErrorfForWrapping(t *testing.T) {
    err1 := errors.New("a")
    err2 := errors.New("b")
    // the parentheses in the string are arbitrary
    wrap := fmt.Errorf("%w (%w)", err1, err2)

    if errors.Is(wrap, err1) == false {
        t.Fatal()
    }
    if errors.Is(wrap, err2) == false {
        t.Fatal()
    }
}

func TestStatusCodeInErrors(t *testing.T) {
    var (
        //
        // client errors
        //
        clientError = errors.New("client error")
        //
        // client error causes
        //
        unauthorized     = errors.New("unauthorized")
        resourceNotFound = errors.New("resource not found")

        //
        // server errors
        //
        serverError = errors.New("server error")
        //
        // server error causes
        //
        badGatewayError     = errors.New("badGateway")
        internalServerError = errors.New("internal server error")
    )

    getStatusCode := func(err error) (int, string) {
        if errors.Is(err, clientError) {

            if errors.Is(err, unauthorized) {
                return 400, "unauthorized"
            }
            if errors.Is(err, resourceNotFound) {
                return 400, "resource not found"
            }
            return 400, "no error type"

        } else if errors.Is(err, serverError) {

            if errors.Is(err, badGatewayError) {
                return 500, "bad gateway"
            }
            if errors.Is(err, internalServerError) {
                return 500, "internal server error"
            }
        }
        return 500, "no error type"
    }

    err := fmt.Errorf("%w (%w)", clientError, unauthorized)

    if code, mex := getStatusCode(err); code != 400 || mex != "unauthorized" {
        t.Fatal()
    }

    err = fmt.Errorf("%w (%w)", clientError, resourceNotFound)
    if code, mex := getStatusCode(err); code != 400 || mex != "resource not found" {
        t.Fatal()
    }

    err = fmt.Errorf("%w (%w)", serverError, internalServerError)
    if code, mex := getStatusCode(err); code != 500 || mex != "internal server error" {
        t.Fatal()
    }
}

type customError struct {
    wrapped []error
}

func (t *customError) Unwrap() []error {
    return t.wrapped
}
func (t *customError) Error() string {
    return "custom error"
}

func TestCustomErrorWrapping(t *testing.T) {
    err1 := errors.New("a")
    err2 := &customError{wrapped: []error{err1}}
    // it's much more convenient to do this
    // wrap := fmt.Errorf("%w (%w)", err1, err2) and you do not need to implement Unwrap

    if errors.Is(err2, err1) == false {
        t.Fatal()
    }
    if errors.Is(err2, err2) == false {
        t.Fatal()
    }
    if errors.Is(err2.Unwrap()[0], err1) == false {
        t.Fatal()
    }
}

What is a containerd snapshotter?

Nicola Apicella — Tue, 14 Feb 2023 17:38:45 +0000

I have recently invested some time to understand how containerd works, in particular the containerd snapshotters. During the process, I took various notes that helped me along the way. Shortly after, I realized my notes could make for a boring article for whoever is curious about containers, containerd, containerd snapshotter and would like to know more. The main questions this article tries to answer are: What is a containerd snapshotter? And how does it work?
The article is a bit long, so make sure you are comfortable before diving in.

Disclaimer: I make no promise that these notes are 100% accurate or complete. The opposite, I am sure they are not, but they should be an ok account of how things work.

Concepts

The section contains concepts mentioned in the containerd documentation and code that we will be utilizing in the rest of this document. I found that once I better understood them, everything else was much easier to digest.

Serialized Mount
It's a representation of the parameters of a Linux mount command/syscall. For example, if you were to mount a device on your system you will type something like this:

mount -t <type> -o <options if any> <device> <mount-point>

The serialization of the mount is represented in containerd with a go struct:

// Mount is the lingua franca of containerd. A mount represents a
// serialized mount syscall. Components either emit or consume mounts.
type Mount struct {
    // Type specifies the host-specific of the mount.
    Type string
    // Source specifies where to mount from. Depending on the host system, this
    // can be a source path or device.
    Source string
    // Options contains zero or more fstab-style mount options. Typically,
    // these are platform specific.
    Options []string
}

func All(mounts []Mount, target string) error {
// omitted

Note that the target is missing from the struct because it's up to the caller to decide the target dir of the mount.

Side note: the Target field was recently added (Feb 2023 circa) as an optional field to the struct. The snapshotters currently baked in containerd do not use/do not need it so for now/rest of the document we will just say that the field was added because some future snapshotters might need it. The github issue and PR contain more details on the reasoning

containerd architecture

containerd is a daemon that can be interacted with via socket with grpc calls. What we will be focusing on in this document are the Snapshotter Service and the Snapshotter module.
containerd supports some default snapshotters (overlay, native, btrfs, etc.) but also allows the implementation of new snapshotters without having to recompile the containerd binary, by building a snapshotter plugin.
containerd uses a plugin architecture for the Snapshotter. To build one, you need to build a grpc service exposed via a socket that can receive requests from containerd. If you build yours in golang, this is much easier since containerd provides bindings for grpc, so "all needed" is to implement an interface and pass that to the grpc server binder.
To make it more concrete, this is how creating a snapshotter looks like in code (error handling omitted for clarity, the complete example is available in the containerd repo):

package main

import (
    "net"

    "google.golang.org/grpc"

    snapshotsapi "github.com/containerd/containerd/api/services/snapshots/v1"
    "github.com/containerd/containerd/contrib/snapshotservice"
)

func main() {
    // Create a gRPC server
    rpc := grpc.NewServer()

    // Configure your custom snapshotter
    sn, _ := NewMyCoolSnapshotter()
    // Convert the snapshotter to a gRPC service,
    service := snapshotservice.FromSnapshotter(sn)
    // Register the service with the gRPC server
    snapshotsapi.RegisterSnapshotsServer(rpc, service)

    // Listen and serve
    l, _ := net.Listen("unix", "/var/my-cool-snapshooter-socket")
    _ := rpc.Serve(l)
}

Note: when using a custom plugin snapshotter, ctr snapshot list does not show the snapshots that were created by the custom snapshotter.
To see the snapshots created by the plugin, use the --snapshotter flag, e.g. ctr snapshot --snapshotter mysnapshotter list.

The Snapshots Service (also depicted in the architecture diagram above) is the component that takes care of redirecting requests to the right Snapshotter among other things.

NewMyCoolSnapshotter needs to return a type that implements the Snapshotter interface. How this interface works is the focus of the rest of the document:

type Snapshotter interface {
    // Stat returns the info for an active or committed snapshot by name or
    // key.
    //
    // Should be used for parent resolution, existence checks and to discern
    // the kind of snapshot.
    Stat(ctx context.Context, key string) (Info, error)

    // Update updates the info for a snapshot.
    //
    // Only mutable properties of a snapshot may be updated.
    Update(ctx context.Context, info Info, fieldpaths ...string) (Info, error)

    // Usage returns the resource usage of an active or committed snapshot
    // excluding the usage of parent snapshots.
    //
    // The running time of this call for active snapshots is dependent on
    // implementation, but may be proportional to the size of the resource.
    // Callers should take this into consideration. Implementations should
    // attempt to honer context cancellation and avoid taking locks when making
    // the calculation.
    Usage(ctx context.Context, key string) (Usage, error)

    // Mounts returns the mounts for the active snapshot transaction identified
    // by key. Can be called on an read-write or readonly transaction. This is
    // available only for active snapshots.
    //
    // This can be used to recover mounts after calling View or Prepare.
    Mounts(ctx context.Context, key string) ([]mount.Mount, error)

    // Prepare creates an active snapshot identified by key descending from the
    // provided parent.  The returned mounts can be used to mount the snapshot
    // to capture changes.
    //
    // If a parent is provided, after performing the mounts, the destination
    // will start with the content of the parent. The parent must be a
    // committed snapshot. Changes to the mounted destination will be captured
    // in relation to the parent. The default parent, "", is an empty
    // directory.
    //
    // The changes may be saved to a committed snapshot by calling Commit. When
    // one is done with the transaction, Remove should be called on the key.
    //
    // Multiple calls to Prepare or View with the same key should fail.
    Prepare(ctx context.Context, key, parent string, opts ...Opt) ([]mount.Mount, error)

    // View behaves identically to Prepare except the result may not be
    // committed back to the snapshot snapshotter. View returns a readonly view on
    // the parent, with the active snapshot being tracked by the given key.
    //
    // This method operates identically to Prepare, except that Mounts returned
    // may have the readonly flag set. Any modifications to the underlying
    // filesystem will be ignored. Implementations may perform this in a more
    // efficient manner that differs from what would be attempted with
    // `Prepare`.
    //
    // Commit may not be called on the provided key and will return an error.
    // To collect the resources associated with key, Remove must be called with
    // key as the argument.
    View(ctx context.Context, key, parent string, opts ...Opt) ([]mount.Mount, error)

    // Commit captures the changes between key and its parent into a snapshot
    // identified by name.  The name can then be used with the snapshotter's other
    // methods to create subsequent snapshots.
    //
    // A committed snapshot will be created under name with the parent of the
    // active snapshot.
    //
    // After commit, the snapshot identified by key is removed.
    Commit(ctx context.Context, name, key string, opts ...Opt) error

    // Remove the committed or active snapshot by the provided key.
    //
    // All resources associated with the key will be removed.
    //
    // If the snapshot is a parent of another snapshot, its children must be
    // removed before proceeding.
    Remove(ctx context.Context, key string) error

    // Walk will call the provided function for each snapshot in the
    // snapshotter which match the provided filters. If no filters are
    // given all items will be walked.
    // Filters:
    //  name
    //  parent
    //  kind (active,view,committed)
    //  labels.(label)
    Walk(ctx context.Context, fn WalkFunc, filters ...string) error

    // Close releases the internal resources.
    //
    // Close is expected to be called on the end of the lifecycle of the snapshotter,
    // but not mandatory.
    //
    // Close returns nil when it is already closed.
    Close() error
}

containerd "smart" client
containerd provides an opinionated client (written in go) which abstract grpc calls to containerd as well as providing utilities, that's why it's called a "smart" client. The ctr command line, uses the client to interact with containerd and its utilities. Other tools that want to build on top of containerd, e.g. nerdctl leverage the smart client utils to perform operations that otherwise they would need to implement themself.
The main point and the reason why I am mentioning the smart client is that I was expecting to find some of the snapshot-related operations in containerd (the agent), but they were not. In particular, at the beginning of my research, I thought snapshotters (or containerd) would perform the download of the layers, before discovering that the download is implemented as util in the "smart client".

DiffService
DiffService also referred to as DiffApplierService or simply DiffApplier is a service registered in containerd. The service (applies) untar the content of the layer into the mount.
This will be much clear later in the doc.

What's a snapshotter?

The main job of a snapshotter is to create a folder that can be used by containerd to unpack a layer. The folder must be prepared so that the act of unpacking layer n (applying in the containerd terminology) into the folder results in having a folder with the content of the layer n plus all other layers that proceeds the current one (n-1...1). The resulting folder is then a filesystem snapshot for the layers 1...n.
During the process of creating the filesystem snapshot, the snapshotter also stores some metadata in the containerd metadata storage to reflect the status of the snapshot.

Let's give an example. The client calls the snapshotter for the first time to "Prepare" a Snapshot for the first layer. Since this is the first layer, it does not have a parent Snapshot. So the snapshotter will create an empty dir, store the key and parent for this snapshot in the containerd metadata storage and return the dir to the caller (it does not return "just" a dir, but a Mount array. Bear with me a little longer). The client will then call the DiffApplier which untar the layer and merges it with the dir returned by the snapshotter (which was an empty dir). After this step, the dir contains the untarred first layer.

At this point, the client is ready for the second layer. It calls the snapshotter, this time providing a parent. The snapshotter will create an empty dir, then will store the key and parent for this snapshot (exactly as before).
This time since the parent is not empty, the snapshotter will take an additional action: copy all the files from the parent dir (which is the one that contains the uncompressed layer) to the empty dir. Finally, it returns the dir to the caller.
The client will then call the DiffApplier which uncompresses the layer and merge it with the dir returned by the snapshotter. This dir now contains the untarred first layer merged with the untarred second layer.
The process goes on for all the layers, and the result is going to be a snapshot that contains all the layers untarred.
Note how in the process we created N Snapshot, with N being the number of layers. The first snapshot contains one untarred layer, the second snapshot contains the first and the second, etc.
This seems to be a little space inefficient. If you have an image with 4 layers, each one 10MB uncompressed, the snapshotter will be creating
4 Snapshots with (10MB, 20MB, 30MB and 40MB respectively) for a total of 100 MB. In other words, we started with an image with 40MB big uncompressed and booked 100MB worth of space on the device.
What we have just described is the "native" snapshotter. Other snapshotters utilize different strategies that will mitigate or completely eliminate space inefficiency. We are going to look into one of them later, the overlay snapshotter.

Before going a little deeper, there is an easy way to see this space inefficiency issue. Create a docker file with 4 layers.

FROM alpine:latest  # alpine image is made of a single layer
RUN dd if=/dev/zero of=file_a bs=1024 count=10240 # 10MB of rubbish
RUN dd if=/dev/zero of=file_b bs=1024 count=10240 # 10MB of rubbish
RUN dd if=/dev/zero of=file_c bs=1024 count=10240 # 10MB of rubbish

After building and pushing the image, we can ask "ctr" to pull it, passing as a parameter the native snapshotter. Then we can see the size of the snapshots.

# snapshot with alpine layer
> ls -lh /tmp/root-dir-example-snapshot/snapshots/1 | head -n 1
total 68K

# snapshot with alpine layer + 10 MB
> ls -lh /tmp/root-dir-example-snapshot/snapshots/16 | head -n 1
total 11M

# snapshot with alpine layer + 10 MB + 10 MB
> ls -lh /tmp/root-dir-example-snapshot/snapshots/17 | head -n 1
total 21M

# snapshot with alpine layer + 10 MB + 10 MB + 10MB
> ls -lh /tmp/root-dir-example-snapshot/snapshots/18 | head -n 1
total 31M

And if we look at the folders, we will see exactly what we expect:

ls -la /tmp/root-dir-example-snapshot/snapshots/1
total 10316
drwxr-xr-x 19 root     root             4096 Feb 10 18:48 .
drwx------ 16 nicola domain^users     4096 Feb 10 18:49 ..
drwxr-xr-x  2 root     root             4096 Feb 10 18:47 bin
drwxr-xr-x  2 root     root             4096 Jan  9 13:46 dev
drwxr-xr-x 17 root     root             4096 Feb 10 18:27 etc
... omitted

ls -la /tmp/root-dir-example-snapshot/snapshots/16
total 10316
drwxr-xr-x 19 root     root             4096 Feb 10 18:49 .
drwx------ 16 nicola domain^users     4096 Feb 10 18:49 ..
drwxr-xr-x  2 root     root             4096 Feb 10 18:49 bin
drwxr-xr-x  2 root     root             4096 Jan  9 13:46 dev
drwxr-xr-x 17 root     root             4096 Feb 10 18:49 etc
-rw-r--r--  1 root     root         10485760 Feb 10 18:27 file_a
... omitted

ls -la /tmp/root-dir-example-snapshot/snapshots/17

total 20556
drwxr-xr-x 19 root     root             4096 Feb 10 18:49 .
drwx------ 16 nicola domain^users     4096 Feb 10 18:49 ..
drwxr-xr-x  2 root     root             4096 Feb 10 18:49 bin
drwxr-xr-x  2 root     root             4096 Jan  9 13:46 dev
drwxr-xr-x 17 root     root             4096 Feb 10 18:27 etc
-rw-r--r--  1 root     root         10485760 Feb 10 18:27 file_a
-rw-r--r--  1 root     root         10485760 Feb 10 18:27 file_b
... omitted

 ls -la /tmp/root-dir-example-snapshot/snapshots/18
total 30796
drwxr-xr-x 19 root     root             4096 Feb 10 18:52 .
drwx------ 17 nicola domain^users     4096 Feb 10 18:51 ..
drwxr-xr-x  2 root     root             4096 Feb 10 18:51 bin
drwxr-xr-x  2 root     root             4096 Jan  9 13:46 dev
drwxr-xr-x 17 root     root             4096 Feb 10 18:27 etc
-rw-r--r--  1 root     root         10485760 Feb 10 18:27 file_a
-rw-r--r--  1 root     root         10485760 Feb 10 18:27 file_b
-rw-r--r--  1 root     root         10485760 Feb 10 18:27 file_c
... omitted

Snapshot key and parent strings

Looking at the Snapshotter interface, we see that most methods receive a key and a parent string. From the snapshotter point of view, these strings are completely opaque and together make up a token to be exchanged for an object. Snapshotter uses the storage.CreateSnapshot(ctx, kind, key, parent, opts...) which returns a Snapshot record object. The method creates one if it does not exist, otherwise it returns the existing one. One of the fields of the Snapshot object is ID, which identifies the snapshot.

Snapshot ParentIDs

The Snapshot object returned by the storage contains an array called ParentIDs. The first element in the array is always the parent ID of the current snapshot (being Prepared, being Committed, etc.)
In the previous example, the Snapshot with ID 3 will have the following ParentIDs list, with 2 being its parent.

ParentIDs
0 = {string} "2"
1 = {string} "1"

Mounts

Earlier I mentioned the snapshotter does not just return a dir. The Snapshotter.Prepare API returns an array of mounts.

Looking at some of the existing snapshotter (like native and overlay), I would assume that most snapshotter only ever need a single mount. The Prepare API interface return type is an array, probably to make the API more generic for snapshotters that might need multiple mounts.

Something obvious, but still worth mentioning. The Type in the Mount returned by the Snapshotter needs to be a filesystem type supported by the underlying kernel. For example, assume someone wants to write a Snapshotter to expose the container filesystem as a RaiserFS. The snapshotter would need to return "raiserfs" as the mount type. The client will call the Unix mount and that will succeed only if raiserfs is enabled in the kernel.

containerd smart client client.Pull

This section answers the question - what happens when calling the smart client "client.Pull"?

image, err := client.Pull(ctx, "docker.io/library/redis:alpine", containerd.WithPullUnpack)
if err != nil {
    return err
}

Following the chain of calls from client.Pull, we see: unpacker.Unpack -> diffRemote -> over grpc -> diffService -> Applier -> which on linux eventually calls the apply function in the apply_linux file. That calls the Apply function in the archive/tar.go file.

Note that containerd only performs the mount for the first layer (see code). That's because the overlay snapshotter returns a bind mount for the first layer so the apply function ends up in this branch. All the mounts returned by the snapshotter.Prepare after that are of type overlay and are not mounted by containerd.

Also note that tar.go does not just untar the content of the layer, but also converts the OCI whiteouts and opaque whiteouts to Overlay whiteouts. This is a topic for a different article, but in short: suppose the file called my-file was removed in layer N. This is stored in the layer as .wh.my-file. During untar for that layer, when the function encounters the OCI whiteout .wh.my-file, it is going to store it in the snapshot folder as an overlay whiteout (a character file with device 0 0 with the same name as the original file, eg. my-file)

Simplifying the chain of calls and leaving out details, we can summarize the process of pulling and creating snapshots with the following pseudocode:

for each layer in the image manifest
  m, err = snapshotter.Prepare
  if prepare returns ErrAlreadyExist continue 
  // this check allows remote snapshots.
  // In reality, the client will also call 
  // snapshotter.Stat before skipping the creation of the snapshot.
  // Will be discussed in the section about remote snapshotter

  download layer
  load layer in the content store
  if m is overlay
    untar + convert from OCI spec to overlay in the overlay upperdir
  else if bind mount
    create temp mount 
    untar + convert from OCI spec to overlay in the temp mount
  snapshotter.Commit
end

Note 1: ctr pull uses the client differently, which results in a slightly different behavior. In particular, all the layers are first downloaded and loaded in the containerd content store, before asking the snapshotter to create new snapshots.

Note 2: In the example, we use the containerd.WithPullUnpack so that we not only fetch and download the content into containerd's content store but also unpack it into a snapshotter for use as a root filesystem.

If you do not pass that parameter, the snapshotter is not called.

ctr run

This section answers the question, what happens when running ctr run from a snapshotter point of view?
During container start, the client is going to call the snapshotter again to create an Active Snapshot that serves as a read-write layer for the container. The active Snapshot is going to be mounted before the container starts and unmounted when the container exit. If we start a long-running container in one terminal, we can hop into another terminal and see the mounts. For example, for an overlay snapshotter we will see:

 mount | grep overlay
overlay on /run/containerd/io.containerd.runtime.v2.task/default/example/rootfs type overlay (rw,relatime,lowerdir=/tmp/root-dir-example-snapshot/snapshots/22/fs:/tmp/root-dir-example-snapshot/snapshots/21/fs:/tmp/root-dir-example-snapshot/snapshots/20/fs:/tmp/root-dir-example-snapshot/snapshots/19/fs,upperdir=/tmp/root-dir-example-snapshot/snapshots/24/fs,workdir=/tmp/root-dir-example-snapshot/snapshots/24/work,xino=off)

for a native snapshotter we will see:

mount
....
/dev/mapper/vg0--aed748-root on /run/containerd/io.containerd.runtime.v2.task/default/example/rootfs type ext4 (rw,relatime,errors=remount-ro)

If you add something to the mount folder, the container is also gonna see it.

At this point, we are in a much better position to understand the documentation of the Snapshotter interface.
Besides the Prepare, Commit, Stat and Mount that we have already mentioned above, the interface also contains Remove which is called when deleting a snapshot (eg ctr snapshot rm <key>) and during container clean up (to remove the active snapshot).

What's a remote snapshotter?

containerd docs description of remote snapshotters is really clear:

Containerd allows snapshotters to reuse snapshots existing somewhere managed by them.

Remote Snapshotter is a snapshotter that leverages this functionality and reuses snapshots that are stored in a remotely shared place. These remotely shared snapshots are called remote snapshots. Remote snapshotter allows containerd to prepare these remote snapshots without pulling layers from registries, which hopefully shortens the time to take for image pull.

image, err := client.Pull(ctx, ref,
    containerd.WithPullUnpack,
    containerd.WithPullSnapshotter("my-remote-snapshotter"),
)

client.Pull with the option "WithPullUnpack" informs the client that it needs to call Snapshotter.Prepare before downloading the layers. If the call returns ErrAlreadyExist, the client calls the Snapshotter.Stat to confirm. If the call returns no error, it skips downloading the layer and skips loading it into the content store.
If the Snapshotter.Prepare instead returns nil, the client is going to download the layer and load it to the content store before calling the DiffApplierService.

This behavior allows us to set up snapshots out of band, that is outside the "normal" workflow. One such example is the soci snapshotter which allows image lazy loading. The snapshotter ships with a "rpull" command which performs this out of band prep. During rpull, the command calls the soci snapshotter which creates fuse mounts for each layer that has an index (remote snapshot). For layers that do not have an index, it downloads them as usual (local snapshot).
The local snapshot is created with overlay mount. Anyway, this is just a detail, the important bit is that folders created for a local snapshot only contain that layer, which is exactly what overlay does.
For example, after rpull we'll see something like the following:

> mount
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/1/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)

soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/4/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)

soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/6/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)

... omitted

During ctr run, the client asks the snapshotter (Prepare) for one more Active snapshot (which is going to be used as a read-write layer for the container). The snapshotter returns an overlay mount that overlays the fuse mounts and the local snapshots together.

> mount
soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/1/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)

soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/4/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)

soci on /var/lib/soci-snapshotter-grpc/snapshotter/snapshots/6/fs type fuse.rawBridge (rw,nodev,relatime,user_id=0,group_id=0,allow_other)

overlay on /run/containerd/io.containerd.runtime.v2.task/default/sociExample/rootfs type overlay (rw,relatime,lowerdir=/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/10/fs:/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/9/fs:/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/8/fs:/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/7/fs:/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/6/fs:/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/5/fs:/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/4/fs:/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/3/fs:/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/2/fs:/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/1/fs,upperdir=/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/11/fs,workdir=/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/11/work,xino=off)

Conclusions

A special thanks to Majd for reviewing and helping me come up with the right questions.
Kudos to Jin, who works on the soci-snapshotter and has been so kind to clarify some of the doubts I had.
Thanks to qiutongs for the comment, which accurately highlighted that containerd performs only one mount operation when performing a pull operation. I have updated the article to reflect that.

If you liked the article you might want to also check out How are docker images built?

For same history on containerd snapshotters, see the moby project blog

Check out Twitter to get new posts in your feed.
Credit for the cover image to Joshua Woroniecki via Unsplash

Your laptop is a distributed system

Nicola Apicella — Tue, 22 Nov 2022 15:59:12 +0000


Photo by Michael Dziedzic on Unsplash

When we think about distributed systems, we picture multiple machines communicating to each other to fulfill a request, perhaps a cluster of nodes storing data in a distributed database, or more abstract concepts like consistency in the CAP theorem[1].

But have you have ever seen a distributed system with your eyes? I do not mean an architecture diagram, I mean REALLY seeing it.

I bet you do and it is likely the thing you are using to read this post, your laptop. If the last sentence surprised you, keep reading and we shall see why.

Consistency

As you already know, memory is the device used to store data used by a computer. Memory varies for its speed, cost, and volatility. At the one end of the spectrum, we have fast but expensive memory like SRAM. At the other end, we have hard drives, slow (when compared to the CPU speed), cheap, and non-volatile as they do not require power to keep data stored. In the middle, we have the DRAM technology, popular due to its cost-effectiveness.

When talking about speed, it’s good to have an idea of the numbers involved to access data stored in memory:


Latency numbers — credit to hellerbarde via Github

Accessing data in an SRAM takes 0.5 nanosecond
Accessing data from the DRAM takes 1000 nanoseconds. Roughly a thousand times slower than access in SRAM
Accessing a date from a hard drive takes 2 million nanoseconds, that is 2 milliseconds. Roughly a million time slower than accessing an SRAM

Let’s assume you have a 1 GHz CPU, granted some simplifications [2] This means your CPU can run one instruction per nanosecond. In a naïve implementation, if we were to access a DRAM, the CPU would need to wait for about 1000 cycles doing basically nothing instead of running 1000 instructions. Lots of wasted compute capacity. If instead, we were to use an SRAM, the memory would be able to keep up with the CPU, thus not wasting precious cycles.

Ideally, we would love to equip our computers with the biggest amount of high-speed memory, but unfortunately, high-speed memory is expensive which brings us to a fundamental theorem of engineering: there is no such thing as free lunch. Engineering is a matter of finding the best trade-off between costs and performance.

Luckily two properties help computer designers with this trade-off: spatial and temporal locality. Spatial locality property says that instructions that are stored nearby to the recently executed instruction have high chances of execution. Temporal locality instead expresses the tendency of programs to use the same piece of data multiple times during the course of their execution. Those emergent properties do not surprise us since the CPU runs instructions one after the other and programs tend to access data multiple times during execution. These somewhat intuitive properties suggest organizing our memory hierarchically, with faster memory close to the CPU (SRAM commonly used for caches and CPU registers) and slower memory in the upper layer of the hierarchy (DRAM also referred to as main memory).

During a read operation, the CPU tries to access the data from the cache and use it if it’s there. In case it is not, the data needs to be copied from the main memory to the cache before the CPU can use it. The CPU does not copy just the data it needs, but a whole block of contiguous addresses (spatial locality).

Because the cache must be smaller than the main memory, at some point it will become full and some of the entries in the cache must be evicted. Which ones? The policy most commonly used is to remove the least recently used item (temporal locality), usually with some approximation depending again on a cost-benefit trade-off.

To gain better performance, system designers build systems with multiple CPUs interconnected on the same physical bus. Each CPU access the same memory which means that memory becomes a shared resource.

To recap, we have multiple CPUs interconnected on a bus, each CPU has its own cache and they all access the same shared memory:


Cache organization — credit to Kunal Buch, CC BY-SA 4.0, via Wikimedia Commons

Each CPU populates the cache based on the data it needs to access, the so-called local copy of the data. The same data could be used by different CPUs which means we could end up with multiple copies of the same data in different caches. But what happens when one of the CPU updates its local copy of the data? We have a typical distributed system problem: keeping multiple copies of the same data synchronized when one of the processors updates its local copy. This is what distributed systems folks call a consistency problem. For hardware engineers instead, this is the cache coherence problem. [3]

Conclusions

Our laptop is a highly coupled distributed system where each node is a CPU.

Albeit hardware engineers need to solve a class of problems frequently encountered in a multi-machine distributed system, a major distinction separates the way the problems are solved. In a computer, components are physically connected via each other on the bus, if not even collocated on the same chip. Multi CPUs systems do not need to deal with network partitions, message reordering, partial failure, and all the other properties of an unreliable network.

Turns out that the network is really what makes distributed systems hard [4].

Follow me on Twitter to get new posts in your feed.
Credit for the cover image to Michael Dziedzic via Unsplash

[1] Consistency - Availability - Partition tolerance tradeoffs. The extension to the CAP theorem is called PACELC
[2] The simplification does not account for pipelining, multiple cores, instructions that take multiple cycles to complete
[3] Out of order execution. The similarities between our laptop and distributed systems do not stop here. Modern CPUs adopts another trick to improve CPU performance: they are allowed to run instructions in an order different from the specified in the program. Of course, this means increasing the complexity of the system which now needs to guarantee consistency in the presence of instructions executed out of order
[4] Of course, this does not mean that CPU design is easy

The API Gateway Impossibility (the story of an API migration)

Nicola Apicella — Thu, 24 Mar 2022 16:11:13 +0000

At work we had to migrate an API Gateway from one AWS account to another. The migration had to be without downtime and be gradual (slowly shifting traffic from the old API to the new one). Since we do not control clients, we can't ask them to gradual shift calls to a new endpoint.
We thought it would easy - and we were quite wrong.

The idea was to point our existing domain name (app.domain.name.com in the following examples) to two different API Gateways, the old and new. And we were quite sure that could be achieved with some simple DNS changes. This is how we thought we would approach it:

Duplicate the API Gateway in the new account
Add a weight to the existing DNS A record (in Route53 hosted zone) which points to the old API Gateway
Insert a new A record with a weight in the Route53 hosted zone which resolves to the new API Gateway

In other words, we wanted to move from this:

Record Type	Weight	Source	Destination
A	-	app.domain.name.com	API-GW-domain-old

To this:

Record Type	Weight	Source	Destination
A	255	app.domain.name.com	API-GW-domain-old
A	255	app.domain.name.com	API-GW-domain-new

Adding the same weight (255 in the example), makes Route53 select with equal probability between the two records. We expected that to be how we could slowly shift traffic from the old api to the new. What follows is the result of trying to implement what described above, failed attempts and misunderstandings in the API Gateway and Route53 integration.

We first created the new API in a new AWS account, just to stumble upon a CloudFormation failure. The API Gateway custom domain resource failed because a custom domain with that domain already existed in another account. Turns out the domain assigned to the regional endpoint must be unique. Thus the CloudFormation failure above. The old API already had an API Gateway custom domain resource with that domain (app.domain.name.com).

Cool, so that does not work. We tried another approach. Create the new API Gateway and assign it a different custom domain, one that would differ only for the left most subdomain. Namely:

Record Type	Weight	Source	Destination
A	255	app.domain.name.com	api-old.domain.name.com
A	255	app.domain.name.com	api-new.domain.name.com
A	-	api-old.domain.name.com	API-GW-domain-old
A	-	api-new.domain.name.com	API-GW-domain-new

Having the api domains differ only in the subdomain makes it simple to create a certificate that covers them both (*.domain.name.com).

So, we updated the certificate, we deployed the new API in the new account (this time that worked), created the DNS records as described in the table above. It was time to verify it was working. It's here that we realized that accessing the old api via the "api-old.domain.name.com" worked, accessing the new api via the "api-new.domain.name.com" worked...but accessing the app via "app.domain.name.com" did not. What we were expecting was a round robin between the old and the new domain, what we got instead was the always eloquent API GW error message: {"message":"Forbidden"}.
Why though? It seems API Gateway service checks the Host header in the request and expects that to be one of the API Gateway custom domain. In other words, the following does not work:

> curl https://app.domain.name.com/demo
{"message":"Forbidden"}

But this does:

> curl -H "Host: api-old.domain.name.com" https://app.domain.name.com/demo
{"message":"Hello, World!"}

Finally we thought about using a wildcard custom domain. That also does not work because as the docs say:

You can't create a wildcard custom domain name if a different AWS account has created a custom domain name that conflicts with the wildcard custom domain name.

The docs also say that it is possible to request an exception for that.

The lesson learned is that pointing a domain name to two API Gateways is impossible. So how do you migrate it? Some options are:

have the new API issue a 301 redirect to the old one
old API proxies to the new one
deploy a proxy which shift traffic between the old API to the new
request exception for wildcard domain to be used in two different AWS accounts

How to monitor your system dependencies

Nicola Apicella — Sun, 03 Jan 2021 18:03:46 +0000

In the past few years, the DevOps reports have shown how high-performance teams use monitoring to quickly detect errors, reduce downtime and costs.
Monitoring a complex application is a significant engineering endeavor in and of itself, but thanks to effort of cloud providers, developers can instrument their system with few clicks or configurations.

When it comes to your system health, monitoring your dependencies is just as important as monitoring your system. If your system is experiencing issues, your system dashboards and alarms should highlight if one of your dependency is having a bad time.
In the rest of the article I show a simple pattern to monitor your dependencies. The examples use CloudWatch Embedded Metric Format and golang but the underlying idea can be applied to other cloud platforms or languages.

What’s EMF?

CloudWatch Embedded Metric Format allows to automatically generate metrics from logs. As a developer you need to log the metrics as JSON to stdout and Cloud Watch takes care of publishing the metric on your behalf.
You can use one of the AWS client libraries for NodeJS, Python or Java to build and log an embedded metric structure. Sadly, at the moment AWS has not published a client library for Golang which means you need to create the JSON object yourself following the schema described in the specification. It’s relatively straightforward and not surprisingly a few community maintained projects do exactly that, for example aws-embedded-metrics-golang.

I find EMF very convenient but it’s usage is not strictly necessary to monitor your dependencies. You can publish the metric yourself if that’s what you want to do.

Instrumenting the client

The main idea is to instrument the client you use to call your service. Regardless of the language or the library you use, it’s almost certain that the client you use to call your dependencies allows to plug in custom logic. You can leverage it to log the EMF metric.
For example, assuming you use a golang http client, you can implement the RoundTripper interface:

The code implements the RoundTripper interface with the logic to log details about the http calls. In the example, we log the Duration and the result of the call (client error or server error ). Once I deployed the code (in my case to Lambda) and triggered it a few times, we can see the metrics published to CloudWatch:

Instrumenting a different client is very similar. For example, here we instrument the golang AWS SDK to log calls made to other AWS services:

Conclusions

With AWS CloudWatch you can easily set up monitoring for your AWS resources and applications. EMF is a very convenient way to plug custom metrics in your system. As I mentioned, the pattern is quite generic and you can easily port it in other languages or even other monitoring systems.

If you develop on AWS take also a look at AWS X-Ray. With AWS X-Ray you can visualize and analyze the health, performance, and availability of your applications in one location, and see an end-to-end view of requests as they travel through your application.

Follow me on Twitter to get new posts in your feed.
Credit for the cover image to Luke Chesser on Unsplash.

This article has 86 reactions and 11 comments

Nicola Apicella — Wed, 13 May 2020 11:57:35 +0000

Tom Scott made a video recently where the title contains the number of views that the video has. The title is kept up to date with the number of views by a program which calls the youtube API. It's a great video if you haven't seen it yet.

Anyway, I really liked that idea and wanted to try it with this dev.to article. The article is different from Tom's video in two ways: the title of the article contains the number of reactions and comments instead of the views. The second thing is the topic. Tom talks about the invention of the APIs. What I want to talk about is how to use APIs to call the dev.to service.

The title of the article won't exactly match the number of comments and reactions because the program that updates the title only runs every minute. Also, I suspect the dev.to api is eventual consistent, which means the number of reactions and comments will be updated at some point, but it does not happen in real time. If, when you are reading the article, the title matches or is close to the number of reactions or comments, it means that the code is still working; but it could stop working any time. The dev.to api I am using is in beta and could change or being dismissed in the future.

In the remainder of the article I am going to talk about the steps I took to use the dev.to api.

OpenAPI specification

The first step to build a program which interacts with a service is to discover its api.

OpenAPI specification defines a standard which allows both developers and machines to discover and understand the capabilities of a service. It serves as documentation and as a contract.

An API defined via an OpenApi document is machine friendly, which means we can build programs that interpret that document and do something useful with it. For example, the oapi-codegen is able to read the document and generate a golang http client to interact with the api. It can also generate a stub service which can be used both as starter or as a mock for your tests.

The other benefit of OpenAPI is that is also human friendly. It can be used as documentation, hosted on a server and browsed via a browser. This is what dev.to uses and anybody can read and download its OpenApi spec here.

AuthN and AuthZ

The second step to build the program is getting valid credentials.

Services need a way to know that you are who you say you are (authentication). Once they have established that, they might also need to verify that you can carry on the action you want to perform (authorization).
In the case of dev.to, the api key is used for authN and authZ. An api key is a pseudo random service generated string which is associated with your account. To obtain one for dev.to:

visit https://dev.to/settings/account
in the "DEV API Keys" section create a new key by adding a description and clicking on "Generate API Key"

Generate the client

Once we have the OpenAPI document and the api key, we are ready to make good use of them.

I have used the oapi-codegen tool to generate a golang client to interact with the api. The process was quick. I downloaded the OpenAPI document for dev.to and run the oapi-codegen, and that was it, no errors or warnings. Sometimes you need to tweak a bit the document or the code generator to make it work, but not in this case. The result was a golang struct with the following methods:

Use the client

After generating the client, I just had to write some code which:

retrieves the number of reactions and comments for the article
updates the title if the number of reactions or comments changed

The second step require a little bit more work than the first one.
Since, I use front matter when writing on dev.to, updating the title means updating the title field in markdown:

---
title:This article has x reactions and y comments
published: true
description: ...
tags: ...
---

The gist of the program is here:

func run(ctx context.Context, env *programEnv, client *ClientWithResponses) {
    r, e := getArticleActivity(ctx, client, env.ArticleID)
    if e != nil {
        log.Fatal(e)
    }
    log.Debugf("Article reactions: %[1]d - Article comments: %[2]d",
        r.ReactionsCount,
        r.CommentsCount)

    newTitle := generateNewTitle(r.ReactionsCount, r.CommentsCount)
    fmEditor, e := newFrontMatterEditor(r.BodyMarkdown)
    if e != nil {
        log.Fatal(e)
    }
    if fmEditor.title == newTitle {
        log.Debug("Article title is already up to date with the number of reactions and comments")
        return
    }

    log.Debug("Updating the title of the article")
    fmEditor.updateTitle(newTitle)
    e = saveArticle(ctx, client, env.ArticleID, fmEditor.markdown)
    if e != nil {
        log.Fatal(e)
    }
    log.Info("Article with the new title has been saved")
}

You can find the rest of the source code is on github.

Profit

Lastly, I create a cron job which runs the program every minute:

SHELL=/bin/bash
* * * * * /home/napicella/wrapper.sh

where wrapper.sh is a bash script which calls the program:

#!/bin/bash

source .env
./update-article >> /home/napicella/ua.log 2>&1

Dev.to documentation does not mention any service quota for the api. I did not get throttled so far, so I guess it's a reasonable rate for the service.

Conclusion

I want to conclude by thanking Tom Scott for the idea. If you haven't seen it, this is the link to the video.

Follow me on Twitter to get new posts in your feed.
Credit for the cover image to GraphicMama-team

Test Driven Development by Example

Nicola Apicella — Sun, 10 May 2020 18:54:05 +0000

Among the few positive aspects of the lock-down, having more time to read is definitely one of them. Two weeks ago I started reading again the Test Driven Development (TDD) bible written by Kent Beck, who is considered by most the father of TDD. Regardless of what your thoughts are about TDD, the book is a gold mine on testing. I highly recommend it.

In the same spirit of the book, this article is going to be a practical walk through on how to develop code driven completely by tests; an example from start to end on how to apply TDD. I am going to start with a brief recap on TDD, then I'll walk you through an example in which we are going to code a throttle the TDD way. Last, I'll share some resources that can be used to practice TDD.

The target audience for this post is people who are considering using TDD in the development process. If you have already delved into TDD or you are already using it, this post probably won't add any new information to your knowledge. However it might still be useful as a reference you can share with others curious about the topic.

Preface

TDD is one of the software engineering practice which has stood the test of time. At the beginning of 2000s Kent Beck came out with the book "Test Driven Development: By Example". The book is twenty years old, though TDD as a concept it's probably older than that. It was Kent Beck himself to say that he did not "invent" TDD, but rather "rediscover" it from old articles and papers. The humble programmer, Dijkstra (1972) and the Report of The Nato Software Engineering Conference (1968) both described the process of testing the specification before writing the code. While, Kent Beck might not have been the one who invented, he definitely was the one who made it popular.

Is a 20+ engineering practice still relevant today?

Everything we do is built upon layer of abstractions and decisions made decades ago. People who made those decisions lived in a different context, had different constraints and problems to solve. What they did, is what we do today: they came up with the best solution they could think of at the time.
Their decisions live with us. But most often, their reasons do not.
Technology changed, the problems we need need to solve changed, the world has changed.

As software engineer one of the most valuable skills I picked up is to question everything, understand why things are the way they are. Searching for the context in which these decisions were made is the key to understand if the same decisions are applicable in the current world.

So, is TDD still relevant today? I think it is, because:

we still need to write unit tests to prove that our code respect the specification
we still want to reduce the number of bugs which make it all the way to production
we still want to iterate fast and integrate changes often
we still want to build highly cohesive and loosely coupled components

I believe the premises of TDD are still valid in the context we live in.

TDD is controversial

Not everybody thinks TDD is useful. I couldn't agree more - not everybody must use it. During the years, a few research studies were made to determine the effectiveness of TDD in the software development process, but they were largely inconclusive. I think that's because quantitative measurements about source code quality and speed of iterations are too noisy and dependent on social factors - all things which are difficult to take into account in a research study.

I want to conclude this quite lengthy preface by saying that I am not religious about TDD - and I hope neither will you. It's like any other tool we have in our toolbox - it allows to see the problem from a different point of view.

TDD

TDD is a predictable way to develop code which relies on the following three steps:

RED - Write a unit test, run it an watch it fail. The unit test should be short and focus on a single behavior of the system under test. By writing the failing test you ensure that your test is calling the correct code and that the code is not working by accident. It's a meaningful failure, and you expect it to fail
GREEN - Write the minimum amount of code needed to make the test pass
REFACTOR - Eliminate the duplication (both in the test and in the code, including duplication between test and code). More in general this is the step in which you would perform refactoring

There isn't much else you need to know to start using TDD. Using it effectively is just a matter of practicing it over and over again. Project after project you become better at it.

Why TDD?

you are always one test away from functional code
tests are more expressive; the result is usually tests that cover the behavior of the module instead of the underlying implementation
increased test coverage and reduced coupling between test and production code
it's very useful when you know what you have to build, but have no idea where to start; a situation quite common when you need to add or change a new feature in a piece of the code base you are not familiar with

Throttling example

In this section we will build a throttle. The end goal of throttling is to limit how many times a function can be called in a given interval of time. It's generally used to avoid overloading the receiver with too many calls (for example a remote server) or because a sample of the events is sufficient to carry on with the functionality.

To sum it up to throttle a function means to ensure that the function is called at most X times in a specified time period (for instance, at most three times every second). The throttle we are going to build, is a slightly simpler version which only allows at most one call in a specified period. This is the spec:

throttle returns a function which is called at most once in a specified time period. 
It takes as input the function to throttle and the period. 
If the period is less or equal than zero, then no throttle is applied.

Let's try to build it. Since we are using TDD, this means writing our test first.

First test

    describe("Given the throttle time is 0", () => {
        it("Runs the function when we call it", () => {
            let count = 0;
            const 
                fun = () => count++,
                funT = throttle(fun, 0);
            funT();
            expect(count).toBe(1);
        });
    });

In the test, we defined a simple function called fun which simply increments a variable called count every time we invoke the function. We call our throttle function giving it as parameter the function we just defined and a throttle period of zero. According to the specification, if the throttle period is zero, the function must be invoked when we call it. We called funT (as in fun Throttled) the result of applying throttle to fun.

Run the test and watch it fail. Now, we have to make it pass by writing the minimum amount of code necessary. So. let's create the throttle function:

function throttle(fun, throttleTime) {
    return () => {
        fun();
    }
};

module.exports = { throttle };

Run the test again, and it's green! To make the test green, we just had to create the throttle function and make it invoke fun. At this point there is nothing to refactor, so we'll move the next test.

Second test

According to the spec, if the throttle period is zero, the function must be invoked "every" time we call it because no throttle is applied. Let's test that:

    describe("Given the throttle time is 0", () => {
        it("Runs the function 'every' time we call it", () => {
            let count = 0;
            const 
                fun = () => count++,
                funT = throttle(fun, 0),
                calls = 10;
            for (let i = 0; i < calls; i++) {
                funT();
            }    
            expect(count).toBe(calls);
        });
    });

Instead of calling funT once like in the previous test, now we are calling it ten times and we expect the count variable to be ten at the end.

Run the tests and...it's green. We did not even have to add any code for it, good. Before we go ahead with the next test, we are going to refactor: the second test includes the first one so we can remove it, which leaves us with the following suite:

describe("throttle suite", () => {

    describe("Given the throttle period is 0", () => {
        it("Runs the function 'every' time we call it", () => {
            let count = 0;
            const 
                fun = () => count++,
                funT = throttle(fun, 0);
                calls = 10;
            for (let i = 0; i < calls; i++) {
                funT();
            }    
            expect(count).toBe(calls);
        });
    });
});

Third test

Let's add another test when the throttle period is negative:

    describe("Given the throttle period is negative", () => {
        it("Runs the function 'every' time we call it", () => {
            let count = 0;
            let count = 0, calls = 10;
            const
                fun = () => count++,
                funT = throttle(fun, -10);
            for (let i = 0; i < calls; i++) {
                funT();
            }    
            expect(count).toBe(calls);
        });
    });

Again, it passes and we did not have to add any code. We can refactor since the test for the negative period and the zero period are very similar:

describe("throttle suite", () => {

    const runFun = (throttlePeriod) => {
        it("Runs the function 'every' time we call it", () => {
            let count = 0, calls = 10;
            const 
                fun = () => count++,
                funT = throttle(fun, throttlePeriod);
            for (let i = 0; i < calls; i++) {
                funT();
            }    
            expect(count).toBe(calls);
        });
    };

    describe("Given the throttle period is 0", () => runFun(0));
    describe("Given the throttle period is negative", () => runFun(-10));
});

Fourth test

describe("Given the throttle period is positive", () => {
        describe("When the throttle period has not passed", () => {
            it("Then `fun` is not called", () => {
                let count = 0;
                const
                    fun = () => count++,
                    funT = throttle(fun, 1* time.Minute);

                funT();
                expect(count).toBe(1);
                funT();
                expect(count).toBe(1);
            });
        });
    });

Run the test and watch it fail:

Failures:
1) throttle suite 

   Given the throttle period is positive 
   When the throttle period has not passed 
   Then `fun` is not called
     Message:
       Expected 2 to be 1.

What's happening here? We expect the first call to funT to go through because the throttle does not apply to the first call. Thus in the first expectation we check if the variable count is equal to one. The second time we call funtT must be throttled because at least one minute needs to pass between the first and the second call; that's why we expect count still to be one in the second expectation. Except it isn't. The count variable is two because we haven't implement any throttling logic yet.

What's the smallest step to make the test pass? What I have come up with is:

check if it's the first time we are calling the function
differentiate between a positive throttle period and a less than zero period

function throttle(fun, throttleTime) {
    let firstInvocation = true;
    return () => {
        if (throttleTime <= 0) {
            fun();
            return;
        }
        if (firstInvocation) {
            firstInvocation = false;
            fun();
        }
    }
};

The introduction of firstInvocation and the if statement was enough to make the test pass.

Fifth test

Next one is interesting.

        describe("When the throttle period has passed", () => {
            it("Then `fun` is called", () => {
                let count = 0;
                const
                    fun = () => count++,
                    funT = throttle(fun, 1* time.Minute);

                funT();
                expect(count).toBe(1);
                // 1 minute later ...
                funT();
                expect(count).toBe(2);
            });
        });

In this test we want to verify that after one minute has passed, the function won't be throttled. But how do we model time? We need to have something that allows to keep track of time, like a timer or something similar. More importantly, we need to manipulate the state of the timer in the test. Let's assume we already have what we need and change the test accordingly:

        describe("When the throttle period has passed", () => {
            it("Then `fun` is called", () => {
                let count = 0, timer = new MockTimer();
                const
                    fun = () => count++,
                    funT = throttle(fun, 1 * time.Minute, timer);

                funT();
                expect(count).toBe(1);
                // fast forward 1 minute in the future
                timer.tick(1 * time.Minute); 
                funT();
                expect(count).toBe(2);
            });
        });

The diff between this version of the test and the previous is the introduction of the MockTimer. It's initialized with the rest of the variables at the beginning of the test. Right after the first expectation the timer tick method is called to move the timer one minute in the future. Since the throttle timeout is one minute, we expect the next call to funT() to go through.

Let's run the test. Not surprisingly, it fails because the MockTimer does not exist. We need to create it.

Before doing that, let's figure how we would use the timer in the throttle function. You can come up with different ways of using it. In my case I decided I needed to have a way to start the timer and check if it is expired or not. With that in mind let's change the throttle function to make use of a Timer that does not exist yet. Using a function before implementing it seems stupid, but in fact it's quite useful because you get to see the usability of the api before writing the code for it.

function throttle(fun, throttleTime, timer) {
    let firstInvocation = true;    
    return () => {
        if (throttleTime <= 0) {
            fun();
            return;
        }
        if (firstInvocation) {
            firstInvocation = false;
            fun();
            timer.start(throttleTime);
            return;
        }
        if (timer.isExpired()) {
            fun();
            timer.start(throttleTime);
        }
    }
};

Established the api, let's implement a mock timer for our test:

class MockTimer {
    constructor() {
        this.ticks = 0;
        this.timeout = 0;
    }

    tick(numberOfTicks) {
        this.ticks += numberOfTicks ? numberOfTicks : 1;
    }

    isExpired() {
        return this.ticks >= this.timeout;
    }

    start(timeout) {
        this.timeout = timeout;
    }
}

Run the test again, and boom, tests are green!

Let's change our test and make it richer:

describe("When the throttle period has passed", () => {
    it("Then `fun` is called", () => {
        let count = 0, timer = new MockTimer();
        const
            fun = () => count++,
            funT = throttle(fun, 1 * time.Minute, timer);

        funT();
        expect(count).toBe(1);

        timer.tick(1 * time.Minute);
        funT();
        expect(count).toBe(2);

        timer.tick(59 * time.Second);
        funT();
        expect(count).toBe(2);

        timer.tick(1* time.Second);
        funT();
        expect(count).toBe(3);

        for (let i = 0; i < 59; i++) {
            timer.tick(1 * time.Second);
            funT(); 
            expect(count).toBe(3);
        }

        timer.tick(1* time.Second);
        funT();
        expect(count).toBe(4);
    });
});

At this point, we just need to plug in an actual timer which we could build with a similar process, for example:

class Timer {
    constructor() {
        this.expired = true;
        this.running = false;
    }

    isExpired() {
        return this.expired; 
    }

    start(timeout) {
        if (this.running) {
            return new Error("timer is already running");
        }
        this.expired = false;
        this.running = true;
        setTimeout(() => {
            this.expired = true;
            this.running = false;
        }, timeout);
    }
}

Tidying up the API

There is one last thing. We can create a default timer instead of requiring the caller to pass it as a parameter:

function throttle(fun, throttleTime) {
    return throttleWithTimer(fun, throttleTime, new Timer());
}

function throttleWithTimer(fun, throttleTime, timer) {
// ... same as before

Finally we can use our throttle function:

throttle(onClickSendEmail, 1 * time.Second);

Practice TDD

If you like the idea of writing your test first, then give TDD a try. In this article I showed the throttle function, maybe you can try the debounce by yourself. When I was thinking about the article I almost settled on using Conway's Game of Life as example, but it did not take me long to realize that the article would be too long. If you are up for it, it's a fun exercise to build with TDD.

You could also try some of the programming Katas available online, like:

Conclusion

Whatever you pick to flex your TDD muscle, my suggestion is to give it sometime. At least for me, TDD did not click right away. The first times I tried it, I got stuck - I could not figure how to write the test before the code. But I kept practicing on my own and ultimately it became natural to think about the test before thinking about the code.

Follow me on Twitter to get new posts in your feed.
Credit for the cover image to GraphicMama-team

How are docker images built? A look into the Linux overlay file-systems and the OCI specification

Nicola Apicella — Tue, 21 Apr 2020 06:40:10 +0000

It's impossible to work with docker containers without docker images. In this post I want to talk about what makes docker images possible: the overlay filesystems.
I'll start with a brief description of overlay filesystems. Then we will see how it applies to docker images and how docker builds an image from a dockerfile. I'll conclude with layers cache and OCI format for container images.

As usual I'll try to make the blog post as practical as possible.

What's an overlay filesystems

Overlay filesystems (also called union filesystems) allow creating a union of two or more directories: a list of lower directories and an upper directory. The lower directories of the filesystem are read only, whereas the upper directory can be used for both reads and writes.

Let's see what that means in practice by mounting one.

Create an overlay fs

Let's create a few folders and combine them.
First I'll create a folder called "mount" which will contain the union of all the other folders. Then I'll create a bunch of folders called "layer-1", "layer-2", "layer-3", "layer-4". Finally a folder called "workdir" which is needed by the overlay filesystem, well, to work properly.

You can call any of the folders as you wish, but calling them "layer-1", "layer-2", etc. will make easier to understand the parallel with docker images as we shall see.

cd /tmp && mkdir overlay-example && cd overlay-example

[2020-04-19 16:02:35] [ubuntu] [/tmp/overlay-example]  
> mkdir mount layer-1 layer-2 layer-3 layer-4 workdir

[2020-04-19 16:02:38] [ubuntu] [/tmp/overlay-example]  
> ls
layer-1  layer-2  layer-3  layer-4 mount  workdir

Let's also create some files into layer-1, layer-2 and layer-3 folders.
We will leave the layer-4 (our upper folder) empty. Again, that's not necessary, it will just make easier our parallel with docker images.

[2020-04-19 16:02:40] [ubuntu] [/tmp/overlay-example]  
> echo "Layer-1 file" > ./layer-1/some-file-in-layer-1

[2020-04-19 16:03:36] [ubuntu] [/tmp/overlay-example]  
> echo "Layer-2 file" > ./layer-2/some-file-in-layer-2

[2020-04-19 16:03:53] [ubuntu] [/tmp/overlay-example]  
> echo "Layer-3 file" > ./layer-3/some-file-in-layer-3

Finally, let's mount the filesystem:

sudo mount -t overlay overlay-example \
-o lowerdir=/tmp/overlay-example/layer-1:/tmp/overlay-example/layer-2:/tmp/overlay-example/layer-3,upperdir=/tmp/overlay-example/layer-4,workdir=/tmp/overlay-example/workdir \
/tmp/overlay-example/mount

Now let's look inside the mount folder:

[2020-04-19 16:13:28] [ubuntu] [/tmp/overlay-example]  
> cd mount/

[2020-04-19 16:13:31] [ubuntu] [/tmp/overlay-example/mount]  
> ls -la
total 20
drwxr-xr-x 1 napicell domain^users 4096 Apr 19 16:07 .
drwxr-xr-x 8 napicell domain^users 4096 Apr 19 16:07 ..
-rw-r--r-- 1 napicell domain^users   13 Apr 19 16:03 some-file-in-layer-1
-rw-r--r-- 1 napicell domain^users   13 Apr 19 16:03 some-file-in-layer-2
-rw-r--r-- 1 napicell domain^users   13 Apr 19 16:03 some-file-in-layer-3

As expected the content of the folders layer-1, layer-2 and layer-3 have been mounted/combined in the mount folder.
Sure enough if we look at the content of the files, we'll find what we have written in the previous step.

[2020-04-19 16:13:33] [ubuntu] [/tmp/overlay-example/mount]  
> cat some-file-in-layer-3
Layer-3 file

Let' try to create a file in the mount folder:

[2020-04-19 16:23:31] [ubuntu] [/tmp/overlay-example/mount]  
 > echo "new content" > new-file

[2020-04-19 16:27:33] [ubuntu] [/tmp/overlay-example/mount]  
> ls
new-file  some-file-in-layer-1  some-file-in-layer-2  some-file-in-layer-3

Where should the new file be? In the upper layer, which in our case is the folder called "layer-4":

 [2020-04-19 16:23:49] [ubuntu] [/tmp/overlay-example]  
> tree
.
├── layer-1
│   └── some-file-in-layer-1
├── layer-2
│   └── some-file-in-layer-2
├── layer-3
│   └── some-file-in-layer-3
├── layer-4
│   └── new-file
├── mount
│   ├── new-file
│   ├── some-file-in-layer-1
│   ├── some-file-in-layer-2
│   └── some-file-in-layer-3
└── workdir
    └── work [error opening dir]

7 directories, 8 files

Let's try to delete a file:

[2020-04-19 16:27:33] [ubuntu] [/tmp/overlay-example/mount]  
> rm some-file-in-layer-2

[2020-04-19 16:28:58] [ubuntu] [/tmp/overlay-example/mount]  
> ls
new-file  some-file-in-layer-1  some-file-in-layer-3

What do you think happened to the original file in the "layer-2" folder?

 [2020-04-19 16:29:57] [ubuntu] [/tmp/overlay-example]  
> tree
.
├── layer-1
│   └── some-file-in-layer-1
├── layer-2
│   └── some-file-in-layer-2
├── layer-3
│   └── some-file-in-layer-3
├── layer-4
│   ├── new-file
│   └── some-file-in-layer-2
├── mount
│   ├── new-file
│   ├── some-file-in-layer-1
│   └── some-file-in-layer-3
└── workdir
    └── work [error opening dir]

7 directories, 8 files

A new file called "some-file-in-layer-2" was created in "the layer-4". The weird thing is that the file is a character file. These kinds of files are called "whiteout" files and are how the overlay filesystem represents a file being deleted:

 [2020-04-19 16:31:09] [ubuntu] [/tmp/overlay-example/layer-4]  
> ls -la
total 12
drwxr-xr-x 2 napicell domain^users 4096 Apr 19 16:28 .
drwxr-xr-x 8 napicell domain^users 4096 Apr 19 16:07 ..
-rw-r--r-- 1 napicell domain^users   12 Apr 19 16:23 new-file
c--------- 1 root     root         0, 0 Apr 19 16:28 some-file-in-layer-2

Now that we have finished with it, let's unmount the filesystem and remove the folder we created:

[2020-04-19 16:37:11] [ubuntu] [/tmp/overlay-example]  
> sudo umount /tmp/overlay-example/mount && rm -rf /tmp/overlay-example

Wrapping up the overlay filesystems

As we said at the beginning, the overlay filesystem allows to create a union of directories. In our case the union was created in the "mount" folder and it was the result of combining the "layer-{1, 2, 3, 4}" folders. Changes to files, deletion or creation will be stored in the upper dir, which in our case is "layer-4". This is why this layer is also called "diff" layer.
Files from upper layer shadow the ones in lower layers, i.e. if you have a file with the same name and relative path in layer-1 and layer-2, the layer-2 file is going to end up in the "mount" folder.

In the next section we will see how this is used with docker images.

What's a docker Image?

A docker image is essentially a tar file with a root file system and some metadata. You might have heard of the expression image layer and that every line in a docker file creates a new layer. For example in the following snippet we will end up with an image with three layers.

FROM scratch
ADD my-files /doc
ADD hello /
CMD ["/hello"]

So what happens when you type "docker run". A lot of things really, but for the purpose of this article we are only interested in the bits concerning the image.
At high level, docker downloads the tarballs for the image, it unpacks each layer into a separate directory and then tells the overlay filesystem to combine them all together together with an empty upper directory that the container will write its changes to it.
When you change, create or delete files in the container, the changes are going to be stored in this empty directory. When the container exits, docker cleans up the folder - that is why the changes you make in the container do not persist.

Layers cache

This way to use the overlay filesystem allows hosts to cache docker images effectively. For example, if you define two images, they can both use the same layers. No need to download multiple times or to have many copies on the disk!

OCI-format container images

Running a container at high level can be seen as a two steps process: building the image and running a container from the image. The popularity of docker has convinced people to standardize both steps - allowing the two pieces to evolve separately. The Open Container Initiative (OCI) is the governance which has been working with the industry to these standards.

The OCI currently contains two specifications: the Runtime Specification (runtime-spec) and the Image Specification (image-spec). The Runtime Specification outlines how to run a “filesystem bundle” that is unpacked on disk. At a high-level an OCI implementation would download an OCI Image then unpack that image into an OCI Runtime filesystem bundle. At this point the OCI Runtime Bundle would be run by an OCI Runtime.

The standardization allows other people to develop custom container builders and runtimes. For example, jessfraz/img, buildah and Skopeo are all tools that allow you to build container images without using docker. Similarly, many tools to run containers (so called container runtimes) have emerged, for example runc (used by docker) and rkt.

Other overlay filesystems

Overlay is not the only union file system that docker can use. Any file system that allows union like features and diff layer could potentially be used. For example docker can use overlay as we have seen, but also aufs, btrfs, zfs and devicemapper.

What happens when you build an image?

Let's assume we have the following dockerfile we want to use to build an image from:

FROM ubuntu
RUN apt-get update
...

At high level, this is how docker builds an image out of it:

Docker downloads the tarball for the image specified in the "FROM" and unpacks it. This is the first layer of the image.
Mounts a union file system, with the lower dir being the one just downloaded. The upper dir is an empty folder
Starts bash in a chroot and runs the command specified in RUN: chroot . /bin/bash -c "apt get update"
When the command is over, it zips the upper layer. This is the new layer of the image we are building
If the dockerfile contains other commands, repeat the process from the second step using as lower dir all the layers we have got so far. Otherwise exit.

Of course this is a simplified workflow which does not take into account different type of commands like "ENV", "ENTRYPOINT", etc. Those things are stored in the metadatafile which is going to be bundled together with the layers.

Conclusion

The idea of zipping a whole root file system in a tar and keeping a tar for each diff-layer turned out to be very powerful. It did not just enabled docker, but turns out to be a concept that can be used in other context as well. I guess, we will see more tools taking advantage of that in the future.

Follow me on Twitter to get new posts in your feed.
Credit for the cover image to unsplash-logofrank mckenna.

AWS Networking cheat-sheet - EIP, ENI, VPC, etc

Nicola Apicella — Fri, 17 Apr 2020 12:52:39 +0000

AWS network infrastructure can become complex.
There is also some overlaps between different technologies which makes it harder to understand what does what.

Some of the questions I hear the most are about the intersection between AWS VPC, Elastic Network interfaces, Elastic IP, and the closely related Security Group and ACL. This is my attempt to clarify in practical terms the difference between these technologies.

If you are eager to look at the cheat-sheet, scroll down to the bottom.

AWS VPC

History - VPC was an afterthought

In the early days of AWS, when EC2 lunched all the instances from all the customers could communicate with each other. If Alice launched an instance in her account and Bob launched another instance is his account, the two instances were able to communicate with each other out of the box.
To prevent that, Alice (or Bob) were required to set a security group, allowing only traffic from selected ips or from instances launched with the same security group.

The problem with security group was that was very easy to mess it up, thus allowing other AWS accounts to breach the instances. This mode of operations is called EC2-classic to distinguish it from the new way in which EC2 instances are provisioned.

In this new mode, EC2 instances are always launched in a VPC (called default VPC) which has a default security group. Instances in different VPCs do not communicate with each other, unless they are peered.
Of course you can also create your own VPC and launch instances in there.

Today EC2-classic only exist for legacy reasons which tie into the challenge of taking back a service once that has been launched.

What's a VPC?

Having covered the history, what's a VPC? It's an isolated portion of the AWS cloud. Instances launched in a VPC cannot be accessed from other VPCs in the same account or other AWS accounts.
It's main property is the CIDR block, which is the range of IPv4 addresses assigned to instances launched in the VPC.
Leaving aside Security group and ACL (which are discussed later), all the instances created in the same VPC can communicate with each other, like in a private LAN.

After creating a VPC, you can add additional subnets by dividing the range of IP addresses assigned to the VPC.

Security group

What's a Security Group?

A security group acts as a virtual firewall that controls the traffic from/to a network interface.
When you launch an instance, you get a default network interface (eth0).
An EC2 instance can have multiple network interfaces and each network interface can have a different security group.
Later I'll discuss how to attach additional network interfaces to an instance.

Confusing terminology

Security Groups are attached to a network interface, not an instance. When you start an instance, it receives a default network interface (eth0). The security group will be attached to that default network interface.
Don't get fooled, every time you specify a security group for an AWS service, behind there is a network interface.
Let's take for example AWS Elastic File System. When you create a File System, you are going to specify security groups. Does that mean that the security group control the access to the File System? Well, in some sense yes. But technically the security group are going to be attached to a Elastic Network Interface (ENI) that EFS creates on your behalf.

I am going to discuss about ENIs later, the takeaway here is: Security groups are associated to Network Interfaces (Elastic or not) and they act as a virtual firewall that controls the traffic from/to a network interfaces.

Even more confusing

Subnets in the VPC can have a default security group. Does this mean that security groups can be attached to network interfaces as well as subnets? Mmm, not really. By defining a default security group for a subnet, you have informed EC2 that the next time you launch an instance in that subnet, it needs to attach that security group to the network interface of the instance. That is, unless you override the security group in the launch instance api call.

Subnets also feature a property called: "auto-assign public IPv4 address" which means that new network interfaces in that subnet automatically receives also a public IPv4 or IPv6 address (in addition to the private IP address). As for security groups, this setting defines the default value that EC2 is going to use when launching an instance in that subnet. For example, let's assume we start an instance with the following command:



aws ec2 run-instances \
  --subnet-id subnet-10000000 \
  --image-id resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2

The instance launched will have an auto assigned ip address if subnet-10000000 has the "auto-assign public IPv4 address" setting set to true. But, like I mentioned this is just a default and we can override this behavior by using the --associate-public-ip-address or the --no-associate-public-ip-address flags. For example if we run the following command:



aws ec2 run-instances \
  --subnet-id subnet-10000000 \
  --image-id resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 \
  --associate-public-ip-address

the instance will get a public IP regardless of whether we have set the "auto-assign public IPv4 address" on the subnet subnet-10000000.

Subnets Isolation

After creating your VPC, you divide it into subnets. In an AWS VPC, subnets are not isolation boundaries. Rather, they are containers for routing policies. Isolation between subnets is achieved by attaching a Security Group to the EC2 instances.

For example let's assume we have two subnets in the VPC: SubnetA is 10.0.0.0/25 and SubnetB is 10.0.0.128/25. To isolate them, create a security group for SGSubnetA which allows all traffic for the CIDR 10.0.0.0/25 and another SGSubnetB that allows traffic for the CIDR 10.0.0.128/25.
Attach the SGSubnetA to the instances launched in SubnetA and SGSubnetB for the one in SubnetB.
Result: Only the instances in the subnet can talk to each other, but instances from SubnetA cannot communicate with instances from SubnetB and vice versa.

VPC ACL

A network access control list (ACL) is a network firewall. With security groups you can control what goes in and out your instances, and with VPC ACL you can control what goes in and out of your VPC.

EIP

An Elastic IP address (EIP) is a static IPv4 address provided by AWS.

You associate an EIP to a network interface. The documentation is a bit confusing on this point because it says you can attach an EIP to a running instance or to ENI. What it does not say is that by attaching the EIP to the instance, you are actually attaching it to the default network interface of the instance (eth0).

You can attach additional network interfaces to an instance, by attaching to it additional ENIs.

Why do you even need an EIP? Are they different from public IPs?

Public IP addresses are dynamic, their value is lost once the EC2 instance using it is stopped or terminated.

Elastic IPs are static, they survive instance stop/start cycles.

ENI

An Elastic Network interface (ENI) is the network stack of the instance.
Each ENI lives within a particular subnet of the VPC and has the following attributes (not exhaustive list):

Private IP Address
Public IP Address
Elastic IP Address
Subnet
MAC address
Security Group(s)

A very important consequence of this new model is that the idea of launching an EC2 instance in a particular VPC subnet is effectively obsolete. A single EC2 instance can be attached to two ENIs, each one on a distinct subnet. The ENI (not the instance) is now associated with a subnet.

This enable two powerful use cases:
Assuming we have created two EC2 instances (Instance-A and Instance-B), an ENI and a public EIP. The EIP is attached to the ENI and the ENI is attached to instance-A. Both EC2 instances run a Web Server.

Let's visualize that:

In the image, the Client sends an http request to the EIP. EC2-A will respond with an html page. Now if we move the ENI from EC2-A to EC2-B (detach from EC2-A and attach to EC2-B), the next request coming from the client will be server by EC2-B.
This is a fundamental building block which allows to build Primary-Failover strategy.
Note: I do not want to muddy the water, so I won’t go into other Primary-Failover solutions. Just for you to know, ENI switching is just one of the possible solution to build Primary-Failover mechanic in a system - Load Balancers or consensus groups are other options.

You can also use it to build dual-homed environment for your web, application, and database servers. The instance’s first ENI would be attached to a public subnet, routing 0.0.0.0/0 (all traffic) to the VPC’s Internet Gateway. The instance’s second ENI would be attached to a private subnet, with 0.0.0.0 routed to the VPN Gateway connected to your corporate network. You would use the private network for SSH access, management, logging, and so forth. You can apply different security groups to each ENI so that traffic port 80 is allowed through the first ENI, and traffic from the private subnet on port 22 is allowed through the second ENI.

Note: When creating a subnet, you can specify some default options that are going to be applied to all the ENIs that are associated to that subnet. Most notably, you can choose to give a public ip to all the ENIs associated to that subnet.

Internet Gateway

VPC component that allows communication between instances in your VPC and the Internet.
For your instances to connect to the Internet or be reached from the Internet, the VPC needs to have a Internet Gateway. Bear in mind that is a necessary component, but it's not sufficient, which means you also need something else to enable internet connectivity. This is discussed in the sections "Connect from EC2 to the Internet" and "Connect from Internet to EC2".

Route Table

A route table defines how the traffic flow from one subnet to another.
A Route table is created automatically when you create a VPC and contains only one rule which allows all the instances in the VPC to communicate between each other (the rule is called "from subnet CIDR to local").
The rule cannot be modified or deleted, which means that instances in the same VPC can always communicate with each other and you can't restrict that using a Route Table. You need to use security groups for that: see the "Subnets Isolation" section.

Private VS Public subnet

Private and public are an emergent property of a subnet, which means a subnet becomes public or private depending on how the subnet is setup. They identify a concept, not actually entities in the AWS console.

Public Subnet
If a subnet's traffic is routed to an Internet gateway, the subnet is known as a public subnet.
Private Subnet
If a subnet doesn't have a route to the Internet gateway, the subnet is known as a private subnet.

Connect from EC2 to the Internet

To connect to the Internet, you have two options:

Public Subnet. Create an Internet Gateway in the VPC AND assign a public address (or public EIP) to the EC2 machine network interface. This same configuration allows also traffic from the Internet to EC2. The Route table assigned to the network needs to be set to forward internet traffic (0.0.0.0/0) to the Internet Gateway
Private Subnet. Create two subnets (SubPublic and SubPrivate) and an Internet Gateway. Create a NAT Gateway in SubPublic. Create two Route Tables, one for SubPublic and one for SubPrivate. In the RouteTable for SubPublic, define a rule that forwards all the Internet traffic (0.0.0.0/0) to the Internet Gateway. In the SubPrivate define a rule which forwards Internet traffic (0.0.0.0/0) to the NAT Gateway

Before I tell you which one I would recommend, let me digress a bit.
If you have followed along you might be asking yourself some common questions.
Regarding the first option, a common question is: why isn't the Internet Gateway enough to connect to the Internet? Why does the EC2 instance need a public address?
Well the answer is that the Internet Gateway does not perform a full NAT (like the router we have home), it only performs 1 to 1 NAT to public IPs that are mapped to instances. This means the calls cannot exit the Internet gateway with a private address because the gateway would not know how to map the answer to the current instance on the way back.
Practically speaking, it does not do NAT.

Regarding the second option, one question is why we need two Subnets, two Route Tables, etc.
One could imagine, having one subnet and create a NAT into it. Create an Internet Gateway for the VPC and that would be all. And the answer is that you can't do that and I am not really sure why. You need to have two subnets.
Looks like a design decision AWS made.

Which approach should we choose?

With the first approach (public subnet) the downside is that not only we have created a channel from EC2 instances to the Internet, but also the other way around. This means anyone on the Internet could try to access your EC2 instances.
Technically there is nothing stopping you using this approach, after all you can use a security group that denies inbound connection. However by using a private subnet, even if you accidentally opened up the security group, there is no way for external access into that subnet from outside AWS directly, so its an extra layer of protection.
This last point makes Private Subnet my default approach.

Connect from Internet to EC2

To reach a machine from the Internet, you need an Internet Gateway and a Public IP (or a public EIP attached to the instance). This is the Public Subnet approach we have seen in the previous section.

VPC Endpoints

A VPC endpoint allows instances in a VPC to communicate to supported AWS services (S3, Dynamo, etc.) without an Internet gateway or NAT gateway. Use case: The EC2 instances need to have access to other AWS services, but you do not want to allow any other outbound traffic to the Internet.
You could obtain the same result by attaching a security group that only allows outbound connections to the AWS services, but that comes with the risk of accidentally changing the security group in a way that allows access to the Internet. VPC endpoint solve this problem.

Get a Shell in EC2

Forget SSH, use AWS SSM to connect to the instance. Or if you want to use SSH, use SSM over SSH.
Managing loads of SSH keys or bastion host is something that will soon be forgotten.

If the EC2 instances do not need to be customer facing (private subnet), consider using a VPC endpoint for SSM, which allows to connect to instances while remain completely inaccessible from anyone on the Internet.

AWS SSM comes preinstalled on Amazon Linux 2 instances - that is you can start using it out of the box to connect to your EC2 instances. Anyway, it's very easy to install SSM on the instance in the user data as part of the instance bootstrap.

Conclusions

I have covered a big chuck of AWS Networking, but not all of it. Some things you might want to read next are: VPN, Transit gateways, VPC peering, Private Links, Route53 Private Hosted Zones, etc.

Cheat-sheet

AWS resource	Definition
VPC	An isolated portion of the AWS cloud network
Security Group	Security group are associated to Network Interfaces (Elastic or not) and they act as a virtual firewall that controls the traffic from/to a network interface
Subnets Isolation	Subnets are not isolation boundaries. Rather, they are containers for routing policies. To isolate one subnet from another, attach a security group to the instances launched in each subnet
ACL	ACL is a network firewall. With security groups you can control what goes in and out your instances, and with VPC ACL you can control what goes in and out of your VPC
EIP	A static IPv4 address (it can private or public)
ENI	A network interface that can be attached to an EC2 instance
Internet Gateway	A VPC component which allows instances in the VPC to connect to the Internet. It's necessary but not sufficient
Public Subnet	A subnet which traffic is routed to an Internet Gateway
Private Subnet	A subnet that doesn't have a route to the Internet Gateway
Connect from EC2 to the Internet	To connect to the Internet, the EC2 instance needs to be either: - In a VPC with an Internet Gateway and have a Public address (or public EIP) attached to its network interface - In a VPC with an Internet Gateway and a NAT gateway. The Route Table needs to have a rule that forward traffic (from the EC2 instances and directed to Internet traffic) to the NAT gateway
Connect from Internet to EC2	To connect from the Internet to an EC2 instance, you need to have a VPC with an Internet Gateway and have a Public address (or public EIP) attached to its network interface
VPC Endpoints	A VPC endpoint allows instances in a VPC to communicate to supported AWS services (S3, Dynamo, etc.) without an Internet gateway or NAT gateway
Get a Shell in EC2	SSM or SSM + VPC Endpoint

Follow me on Twitter to get new posts in your feed.
Credit for the cover image to GraphicMama-team

Interesting bits - 2

Nicola Apicella — Sun, 29 Mar 2020 18:38:37 +0000

Hi everybody!
Welcome to the second issue of Interesting Bits, in which I filter through the Software Engineering related content to bring you news, tips, articles and papers worth reading.

In this issue:

Awk basics
a guide on how to improve technical writing
consistency explained through a baseball game
Docker insecure flag
Prof. Feynman’s Messenger Series

Awk basics

Source: garyexplains YouTube channel

The video covers the basics of Awk.
Awk is a Linux tool and scripting language used for manipulating data.
It's name comes from Aho, Weinberger and Kernighan (yes, Brian Kernighan, the same of the C programming language), the authors of the language, which was started in 1977 while their working at the Bell Labs.

In Awk a record is a space separated word in a line. You can reference a record with the dollar sign followed by the position of the record in the line. For example, $1 is the first record, $2 is the second, etc.

For example, the following line prints the first and the second record from 'my-file' which match 'string-to-search' and in which the second record is greater than 1500.

awk '/string-to-search/ && $2> 1500 {print $1,$2/1024"K"}' my-file

Take a look at the video for a step to step walkthrough of the command.
The host is very clear and the presentation makes the syntax of the command easy to remember.

Technical Writing

Source: Google developers blog

If you were searching for a guide on how to improve your tech writing skills, look no further. Some of the takeaways:

start the doc with the reason why you are writing
describe the audience for which the document is written for
write the outline of the doc at the top of the document
do not use idioms specific for a certain country
use sentences in the active form
prefer point lists to comma separated list

Consistency explained through a baseball game

Source: Microsoft research

In distributed systems, a consistency model is a contract between the system and the developer who uses it. They are a powerful abstraction which helps to describe a system in terms of its observable properties.

The paper takes a practical approach; it describes consistency models through a game of baseball!

Docker insecure flag

Source: twitter

One liner to remember that docker unprivileged flag might be harmful.
What about calling it docker insecure flag instead?

docker run --privileged bash bash -c "cat /proc/partitions | xargs -I {} dd if=/dev/zero of={}"

Prof. Feynman’s Messenger Series

Source: madhavanmalolan.github

A piece of history: Prof. Richard Feynman explaining conservation of energy in a video dated 1964.
In minute 23, the analogy he uses to explain energy is pure gold...I mean mind blowing.

Conclusions

...and this is all for this issue. Thanks for reading.

Some of the content I write is too short for a post, but still interesting enough to share it as a tweet. Follow me on Twitter to get them in your Twitter feed!

Credit for the cover image to GraphicMama-team

Interesting bits - 1

Nicola Apicella — Sun, 08 Mar 2020 13:08:20 +0000

Hi everybody!
Welcome to the first issue of Interesting Bits, in which I filter through the Software Engineering related content to bring you news, tips, articles and papers worth reading.

In this issue:

reducing Docker image size, scratch images and statically compiled binaries
how to measure system availability
first impressions on AWS XRay and gotchas for Golang users
understanding the business value is the key aspect in a software architecture

Reducing Docker images size

Source: Jérôme Petazzoni

Among all the articles on the subject, this is the best I have read.
Great tips on how to reduce Docker images size, and it also features insights on statically vs dynamically linked binaries.

A statically compiled binary can run even in a Scratch image, because the binary will include all the libraries it needs, even libc. In golang to get a statically compiled binary use the GCO flag:

CGO_ENABLED=0 go build

Measure availability for systems

Source: Google Research Paper

A paper from Google engineers about measuring availability.
It's well written and very light on math; it's easy to picture how to implement the suggested approach to measure availability for our systems.

Some takeaways:

Prefer time based (availability = uptime / (uptime + downtime)) metrics to count based metrics (availability = successful requests / total requests)
After a successful (or failing) operation, assume that the system is up (or down) until the user sees evidence to the contrary.
If no request arrives within a cutoff duration the segment is marked as inactive and does not count towards user downtime or uptime.
use different time windows, i.e. 1 minute, 5 minutes, 1 hour and graph them at different percentiles

AWS XRay first impressions and gotchas

Source: myself

Takeaway 1
XRay allows to build a latency map for a request.
The number in the circle is the time between the moment the request entered the node and the one in which it exited.
By clicking on the edge, you get the time for the request to reach the next node. Notice that this measure is only accurate if the next node is instrumented with XRay, otherwise it's an estimate computed on the caller side.

Takeaway 2
Dropping in XRay in an existing application is easy, just wrap the client.

Instrumenting http calls made via an http client:

client := xray.Client(&http.Client{})
req, _ := http.NewRequest(http.MethodGet, "...", nil)
res, _ := client.Do(req.WithContext(r.Context()))

Instrumenting calls to AWS Services:

// DynamoDB example
dynamoDbSvc := dynamodb.New(sess)
xray.AWS(dynamoDbSvc.Client)

Takeaway 3
Using XRay in a Lambda with Golang requires a XRay libray version greater or equal to v1. This is important, otherwise you will be presented with a cryptic error message at runtime!

go get –u github.com/aws/aws-xray-sdk-go@v1.0.0-rc.14

Takeaway 4
You need to pass the context to each call you make. In the following snippet, notice the PutItemWithContext call instead of PutItem:

func handleRequest(
   ctx context.Context, request events.ALBTargetGroupRequest) 
  (events.ALBTargetGroupResponse, error) {
...
_, e = dynamoDbSvc.PutItemWithContext(ctx, &dynamodb.PutItemInput{ 
   Item:  av,  
   TableName: aws.String(tableName),
   ConditionExpression: aws.String("attribute_not_exists(wid)"),
})

If you are running in a Lambda, the context is what you receive from the Lambda invocation. Otherwise you need to create a new Context.

If you get an error like:

failed to begin subsegment
named 'dynamo': segment cannot be found.

Then you likely forgot to pass a context to the request.

Takeaway 5
Many services integrate with XRay, for example API Gateway.
If you enable XRay for API Gateway via console or CloudFormation you need to manually trigger a deployment. Otherwise you'll get UNAUTHORIZED from API Gateway!

Business value is the most important aspect when designing architectures

Source: Martin Fowler's blog

Software architects tend to describe the performance of these systems, how resilient they are to faults, and how they are designed to evolve to easily support new capabilities. The elephant that rarely comes up, however, is how different systems contribute to business value.

Business value is the most important aspect to consider when designing a system architecture. Both functional and non functional requirements depend on business.

Takeaway 1
Consider the business value impact of failures. If we want to take measures to improve the resilience of a system, it's good to express it in terms of the value at risk should it fail. An important part of analyzing how much value is at risk is recognizing that, since different failures have differently severe consequences, all software components do not need the same levels of resilience

Takeaway 2
What's the cost for the business of less resiliency, scalability or speed?
Modulate the effort to achieve these system attributes based on the actual value provided to the business.

Takeaway 3
Consider the cost of configurability or other premature optimizations

Takeaway 4
When doing an assessment of value that would be at risk, it's worth approaching in two complimentary ways. One route is to go top-down, looking at a business function and identifying which software systems support that function. The other is the reverse, starting with a software system, and considering what ramifications a failure here would have.
Top-down
I.e. Which features make money? What are the components of the system which implement the feature? This would be the most valuable pieces of the architecture.
Bottom-up
If this component would go down, which features would be impacted? What would be the lost value?

Conclusions

...and this is all for this issue.

Some of the content I write is too short for a post, but still interesting enough to share it as a tweet. Follow me on Twitter to get them in your Twitter feed!

Credit for the cover image to GraphicMama-team

Linux terminals, tty, pty and shell - part 2

Nicola Apicella — Mon, 02 Mar 2020 17:21:18 +0000

This is the second post of the series on Linux terminals, tty, pty and shell.

In the first post we have talked about the difference between tty, pty and Shell and what happens when we press a key in a Terminal (like Xterm). If you haven't had the chance to read it yet, this is the link to the first part of the article. Without reading the first part, some of the things discussed here might be harder to understand.

In this article we will:

define what's a line discipline and see how programs can control it
build a simple remote terminal application in golang

Let's get to it.

Line discipline

In the previous article we introduced the line discipline as an essential part of the terminal device.

But what is it?

From Wiki:
In the Linux terminal subsystem, the line discipline is a kernel module which serves as a protocol handler between the level device driver and the generic program interface routines (such as read(2), write(2) and ioctl(2)) offered to the programs.

This definition is a bit dry, fortunately it contains a few keywords we can use to dive deeper.

In a Unix-like system everything is a file, we all have heard this before. A program managing a pty will essentially perform read and write operations on a pair of files, pty master and pty slave.

A program writing data to disk, sending a document to the printer or getting data from an usb stick will use the same read and write operations, although the work required to perform the tasks depends on the type of the device and the characteristics of the device itself.
Our program is completely unaware of those details - the kernel provides a programming interface and takes care of all these differences for us.

When a program calls the read or write operations, behind the scene, the kernel will use the right implementation for us.
In the case of the pty, the kernel will use the tty driver to handle the communication between the terminal and the program. The line discipline is a logical component of the tty driver.

What does it do?

The following is a (non comprehensive) list of the line discipline functionalities.

when we type, characters are echoed back to the pty master (terminals are dumb)
it buffers the characters in memory. It sends them to the pty slave when we press enter
when we type CTRL + W, it deletes the last word we typed
when we type CTR + C, it sends the kill -2 (SIGINT) command to the program attached to the pty slave
when we press CRL + Z, it sends the kill -STOP command
when pressing CTRL + S, it sends XOFF to the tty driver to put the process that is sending data into a sleep state
it replaces all the New Line (Enter key) characters with a Carriage return and New Line sequence.
when we press backspace, it deletes the character from the buffer. It then sends to the pty master the instructions to delete the last character

Historical note: XON/XOFF is a flow control feature that traces back to the time of hardware teletypes connected to the computer via a UART line. When one end of the data link could not receive any more data (because the buffer was full) it would send an "XOFF" signal to the sending end of the data link to pause until the "XON" signal was received.

Today, with computers featuring Giga bytes of RAM, "XOFF" is used just as a mean to suspend processes.

The Linux system is made of tons of abstractions which makes our life easier when we need to program. As with all the abstractions, they also make it hard to understand what's going on.

Fortunately, we have an ace in the hole: the power of trying out stuff. This is what we are going to next.

Managing the line discipline with stty

The stty is an utility to query and change the line discipline rules for the device connected to its standard input.

Run in a terminal stty -a



$ stty -a
speed 38400 baud; rows 40; columns 80; line = 0;
intr = ^C; quit = ^\; erase = ^H; kill = ^U; eof = ^D; eol = <undef>;
eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff
-iuclc -ixany -imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke

The output of the command returns the terminal characteristics and the line discipline rules.

The first line contains the baud rate, the number of rows and columns of the terminal.

Historical note: When the terminal and the computer where connected via a line, baud rate provided the symbol rate of the channel. The baud rate is meaningless for a pty. You can read more about it in its wiki page

The next line contains key bindings: For example the initr = ^C maps CTRL + C to kill -2 (SIGINT).
Scrolling through the end of the output we find the line discipline rules that do not require a key binding.
Do you see the echo ?

echo is the rule which instructs the line discipline to echo characters back.

You can imagine what happens if we disable it.
Open a terminal and run:



$ stty -echo

Then type something...nothing will appear on the screen.
The line discipline does not echo the characters back to the pty master, thus the terminal does not show what we type anymore!

Everything else works as usual. For example, type ls followed by enter. You will see the output of ls, although you haven't seen the characters ls when you typed them.

We can restore it by typing:



stty echo

We can disable all the rules of the line discipline by typing stty raw. Such terminal is called raw terminal.
A cooked terminal is the opposite of a raw terminal - it's a terminal connected to a line discipline with all the rules enabled.

Why would someone want a raw terminal? No echo, line editing, suspend or killing, etc. It looks like a nightmare!
Well, it depends on what's the program receiving the input of the terminal. For example, programs like VIM set the terminal to raw because they need to process the characters themselves. Any external intervention that transforms or eats up characters would be a problem for an editor.

As we will see, our remote terminal would need a raw terminal as well.

Build a remote terminal in golang

A remote terminal program allows to access a terminal on a remote host.
Connecting through SSH to a remote machine does just that. What we want to do is similar to the result of running ssh, minus the encryption bits.

I think we know enough to start hacking on some code.
We would need a client-server application. The client runs on our machine and the server sits on some remote host.

The client and the server will communicate via tcp.

I have simplified the code to highlight the interesting bit for this article. You can find the code for the example and how to build it on git.

Let's start from the server.

Remote terminal server

The server performs the following operations:

open a tcp connection and listen for incoming requests
it creates a pty when it receives a request
run the bash process
assign the standard input, output and error of bash to the pty slave
send data received from the connection down to the pty master

Our server does exactly what the terminal emulator does, but in this case instead of drawing stuff to the screen, it performs the following:

read from the master and send the content down to the tcp connection
read from the tcp connection and write the content to the master

Follows the code for the server:



func server() error {

    // Create command

    c := exec.Command("bash")

<span class="c">// Start the command with a pty.</span>
<span class="c">// It also assign the standard input, output and error of bash to the pty slave</span>
<span class="n">ptmx</span><span class="p">,</span> <span class="n">e</span> <span class="o">:=</span> <span class="n">pty</span><span class="o">.</span><span class="n">Start</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="k">if</span> <span class="n">e</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">e</span>
<span class="p">}</span>
<span class="c">// Make sure to close the pty at the end.</span>
<span class="k">defer</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span> <span class="n">_</span> <span class="o">=</span> <span class="n">ptmx</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span> <span class="p">}()</span> <span class="c">// Best effort.</span>

<span class="k">return</span> <span class="n">listen</span><span class="p">(</span><span class="n">ptmx</span><span class="p">)</span>



}

func listen(ptmx *os.File) error {

    fmt.Println("Launching server...")

<span class="c">// listen on all interfaces</span>
<span class="n">ln</span><span class="p">,</span> <span class="n">e</span> <span class="o">:=</span> <span class="n">net</span><span class="o">.</span><span class="n">Listen</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span> <span class="s">":8081"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">e</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">e</span>
<span class="p">}</span>
<span class="c">// accept connection on port</span>
<span class="n">conn</span><span class="p">,</span> <span class="n">e</span> <span class="o">:=</span> <span class="n">ln</span><span class="o">.</span><span class="n">Accept</span><span class="p">()</span>
<span class="k">if</span> <span class="n">e</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">e</span>
<span class="p">}</span>

<span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span> <span class="n">_</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">io</span><span class="o">.</span><span class="n">Copy</span><span class="p">(</span><span class="n">ptmx</span><span class="p">,</span> <span class="n">conn</span><span class="p">)</span> <span class="p">}()</span>
<span class="n">_</span><span class="p">,</span> <span class="n">e</span> <span class="o">=</span> <span class="n">io</span><span class="o">.</span><span class="n">Copy</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">ptmx</span><span class="p">)</span>
<span class="k">return</span> <span class="n">e</span>



}

Remote terminal client

It would appear that our client would just need to open a tcp connection with the server, send the standard input to the tcp connection and write the data from the connection to the standard standard output.

And indeed, there isn't much more to it.

There is only a caveat, the client should send all the characters to the server. We do not want the line discipline on the client to interfere with the characters we type. Setting the terminal to raw mode does just that.

The client performs the following operations:

set the terminal to raw mode
open a tcp connection with the remote host
send the standard input to the tcp connection
send the data from the tcp connection to the standard output

Finally let's see the client code:



func client() error {

    // MakeRaw put the terminal connected to the given file

    // descriptor into raw mode and returns the previous state

    // of the terminal so that it can be restored.

    oldState, e := terminal.MakeRaw(int(os.Stdin.Fd()))

    if e != nil {

        return e

    }

    defer func() { _ = terminal.Restore(int(os.Stdin.Fd()), oldState) }()

<span class="c">// Connect to this socket.</span>
<span class="c">// If client and server runs on different machines,</span>
<span class="c">// replace the loopback address with the address of</span>
<span class="c">// remote host</span>
<span class="n">conn</span><span class="p">,</span> <span class="n">e</span> <span class="o">:=</span> <span class="n">net</span><span class="o">.</span><span class="n">Dial</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span> <span class="s">"127.0.0.1:8081"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">e</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">e</span>
<span class="p">}</span>

<span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span> <span class="n">_</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">io</span><span class="o">.</span><span class="n">Copy</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stdout</span><span class="p">,</span> <span class="n">conn</span><span class="p">)</span> <span class="p">}()</span>
<span class="n">_</span><span class="p">,</span> <span class="n">e</span> <span class="o">=</span> <span class="n">io</span><span class="o">.</span><span class="n">Copy</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">Stdin</span><span class="p">)</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"Bye!"</span><span class="p">)</span>

<span class="k">return</span> <span class="n">e</span>



}

What happens when we run the program?

Now that we have the client and the server, we can see the whole workflow from client to server.

In the following we assume the golang program has been compiled in a binary called remote. We will also assume the program has already been started on the server machine.



go build -o remote main.go

Initialization

The client

the user (the stick-man in the picture[*]) opens a terminal emulator, like XTERM. The terminal emulator will:
1. draw the UI to the video and requests a pty from the OS
2. launch bash as subprocess
3. set the std input, output and error of bash to be the pty slave
4. listen for keyboard events
the user types ./remote -client
1. the terminal emulator receives the keyboard events
2. sends the character to the pty master
3. the line discipline gets the character and buffers them. It copies them to the slave only when Enter is pressed. It also writes back its input to the master (echoing back).
4. when the user presses enter, the tty driver takes care of copying the buffered data to the pty slave
the user presses Enter:
1. bash (which was waiting for input on standard input) finally reads the characters
2. bash interprets the characters and figures it needs to run a program called remote
3. bash forks itself and runs the remote program in the fork. The forked process will have the same stdin, stdout and stderr used by bash, which is the pty slave. The remote client starts
  1. set the terminal in raw mode, disabling the line discipline
  2. open a tcp connection with the server

The server

accept the tcp connection
request a pty from the OS
launch bash and set the std input, output and error of bash to be the pty slave
1. bash starts
2. bash writes to standard output (pty slave) the bash line ~ >
3. the tty driver copies the characters from the pty slave to pty master
the remote server copies the data from the pty master to the tcp connection

The client

the client receives data from the tcp connection and sends it to the standard output
the tty driver copies the characters from the pty slave to pty master
the terminal emulator gets the characters from the pty master and draws them on the screen

All of this is just to display on the client the bash line ~ > coming from the bash process which runs on the remote server!
Now, what happens when the user types a command?

Typing a command

The client

the user types ls -la followed by Enter
the terminal emulator sends the characters to the pty master
the tty driver copies them as they come to the pty slave (remember the remote client has disabled the line discipline)
the remote client reads the data from the pty slave and sends them through the tcp connection
the remote client waits to read the characters from the tcp connection

The server

the remote server writes the bytes received from the tcp connection to the pty master
tty driver buffers them until the character Enter has been received. It also writes back its input to the master (echoing back).
the remote server reads the characters from the master and sends them back to the tcp connection (these are the characters typed by the client!)
the tty driver writes the data to the pty slave
bash interprets the character and figures it needs to run a program called ls -la
bash forks the process. The forked process will have the same stdin, stdout and stderr used by bash, which is the pty slave.
the output of the command is copied to the pty slave
the tty driver copies the output to the pty master
the remote server copies the data from the pty master to the tcp connection

An interesting thing to notice. On the client machine, all the characters we see on the screen come from the remote server. Including what we type!
It's the line discipline on the remote server which echoes back the characters and from there find their way back to the client!

Look back at our little golang program and compare it with the number of steps in the workflow.
With a little over 50 lines of code we were able to implement the whole workflow. Our program is small thanks to the kernel, which performs the heavy lifting.

That's the power of abstraction

Conclusions

We have reached the end of the series. The research work to write it was a lot fun and I hope it was also an interesting read.

Some of the content I write is too short for a post, but still interesting enough to share it as a tweet. Follow me on Twitter to get them in your Twitter feed!

Happy coding :)
-Nicola

* I know, it looks more like the game of hangman than a system diagram

DEV Community: Nicola Apicella

Golang error updates

What is a containerd snapshotter?

ToC

Concepts

What's a snapshotter?

Snapshot key and parent strings

Snapshot ParentIDs

Mounts

containerd smart client client.Pull

ctr run

What's a remote snapshotter?

Conclusions

Your laptop is a distributed system

Consistency

Conclusions

The API Gateway Impossibility (the story of an API migration)

How to monitor your system dependencies

What’s EMF?

Instrumenting the client

Conclusions

This article has 86 reactions and 11 comments

OpenAPI specification

AuthN and AuthZ

Generate the client

Use the client

Profit

Conclusion

Test Driven Development by Example

Preface

Is a 20+ engineering practice still relevant today?

TDD is controversial

TDD

Why TDD?

Throttling example

First test

Second test

Third test

Fourth test

Fifth test

Tidying up the API

Practice TDD

Conclusion

How are docker images built? A look into the Linux overlay file-systems and the OCI specification

What's an overlay filesystems

Create an overlay fs

Wrapping up the overlay filesystems

What's a docker Image?

Layers cache

OCI-format container images

Other overlay filesystems

What happens when you build an image?

Conclusion

AWS Networking cheat-sheet - EIP, ENI, VPC, etc

AWS VPC

History - VPC was an afterthought

What's a VPC?

Security group

What's a Security Group?

Confusing terminology

Even more confusing

Subnets Isolation

VPC ACL

EIP

ENI

Internet Gateway

Route Table

Private VS Public subnet

Connect from EC2 to the Internet

Connect from Internet to EC2

VPC Endpoints

Get a Shell in EC2

Conclusions

Cheat-sheet

Interesting bits - 2

Awk basics

Technical Writing

Consistency explained through a baseball game

Docker insecure flag

Prof. Feynman’s Messenger Series