A Lovely Stroll From Launching CPython to Code Execution

Disclaimer: This article may contain more C code than Python code.

Python is fascinating and probably the closest that humanity has come so far to executable pseudo code. It attracts plenty of people who never coded before to come and try and possibly discover entirely new talents — like me. A couple of years down the line, I have moved on to core topics of Computer Science, stretching the entire curriculum of a Computer Science Degree. I have not forgotton where I came from and I look back with bliss at the time, where I bent my mind admittedly over the simplest concepts, like loops, functions and classes. Yet, the idea that I would need to run a separate program to execute my code, while since my childhood all it took to run any program on my computer was to double click a single .exe file on screen, was long time beyond me.

In this one and the next articles, I would like to explore the inner life of the CPython interpreter, foremost its runtime environment. It is not difficult to find material related to other internal features of the Python interpreter — like the compilation process or the interpretation of Bytecode. My overview is supposed to be focussing on the machine aspect of Python’s inner virtual machine. A deep dive into the source is going to be inevitable to see and understand what is actually going on. Let us first naively stroll down the callstack und miraculously gaze at how the runtime is evolving around us. We will come to the more intriguing questions eventually.

In the end, I would like to find the answers to the following questions: What does the machine in the Python virtual machine look like? How does it manage processes and threads? What is the memory layout inside the virtual machine?

From here on Python is refering to CPython version 3.9, the latest as of the time of writing. I used Windows 10 64-bit and Visual Studio 2019 to build and analyse the source. The things discussed here will likely not hold true for other implementations of Python, like Jython or IronPython. I expect the differences in using another operating system to be rather obvious whenever they occur.

I have edited and annotated any source code shown here for brevity, clarity and readability. I highly recommend reading the CPython source alongside for the having full picture.

The High-Level Overview

Is Python a compiled or interpreted language? Both. Python is compiled into bytecode, which is then interpreted by a Virtual Machine. When we feed the interpreter with Python source code, we can conceptionally imagine two steps taking place:

This is of course an overly simplified model. We will later have a brief look at the components of the Python compiler, but our focus will be the anatomy of the interpreter and its runtime.

The Project Layout

To gain a better orientation before diving into the source code, it might help to make oneself familiar with how the source directory of the CPython project is organised:

The interpreter is implemented as a shared library in the three subdirectories Objects/, Include/, and Python/. The implementation of Python’s standard library lives in a separate folder, but the C extension modules — located in the Modules/ directory— also use the Python headers, effectively loading parts of the interpreter as a library. The same is true for third-party libraries like numpy which are implemented via the Python/C API.

Traversing Down `main`

The main function shown below is located in Programs/python.c. But the actual entry point to the interpreter is Py_Main located in Modules/main.c.

Even if you have not programmed natively before, you are probably familiar with a main function. But what is wmain? Well, Windows supports both 8-bit ANSI character types as well as UTF-16, the native character type on Windows. In addition to the standard C character strings, the Windows API provides variants to all its functions accepting also the native Windows wide character strings, or UTF-16.

Three levels down the call stack, we pass by pymain_main. So far, we have fiddled a bit with command line arguments. Next a couple of initilization routines are running where a configuration object is assembled from command line arguments and environment variables. Two levels further, inside pymain_run_python, we are reaching a crossing point:

Depending on how it was invoked, the interpreter has to decide in which mode to run. Interesting note on the side: The Python interpreter has a source code representation of itself with the PyInterpreterState struct. The command line flags and arguments provided determine how to continue from here:

• The -c flag invokes the pymain_run_command branch
• The -m flag takes the pymain_run_module branch
• If instead a file name is provided, pymain_run_file is called
• Else read anything that has potentially been piped in via <stdin> and enter the interactive mode (pymain_run_stdin and pymain_repl)

The Bit Between Writing Code and Running Code: The Compiler

Regardless of which branch is taken, at one point the interpreter will have to take in Python source code and compile it. The Python compilation process involves four separate tansformations.

The raw source is first recieved by the parser for tokenization. The tokens are arranged as nodes in a Parse Tree, representing the lexical structure of the code. The Parse Tree is then transformed into an Abstract Syntax Tree (AST), where the tokens are grouped and interpreted as syntactic elements. The “grammar” of the Python language determines, whether a stream of lexical tokens represents syntactically correct Python code. Third, the AST is transformed into a Control Flow Graph (CFG). The CFG still has a tree structure. The compiler therefore must first flatten the graph before it can generate Bytecode. Finally, the compiler emits its output in the form of code objects, which contain the generated bytecode bundled with extra information necessary for the execution of this code unit.

Code objects are full-fledged Python objects as suggested by a) the object’s name and b) the PyObject_HEAD field in the PyCodeObject struct. That also means that they are fully inspectable at runtime — for example like so:

>>> def foo(x, y):
...     return x + y
...
>>> foo.__code__
<code object foo at 0x00...0F50, file "<stdin>", line 1>

>>> foo.__code__.co_code
b'|\x00|\x01\x17\x00S\x00'

We can see that the bytecode representation is very compact. In case one prefers a more human readable representation, one can use the dis module which is part of Python’s standard library:

>>> from dis import dis
>>> dis(foo.__code__)
  1           0 LOAD_FAST                0 (x)
              2 LOAD_FAST                1 (y)
              4 BINARY_ADD
              6 RETURN_VALUE

This is the disassembled version of the bytecode emitted by the compiler for our foo function. Does the Python compiler optimize code? Yes, it does — but to a very limited degree. For example, it eliminates dead code and folds simple constant expressions. If you are interested in the kind of optimizations that Python applies, have a look at PyCode_Optimize in Python/peephole.c. Before continuing our traversal, I would like to point out one oddly named field in PyCodeObject, called void *co_zombieframe. It is an artifact of Python’s memory management strategy and it will reappear later in a more sensible context when talking about memory management.

This is as much as we are going to discuss the compiler. Interested readers can find more detailed information in the Python Developer Guide (s. [5]) and although it is a bit dated by now, I can also highly recommend Eli Bendersky’s Blog (s. [2]).

The Final Mile: Code Objects and Code Evaluation

Back to where we were: Retrieving our source code is a bit more involved when running in interactive mode or when running a module in Python, because Python has more work to do gathering all the necessary files or prompting input from the user. In fact, when you are running a module, the interpreter actually dispatches most of the responsibility to the runpy module — also part of Python’s standard library. But regardless of the interpreter mode, paths converge again when the code object is handed to run_eval_code_obj. From there it is passed down some unspectacular functions until it hits _PyEval_EvalCode. Meanwhile, we have arrived in Python/ceval.c.

Here are three objects that are tightly connected to teach other: The already familiar PyCodeObject, PyFrameObject, and PyThreadState. The original function including the parts that I have omitted is a bit of a mouthful. Most of it serves in the initialization of the frame object. A frame object can be understood as a runtime representation of a code object, not unlike a process is a runtime representation of a program.

If you already had some familiarity with Python’s internal architecture, you may already know what comes next: The core interpreter loop with an infamously large switch-statement.

Huh, odd... This is not the 2000+ lines core evaluation loop that we expected to see. The evaluation function for this frame is dynamically invoked through a function pointer that had been stored inside the PyInterpreterState instance at some point earlier during initialization. The function pointer is opaque to where precisely it dispatches, but its naming gives a clue. To understand what happens beyond this point, let us step back again and look at the initialization process.

The Runtime

Going back to where we came from, remember that we passed the first initialization routine in pymain_main — the third level in the call stack. We briefly mentioned that initialization took place, but stepped over the call to pymain_init.

Runtime State

The initialization of Python takes place in three distinct steps. The first is the initialization of the Python runtime. The _PyRuntime is statically initialized in Python/pylifecycle.c. It is a _PyRuntimeState struct, which itself is defined in Include/internal/pycore_runtime.h. It monitors a number of behind-the-scenes-states not directly exposed to userspace.

Figure 4 shows the three fields of the runtime state, which are going to be of most interest going forward. The first field appears to be a linked list of interpreter states. The next two fields point two subsystems whose naming is a bit less obvious: _ceval_runtime_state and _gilstate_runtime_state.

The ceval state is the first subsystem that gets initialized by _PyRuntimeState_Init_impl in Python/pystate.c. The ceval state is a proxy for Python/ceval.c. Its responsiblity is to manage and ensure save access to the frame evaluator — _PyEval_EvalFrameDefault, which we have not looked at yet. Oddly enough, the evaluator is not bound to the ceval state, but — as we have seen before — it is referenced and called through the running interpreter state instance. Ceval state still fulfills a crucial role in the evaluation of frame objects: It keeps track of who is currently in possession of the Global Interpreter Lock (GIL) and who is therefore allowed to enter the evaluation loop. What precisely is the GIL? This is a heavily debated relic of the of the time when mulit-threading still meant multiple threads executing on one core, not many. The role of the GIL and what it does will become clearer in the next article, when we are talking about parallelism and data coherence.

After setting the locale and some environment variables, the default allocator for the interpreter is configured. There are four different allocators available and each comes in three domain specific flavours: PYMEM_DOMAIN_RAW, PYMEM_DOMAIN_MEM and PYMEM_DOMAIN_OBJ. The last one is obviously the domain for allocating Python objects, while the prior two are passing allocation requests through to the system allocator and differ only in whether they control for thread safety or not. The available allocators are default, debug, pymalloc — incidently the default allocator — and malloc.

Interpreter, Garbage Collector and the Main Thread

The last bit of the initialization takes place in two steps: First the core initialization and then the main initialization. pyinit_core in Python/pylifecycle.c creates the first interpreter instance and with it the first thread. In PyInterpreterState_New we find what we were looking for:

The interpreter receives its frame evaluator: A function pointer to _PyEval_EvalFrameDefault. The new interpreter is then added to the list of runtime interpreters that we have seen in figure 4. We can spot one more important submodule of the interpreter being intiailized: the Garbage Collector.

The initalization of the garbarge collector is done by _PyGC_InitState in Modules/gcmodule.c. Without knowing yet the precise inner workings, we see the garbage collector receiving an array of three GC generations. Each generation in turn has a head pointer to a linked list of objects that are tracked by the garbage collector. The garbage collector itself maintains a head pointer to the generation zero list of objects.

The count is the number of live objects currently tracked in each generation and threshold is the number of tracked objects per generation that will trigger a collection attempt by the garbage collector. The thresholds are initialized to 700, 10, 10 for the first, second and third generation respectively. You may think that the thresholds appear low enough that they should be frequently hit even by average applications. After all, every integer, every float is a PyObject with all the associated baggage. But the Python garbage collector in fact does not track every type of Python object, but only those at risk of creating reference cycles — primarily mutable containers. Reference cycles prevent the reference count of an object to ever reach zero and therefore, from being deallocated — in other words: they leak memory. The garbage collector’s role in Python is to prevent this from happening.

Next a new PyThreadState is created. It receives a accessor method to retrieve its own current frame and then takes the Global Interpreter Lock.

After the runtime core has been initialized, the first interpreter and its first thread is up and running. The main initialization, which finalizes the initialization process, lastly sets up all the builtin modules, like sys and __main__, and other features exposed to the Python programmer.

With the frame evaluator we have all the building blocks together to enter the evaluation loop: An interpreter and a thread providing the necessary context for the evaluation, a frame object to evaluate and the associated code object with the list of opcodes. Before we proceed, I highly recommend taking a look at the main evaluation loop — it is well document, it is massive, it is beautiful, it is most of all outstandingly intuitive to read, which is an achievement in itself.

Conclusion

The attentive reader will have noticed, that I have not in fact answered the initially posed questions to a satisfying degree. But for the sake of my own sanity and the attention span of everyone else, we are going to spend time on each question in a dedicated article respectively, looking at the Python runtime from various angles.

In this article we have approached the CPython source code from a naive, but hopefully intuitive perspective: We started from the entry point of the interpreter, gradually proceeding further downward the callstack and closely observing how the execution environment evolves. In the process we have located a whole series of crucial subsystems almost in a drive-by fashion.

The next article will be concerned with Python processes and threads. We are going to explore in more detail the roles that the interpreter state, the thread state and frame objects play in the evaluation of Python code. We are going to look at how Python organises and retrieves data at runtime, how it maintains data coherence and order of execution in a multi-threaded, multi-process environment, and in the process lift the mystery around the Global Interpreter Lock.

Lastly, we are going to have a look at Python’s memory management strategy. This involves Python’s strategy for allocating and deallocating objects. But it also touches the life time management of objects and how to use the available memory space most efficiently to reduce performance overhead.

It is going to be tough work, but I am already looking forward to it.

References

[1] A. Shaw: Your Guide to the CPython Source Code. Real Python, 2019, https://realpython.com/cpython-source-code-guide/. Last visited Aug. 02, 2020.

[2] E. Bendersky: Python internals. 2009–2015, https://eli.thegreenplace.net/tag/python-internals. Last visited Aug. 02, 2020.

[3] G. v. Rossum: The History of Python. Python’s Design Philosophy. 2009, http://python-history.blogspot.com/2009/01/pythons-design-philosophy.html. Last visited Aug. 02, 2020.

[4] P. Guo: CPython internals: A ten-hour codewalk through the Python interpreter source code. 2014, http://pgbovine.net/cpython-internals.htm. Last visited Aug. 02, 2020.

[5] Python Developer’s Guide: Design of CPython’s Compiler. Python Software Foundation, 2020, https://devguide.python.org/compiler/. Last visited Aug. 02, 2020.

[6] Python Developer’s Guide: Exploring CPython’s Internals. Python Software Foundation, 2020, https://devguide.python.org/compiler/. Last visited Aug. 02, 2019.

[7] Y. Aknin: Python’s Innards. 2020, https://tech.blog.aknin.name/category/my-projects/pythons-innards/. Last visited Aug. 02, 2020.

Python Internals: An Introduction was originally published in Sourcerer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Sourcerer has come a long way from Sergey and I’s original idea. Since then, we’ve managed to create and grow the only engineering online resume that is built entirely from an engineer’s commits. This resume is used by the community to not only find their perfect job by matching their confirmed skills and abilities to the repositories of potential employers but to also learn about themselves, connect with others, and grow professionally as an engineer.

Check out a few of our resumes:

1. https://sourcerer.io/wanghuaili
2. https://sourcerer.io/carlomazzaferro
3. http://sourcerer.io/lmsanch

We have a lot planned for the future but one specific vector we’ve been hearing about has attracted our attention immensely. Specifically, on how we can use our technology to improve engineering retention within an organization.

Engineer’s are valuable, extremely valuable given this day and age. They are the most powerful workforce group in the world and their demand is easily seen with 11.6MM total tech jobs and 3.7MM job postings in the US for 2018 alone. While impressive on it’s own, another way to look at these massive numbers is to say that every third employee can walk out and get a job elsewhere. This satiation, is amazing and I can only see it increasing as time goes on and continued investments in emerging technologies grows (74% YoY).

This incredible momentum in the industry unfortunately also creates an increasing challenge to retain and satisfy the existing workforce. The latest data out there is showing that the median tenure at top tech companies to be 2 years. Yes, that’s right, only 2 years and an employee is onto his or her next opportunity. The most interesting thing about this though is the cited reasons for leaving. In Hired’s, “2019 The year of the software engineer” study they have found that while compensation is one pretty obvious cause, one of the more striking reasons is that because “40% of engineers leave to learn something new”. This is incredibly insightful, especially when you look at the cost on average to lose a tech employee is 125% of salary. This cost and frequency is a real problem and begs for a solution.

A recent article in the Washington Post, highlights this issue very well and details the problem set and advancements IBM has specifically done. IBM wants to keep its employees from quitting, and it’s using artificial intelligence to do it. The CEO of IBM, Ginni Rometty said that thanks to AI using their internal “proactive retention” tool, IBM can predict with 95 percent accuracy the employees who are likely to leave in the next six months.

Further in the article, Diane Gherson, IBM’s chief human resources officer, says that they consider thousands of factors such as job tenure, internal and external pay comparisons, and recent promotions all the way to much more simple factors such as the length of an employee’s commute to discover the patterns needed to predictively determine potential losses.

While utilizing this HR “metadata” that IBM is working on is extremely fascinating, what we realized is that specifically for technology employees, an engineer’s authored source code is the best data to be analyzing. Something we already have been capturing and processing with our sourcerer.io engineering resumes.

Over the past month we have devoted some real R&D efforts to explore how our existing technology can be used to solve this. The results have been enlightening and has resulted into a product that we’re calling tech talent analytics for human resource and engineering management that helps companies focus their IT retention efforts by mining the source code that their software engineers author. Our AI summarizes an engineer’s expertise, engagement, influence, and importance by observing their work in real time. We then connect these signals to compensation data and other employee metadata along with global trends to help companies map out their IT talent, identify critical contributors, plan succession, and discover training opportunities.

These retention intelligence tools are the clear future. They are automatic, real time, objective, and assessed alongside the actual skills, abilities, and habits of the workforce as compared to a world where retention is currently being monitored manually, infrequently, invasively, and subjectively.

There is obviously a lot here and still more that I have yet to capture in writing but would be very interested to hear from human resource and engineering leaders to refine our product. We are also looking for organizations that would be interested to pilot with us. As a pilot participant you’ll have 24/7 customer support for your trial integration, direct feedback & customizations for your specific client needs, and preferential support for 3rd party HR system integrations.

Request access to our beta pilot program today.

If you happen to stumble on this post and have any interest, we’d be very eager to hear from you.

Using AI to keep engineers happy at work was originally published in Sourcerer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

During the last years, thanks to github, gitlab, bitbucket, launchpad and other products, issues have become popular even for small projects. We are now pushing, merging pull requests, having release cycles that follow semantic release standard, having our software controlled by CI and indexed into package managers. There are still ways our work of developers can be made easier and more stable through tools and processes. Today I am going to explore with you a way to deal with some side work that you already should be managing with any of your projects: dealing with changelogs.

Commit messages have always been a great way to keep track of changes in a project. However, in a lot of projects (either personal or professional ones, maintained by a company or not), commit messages were often quite messy and it was not possible to use them to propose to end users something to read. This is due to some regular problems occurring throughout the life of your project :

• Everybody has their own way of committing and writing commit messages. Even if there was a standard established, it is quite often that people can’t follow EXACTLY the commit message guidelines on their own. There is always a commit message which will be a bit different, because somebody forgot to put the issue number they were working on, or to specify if the commit is a fix, a new feature…
• Commits are sometimes made in a hurry, or in bad conditions, and you definitely can’t provide to your end users a commit log. There will be a lot of polluting data among what you want to bring to them.
• Commit messages are not tied to releases of your project. So using these messages as changelogs would not provide enough information.

If you could get rid of those blocking points, you would have a perfect way to automatically log info painlessly while developing, without spending time afterwards to publish and think about the changes you made between two releases.

Introducing Commitizen

Commitizen is a node package that provides a way to easily standardize commit messages, and help you keeping a clear and parsable convention thoughout your project history. You will then be able to distinguish commits that :

• fixes an existing issue (and add a link to an issue discussion)
• adds a feature (with a link to the specification)
• change the build system
• or any other type of change for your project.

Everything is handled via a script call that shows a commit wizard into the console, that replaces the git commit command. Step by step, you will be asked for the type of commit, the commit message, the long description, and also asked to provide an issue that gets solved by the commit, and so on.

This wizard will prepare a nice and parsable commit message for you. Using this wizard, there is no way to find a poorly formatted commit message among your commit history.

You can set up commitizen to adapt to different standards by adding configuration into the package.json of your project, or having a configuration file into the root directory of your project.

Once this is set up, you will see you commit history become more and more tidy.

Now that you can rely more on the data displayed by a git log command, you can use it to generate a changelog into your project.

There are different ways to achieve this, but Auto changelog might be a good start. This node package can be used to generate a file containing a changelog. You can use it to generate a changelog.md file, using the default options.

$ auto-changelog --output changelog.md

You can put this in your scripts in the package.json file, and you may want to add a commit hook as well to generate your changelog. If you manage versions of your project with git releases, Auto-changelog will take care of writing your changelog entries sorting them by version.

You now have something relevant and fully automated to integrate to your project. No more changelog writing.

But wait, there is more.

Integrating the changelog in your own app

Having a markdown changelog is one good thing for your project. But you can also use auto-changelog to bring this data straight into your app. In this part, we will assume that you are developing a web project and integrate this information into you frontend, in a dedicated section.

auto-changelog --template json --output changelog-data.json

This command will generate a json document in your project, ready to be integrated in your project.

Now in your source folder, you can generate a complete page using this.

It is then conceivable to build a full interface into your webapp to display your changelog.

You can push the implementation and go really far with that. You can think about making a full page where people can browse past version, see issues linked to commits, and even interact with those issues (see the whole discussion, add some comments, …).

In fact, if users are able to interact with your changelog, they are able to interact with the past of your product. Making them able to add some questions to changelog items would help you dealing with support-related concerns, helping them with questions they might have about the present state of your project. And if you let them see ongoing issues, you can make them able to contribute more easily, thus taking part in future developments.

Conclusion

Integrating this kind of workflow can cost you quite some times as it may require to set up commit hooks and install several dependencies into your project. However, you will only have to set it up once and it saves you time and efforts on parts that requires meticulousness.

As communicating with your users is a very important and often neglected part of the work for technical teams, It will give you some easy extra points on the long run.

Integrating a page to see changelog will show to your users that you care about communication, transparency. And adding a page to add and comment issues will prove them you take care about their voices.

Resources

The commitizen command line utility :

GitHub - commitizen/cz-cli: The commitizen command line utility. #BlackLivesMatter

Auto changelog, used to generate the changelog from commits :

CookPete/auto-changelog

Using commit message standardization to enhance your release and feature management. was originally published in Sourcerer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Node is becoming one of the most elected choices by developers for modern web servers. You can build web servers in various elegant ways, using up-to-date tools such as ExpressJS.

However, a lot of developers had a hard time to stick on a high level framework to build web servers for their applications, sticking with tools that will stay light but will require a lot of configuration and wiring to produce a complete setup for a large project.

Today, we are going to focus on a higher-level framework that comes with batteries included, and that will allow you to implement advanced features easily.

Meet AdonisJS

AdonisJS is a node framework inspired by the well-known Php framework Laravel. It relies on concepts such as dependency injection and service providers to make you design beautiful, reliable and easily testable code.

Get prepared to leave spaghetti code in favor of reusable OO structure.

Installation

First things first, to create a new AdonisJS server, you will need to install Adonis CLI, the command line tool that will help you manage your projects :

npm i -g @adonisjs/cli

This will provide you a “adonis” command to use in your terminal

To create a new adonis application, use the subcommand “new” :

adonis new my-adonis-server

As explained in the log, is uses the default fullstack-app template (adonisjs/adonis-fullstack-app), to create your project in a “my-adonis-server” folder.

You can now go to this folder and start serving your app in development mode :

cd my-adonis-server
adonis serve --dev

Your app is now served from your machine, you can now dive into your project !

Keep the server running for the rest of the introduction.

A bit of modeling

Let’s now create one API to manage a resource. In the following chapter, we will create an endpoint to manage tasks.

First, let’s setup the project so it uses sqlite :

adonis install sqlite3

Then create the Task SQL table :

adonis make:migration tasks
# select "create table"
adonis migration:run

Note that every time you want to edit your database, you will have to create a migration.

Migrations will allow you to save every modification you are doing to your models, and will update data to follow the new formats and rules you are implementing.

It is possible to define a “up” method, and a “down” method, depending if you want to switch to go to a newer or older version.

Then, create the model and controller related to this table :

adonis make:model Task
adonis make:controller TaskController
# select "For HTTP requests"

You now have set up you database, created a table, a model, and a controller. The source code for the model and the controller is present on the app folder.

You might have noticed that as you entered the last commands, the server automatically detected that your project changed, and took care of reloading the new setup.

Now we can add some content to our model. By creating another migration.

adonis make:migration tasks

Now edit the new file under database/migrations

And run this migration

adonis migration:run

Now, you tasks models have a title, description, and done properties.

Creating a page displaying content

Now let’s create a page that displays a list of tasks.

//Route.on(‘/’).render(‘welcome’)
Route.get('/', 'TaskController.home')

Then, on your TaskController, add the code handling the “/” route.

And add the template of the page in “resources/views/tasklist.edge”

In the “public/style.css”, delete all the css rules, to put :

This will display an empty list of tasks on “localhost:3000/” (so basically, nothing at the moment !)

It is currently empty because there is no tasks at the moment on the database. Let’s fix this!

Creating tasks in your database

For the sake of this tutorial, we will create our first tasks on the TaskController method we already defined:

Load the tasklist once to insert your tasks in the database, then erase or comment those methods.

You should now see your tasks on the page.

Creating new tasks

Once this working, we would like to add new content via a new task form.

On your task controller, enter the task creation logic :

The task creation form in your tasklist template :

And the post route in “start/routes.js”

Route.post(‘/task/create’, ‘TaskController.create’)

Now you can add tasks from the form displayed in your task list.

Deleting Tasks

To delete tasks, the implementation is pretty much the same as the one we did to create tasks :

• Add a delete method in the controller
• Adapt the template
• Create the DELETE route

TaskController :

“resources/views/tasklist.edge”

“start/routes.js”

Route.get(‘/task/delete/:id’, ‘TaskController.delete’)

A bit of templating

Our application is going to grow. In order to reuse some parts of the html, we are going to define a main layout, and include the specific code of each page into it.

Create a “layout_main.edge” file into “resources/views”. This file will include the base of our page, and will be used by each page we create.

Now you can refactor tasklist.edge

Authentication

You have probably already seen that there are some files to manage users in your project (“app/Models/User.js”)

First, let’s add a UserController :

#Choose "For HTTP requests"
adonis make:controller UserController

Go to the router (“start/routes.js”) and some routes :

• two routes for the login and register process, displaying the templates with the forms
• two routes for receiving login and register data and handling user creation and login
• One route for the logout process

Route.on('/register').render('register')
Route.on('/login').render('login')

Route.post('/register', 'UserController.create')
Route.post('/login', 'UserController.login')

Route.get('/logout', 'UserController.logout')

Then, add the templates :

“resources/views/login.edge”

“resources/views/register.edge”

“Controllers/http/UserController.js”

You can also modify the tasklist template to add the register and login links, and a logout link if the user is logged in.

You can also include the create task form into the loggedIn conditionnal, in order to prevent anonymous users to create tasks.

More features

We now have seen what a basic and naive approach to build a web app can look like. You might have thought about a lot of other features to include in the project. Here is a quick list of other things Adonis can provide to you :

Relations between models

You want the tasks to be owned by users, you can create relations between models with the following syntax :

You can also use hasOne, belongsTo, belongsToMany and manyThrough.

See the docs for more details https://adonisjs.com/docs/4.0/relationships

Validators

You can use validators to check is the data flowing into your controllers has the right format, and emit messages (via session.flash for instance) when some errors occurs.

Validators are a 3rd party npm module : https://www.npmjs.com/package/adonis-validator

Using websocket instead of HTTP requests

As you might have seen when creating controllers, you can also generate controllers that are designed to use websockets.

More info here : https://adonisjs.com/docs/4.1/websocket

Internationalization

A full guide to make you app multilanguage is available on the Adonis docs as well :

https://adonisjs.com/docs/4.1/internationalization

Conclusion

Adonis is a great choice for those who need a full-featured web server framework, and who need to keep control over your implementation.

This framework could be of a great use if you want to kickstart a project, if you want to follow usual guidelines and concepts. It will help you to implement data migrations, keep your code clean, handle data validation…

However, integrating exotic libraries can be painful. Adonis extensions need to be specifically built for this framework. This would be perfect for Adonis and it’s users if the framework would be in a monopoly situation, which is not the case.

This guide was just covering the basics of Adonis, and there is still a lot to write about. If you enjoyed this guide, I encourage you to go to see the official Adonis website.

https://adonisjs.com

I personally think that this framework can provide a really nice way to deal with a lot of problems that can appear while developing a big project. I really like the way data migration is handled, and how you can easily split your code to avoid having a messy codebase.

I feel a bit concerned about documentation and modularity though; guides and API reference are present are nicely managed, but as every adonis extension is designed only for this very framework, it make me feel concerned about having a lot of specific guides to read to be able to use add-ons, and those guides won’t help me with another framework.

And you? What are your impressions on Adonis? Would you rather use a full-featured framework with all batteries included, or do you prefer the modulable Express? Feel free to comment!

AdonisJS : a full-featured node framework for modern web servers was originally published in Sourcerer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Simple example how to use Value Object in your go project

Let’s consider super simple example how you can use Value Object in your go project.

Prerequisites

Suppose we have simple project and we have to implement user registration form, for now we care only about 2 fields: name and email.
Super simple case so far…
And with this created user we have to do 3 actions: save user in db (obviously), send welcome email and add user into search.
Still nothing difficult…

Preparation

With thoughts in mind about: separation of concerns, reusability, high cohesion, low coupling, and single responsibility we will create 3 services (or components or modules or bridges to separated microservices) which will have names:

• user persistence
• mailer
• user search

Names may be differ, but I’m sure you’ve got the gist of all these services.
Why do we need this? Couple of reasons:

1. Because user may be created in db, may be updated, may be deleted from db, etc. — whole bunch of stuff related only to user persistence in database.
2. Because we need to send not only welcome email, but also confirm email, or halloween greeting, etc. — whole bunch of stuff related only to emails.
3. Because we need to add user into search, delete user from search, update user, etc. — only search related stuff.

Implementation #1

For first simple implementation we may have something like this:

After glance it looks pretty common… nothing ugly or dangerous so far…
Here we just obtain name and email from params or request and pass these data to all downstream services.
But, hold on…
We have to validate name and email in function CreateNewUser to send response with errors in case of invalid data. And we have to validate name and email in function db.SaveUser because this function might be called from different places (create, update, delete, create after oauth2 login, etc.) and there is no guarantee about data validity, therefore have to validate.
Same with mailer.SendEmail it may be welcome email or reset password email from another service, or personalized marketing email, or email about suspending account by moderation team which is called from another internal service without validation, etc. — same situation with data validity….
And same situation with search.AddUser maybe devops team is creating new search cluster and call this service from shell script to put all users into search, who knows about data validity… have to validate.

And it comes down to situation when we have same validation in many places — it’s bad. Moreover, suppose business team decided to add one more field: age or country or phone number… we have to update all validations and all functions signatures — it’s very bad.
Value Object to the rescue!

Implementation #2

It isn’t gorgeous invention to say that we can put all fields related to create new user process in one struct, like this:

And let’s add one more addition to this struct: this struct must carry only valid data, and must ensure that it contains 100% valid data!
Let’s see how it may be bone:

No big difference here, added errors to hold all validation errors for all fields and to provide all errors in one call. Also name and email non-exported — it ensures that no one will create value object by doing something like this: vo := Instance{Name: “invalid email”}

To create new value object you have to use New function:

It’s the only way to create value object
(technically it’s possible to do vo := Instance{} and use it downstream, but this blank value object will produce development time errors, hence it is useless). In this example New function receives map with fields name and email but you can pass JSON string or even request body or query string or something else, it’s up to you how to get data into value object.
Main purpose of this function — perform validation to provided data, for this we have vo.initName(data) and vo.initEmail(data), also these functions perform assignment of valid data to struct’s fields. You may perform extra actions here, if you wish (convert from one data type to another and so on).
Also it’s important to admit that this function returns value not pointer to struct, it’s done intentionally with purpose to provide immutability. Due to this once validation performed successfully — value object contains only valid data (and can’t be changed).
Value object must have something like this:

Here we have super simple validation, but you may use external library here or create own to validate data provided by user. Also you may convert data from one data type to another, or assign default value.

And last part — getters:

Getters little bit boring, but as far as we have non-exported properties we have to have these getters, and the good thing about getters — they tiny and super simple.

Let’s see whole value object:

Hope you find that whole value object looks simple, concise and clear.
There is no complicated or confusing stuff. And it’s super simple to use, re-use, cover with tests, extend and maintain this value object.

And the good part — our main function CreateNewUser now may looks like this:

Benefits of doing so:

• Value object alway valid and alway contains valid data.
• We have map of all validation errors in case when provided invalid data into value object.
• Value object immutable.
• We can loosely use this value object downstream without validation and other repetitive stuff.
• In case we need to add one more parameter — we don’t have to change all functions signatures.
• We have separation of concerns and follow single responsibility principle and many other nice principles.

Conclusion

Hope you find value object as good idea and will use it in your project.

Also you may check out this value object from real life project and see the whole picture, and how all project’s layers looks together (value object, controller, service and database).
With this example truly believe you admit that value objects help to write more clear code and help focus on important stuff but not on validation stuff.

GO ValueObject was originally published in Sourcerer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ruby is a beautiful language. It’s worst, and best quality is that once you use it, you likely won’t want to use anything else.

That’s good because you can do nearly anything with it. It’s bad because it’s not as commonly used as it should be.

Ruby’s approach to treating everything as an object and meta-programming abilities give it an array of immensely powerful debugging tools and techniques.

Whether you are a beginner or an experienced Ruby developer, working on Ruby applications or Ruby on Rails websites, I aim to share some helpful, late-night-saving tips that will help you find those pesky bugs in your Ruby applications.

Inspector Ruby

The “inspect” method is built into every class and is incredibly useful in peering inside objects. Here’s a trivial example:

irb(main):001:0> a = "Hello, World!"
=> "Hello, World!"
irb(main):002:0> a.inspect
=> "\"Hello, World!\""

In this case, inspect works much like var_dump in PHP. Let’s look at an array:

irb(main):001:0> a = [“Hello”, “World”]
=> [“Hello”, “World”]
irb(main):002:0> a.inspect
=> “[\”Hello\”, \”World\”]”

Helpful, but for even better output in JSON, require the json module and use .to_json on any object:

[1] pry(main)> require 'json'
=> true
[2] pry(main)> a = ["Hello, World!"]
=> ["Hello, World!"]
[3] pry(main)> a.to_json
=> "[\"Hello, World!\"]"

Monkey Patching to the Rescue

Ruby allows for monkey patching — a goofy name for an incredibly powerful technique. At any time during program execution, you can augment existing code by simply redefining it. Consider the following code:

#!/usr/bin/env ruby

class Demo
  def initialize
    puts "Hello, World!"
  end
end

a = Demo.new

class Demo
  def initialize
    puts "Goodbye, World!"
  end
  def wave
    puts "*waves*"
  end
end

b = Demo.new

b.wave
a.wave

In this example, we see that the Demo class acts as expected when we create a new object named “a”. But then we “redefine” the class, both changing the initialize method and adding a new method. When we create a new object named “b” it runs the initialize method with the new text.

However, note that object “a” takes on the “wave” method as well, even though it is already defined. Ruby’s monkey patching allows for surgery on any class even if it already has existing objects created from that class.

How does this help us with debugging? By being able to insert new methods or redefine existing methods in classes we can add debugging printouts or override variables or tests on said variables. You can even override methods to include a logging function to log the details of objects and the actions taken upon them to disk.

Being able to do this at any time is useful, but why not put this meta-programming power to even better use with on-demand introspective debugging.

There’s a tool to do just that — Pry.

Prying Into Your App

Pry is an incredible tool for debugging. It’s a full fledged replacement for irb that is great for standalone use or can be easily embedded into your project.

Pry treats scopes like directories, allowing the cd command to be used to navigate between scope layers, and ls to list instance and local variables:

[1] pry(main)> class Demo
[1] pry(main)*   @a = "This is a demo."
[1] pry(main)* end  
=> "This is a demo."
[2] pry(main)> cd Demo
[3] pry(Demo):1> ls
instance variables: @a
locals: _  __  _dir_  _ex_  _file_  _in_  _out_  _pry_
[4] pry(Demo):1> cd

True to Ruby’s monkey-patching nature, you can use Pry to edit your code within the execution of your program. Here’s an example:

[1] pry(main)> class Demo
[1] pry(main)*   @a = "This is a demo."  
[1] pry(main)* end  
=> "This is a demo."
[2] pry(main)> cd Demo
[3] pry(Demo):1> ls
instance variables: @a
locals: _  __  _dir_  _ex_  _file_  _in_  _out_  _pry_
[4] pry(Demo):1> @a
=> "This is a demo."
[5] pry(Demo):1> @a = "Hello, World"
=> "Hello, World"
[6] pry(Demo):1> ls
instance variables: @a
locals: _  __  _dir_  _ex_  _file_  _in_  _out_  _pry_
[7] pry(Demo):1> @a
=> "Hello, World"
[8] pry(Demo):1> cd ..
[9] pry(main)>

Note at the end I used “cd ..” to move “up” out of the “Demo” class scope and back to the default scope.

The “show-method” function allows you to display the source code of a method:

[1] pry(main)> class Demo
[1] pry(main)*   def initialize
[1] pry(main)*     puts "Hello, World!"
[1] pry(main)*   end  
[1] pry(main)* end  
=> :initialize
[2] pry(main)> show-method initialize
Error: Cannot locate this method: initialize.
[3] pry(main)> cd Demo
[4] pry(Demo):1> show-method initialize

From: (pry) @ line 2:
Owner: Demo
Visibility: private
Number of lines: 3

def initialize
  puts "Hello, World!"
end
[5] pry(Demo):1>

Note that the first command of “show-method” didn’t work because I wasn’t in the “Demo” scope.

If you install the “pry-doc” gem, you can see the C source code for built-in objects:

[1] pry(main)> show-method String#puts

From: io.c (C Method):
Owner: Kernel
Visibility: private
Number of lines: 8

static VALUE
rb_f_puts(int argc, VALUE *argv, VALUE recv)
{
    if (recv == rb_stdout) {
 return rb_io_puts(argc, argv, recv);
    }
    return rb_funcallv(rb_stdout, rb_intern("puts"), argc, argv);
}
[2] pry(main)>

This is extremely useful when trying to debug behavior with a built-in method.

Adding Pry to an Existing Ruby Program

If you enjoy games, especially text-based games, and you haven’t read about Legend of the Sourcerer, go do it now. Open it in a new window. I’ll wait.

Neat, huh? Since it’s command-driven, we can add a command “~” that will break out into a Pry session.

In LOTS, you can use Pry to examine current game variables and even edit the map. While useful for cheating, you could also refer to this handy console when trying to debug new features you add to the game.

Pry integration into LOTS is in the current version on GitHub. Here’s the commit where I integrated Pry. But adding Pry to an existing project is as simple as adding:

require 'pry'

Then put:

binding.pry

Where you’d like to invoke Pry. When this line executes, the session will start immediately.

Using Pry with Rails

There is a gem which handles Pry integration into Rails for you. If you’d rather not change your app and just want to use the Pry console with Rails, you can do so by running

pry -r ./config/environment

This will start a Pry session with your Rails app loaded. Navigate your application class structure with ease with all of that Pry debugging goodness at your fingertips.

Deep in the Ruby Mines

I hope you’ve enjoyed and gleaned some helpful hints from this look into debugging Ruby programs. Armed with Ruby’s powerful meta-programming and introspection and the awesomely astounding Pry gem, your bugs don’t stand a chance.

Turning Bugs into Gems: Debugging Ruby Applications was originally published in Sourcerer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today’s web content is amazingly rich. It varies from standard HTML to complex web apps full of media; such as animation, data visualization, video games, mixed reality and VR, to name a few. Such content is often inaccessible or poorly accessible, which means a broken user experience for those who rely on assistive technologies to interact with web sites. This experience, can be annoying at its best to empty web pages as its worst. Sometimes web authors even provide alternative content for these users. Needless to say, it’s not always equivalent. What alternative content for a video game is out there?

Websites are built inaccessible not only because rich media content is highly dynamic which makes it not trivial to describe its semantics but also because there’s no proper technology on the web to express such content semantics to the assistive technologies. You may wonder why, with all the technologies to create web content out there, why we do not have one to make it accessible? Here’s an explanation.

WEB STANDARDS

A key web technology to make content accessible is ARIA. ARIA stands for the Accessible Rich Internet Application. This is a web standard, which defines a collection of attributes you can place on a DOM element to specify role, properties and state. For example, <div role=”button” aria-disabled> will turn the HTML:div element into a disabled button semantically, what makes the assistive technologies users to perceive the div element as a button.

It’s fair to say that ARIA is generally recognized as a complete set of semantic annotations available to web developers. In other words ARIA shapes the web author’s vocabulary when it comes to accessibility.

ARIA is a remarkable technology indeed, and it has a good story of success. It enables a great number of websites to be accessible. There’s one important bit in this story though: ARIA vocabulary replicates HTML semantics at large. For sure, it provides a bunch of extra cool things, such as grid or tree controls, but it doesn’t let you go very far beyond.

So what If you have a use case that falls out standard HTML? First, you can work on alternative content that can be mapped into HTML. You may succeed or not, which depends on complexity of your case. And second, you can propose an ARIA extension. It’s not as bad as it may sound. There are good precedents, for example, ARIA-DPUB extension. I know it is a slow process, but it works. You’d better have a good weight in the industry though to make it happen.

PLATFORM APIs: GLANCE AT HISTORY

ARIA vocabulary is not the only problem. There’s another part of puzzle, which is no accessibility API was ever designed to expose such unalike content to the assistive technologies.

All accessibility APIs take their roots a couple of decades ago, when inaccessible JavaScript controls was a unique challenge to solve. Over time HTML has matured and evolved into HTML5, acquiring a hefty amount of new features. It urged accessibility API vendors to adjust APIs to make the new features accessible. Also some efforts were made to make graphical content accessible, such as SVG and HTML canvas drawings. However most, if not all initiatives, were purely inspired by HTML-like use cases.

In case of non HTML content, for example, math, no breakthrough happened. It seems no API vendors made any decent moves to address the problem at large. Agreed, there were some notable exceptions like OS X API, which has got extra properties for math, but you know the exception just proves the rule.

You can say it happens for a reason, such as there’s no great interest in a such low volume content — indeed, it requires to make vicious effort to push things forward and there’s always a bigger fish to fry. But the fact remains: the web authors expectations weren’t met. In consequence web authors tried to reuse existing know-hows to expose math content, which was mostly about generating human readable annotations, something like, “a plus b divided by zero”. Some screen readers started to parse MathML on their own in order to extract semantics. Needless to say, it is rather hacky workarounds, than neat solutions.

Admittedly, there has been good progress in making web accessible over the last decade. However, overall tendency was about bringing existing expertise to new platforms and technologies. It allowed to keep traditionally accessible content accessible, but it mostly left unaddressed all other kinds of content, both long-time acquaintances and newbies, popped out several years ago and sprouted throughout the modern web.

ACCESSIBILITY CONCEPTS

All accessibility APIs are built around similar concepts if not to say the same concepts. These concepts serve to allow the authors to describe and identify blocks of content and connect the blocks to each other. The browser exposes these blocks to the assistive technologies via accessibility APIs, and the assistive technologies transform them into a format that the users may perceive, understand and operate with.

Here’s a short overview of accessibility concepts for the sake of providing context. You may scroll down to the end of this chapter, if you feel bored :)

Role is a cornerstone concept in semantic description of a block. It is a primary characteristic of a block and serves to identify block’s type. For example, this is a button or this is a paragraph. It also scopes set of possible properties and states and defines behavioral patterns. For example, this is a selected menu item or this is a text field that can be focused. Or this is a grid cell; it has row and column indices, which helps the user to navigate the grid. Also a block may have a human readable label — the first thing the user will hear, when the focus lands on the block.

Another key concept is relations, which describe connections between the blocks, i.e. how the blocks relate to each other. For example, most of the blocks are connected via parent-child relations like grid and its grid cells. Or an error message, shown if the user mistyped a password, is connected to a password field. Relations allow the user to navigate back and forth between blocks depending on content type and its state at user’s discretion.

Last but not least concept is actions. It is behavioral concept, used to describe interactions the user may take on a block. If this is a button, for example, then you can click it and so on.

A long story short, a block is described by role, states and attributes, which allows the user to perceive the block. A block’s actions allow the user to interact with content, and relations are for navigation.

Also, web content may change over time, which means the blocks can be created or removed or altered, as well as relations between them. All the changes are piped to the assistive technologies via eventing mechanism of accessibility APIs. This is not a 100% separate accessibility concept, but it is an integral part worth to mention for the sake of further conversation.

UNIVERSAL LANGUAGE

You may find it surprising, but these concepts are universal enough to describe almost any kind of content, you can see on the web today. Indeed, as long as you can describe the content by words, you can fit these verbal descriptions into accessibility concepts — all you need is to name objects, list their properties, define relations, and to explain how to interact.

Need an example? Sure. Say, you have a pie chart — this is something that falls out standard HTML, and thus you typically have to use bunch of tricks to make it accessible. However the concepts make a perfect match here. Indeed, a pie itself is a role, its title is a label. A pie chart consists of sectors, each of them also has a title, and all sectors are connected by relations, defining which one goes next. So the user can read a pie chart by navigating all sectors one by one.

Or let’s consider a video game, where a teddy bear teaches you some dancing. The bear shows dancing moves like heel turn, heel pull, dos-a-dos. Cameras keep track of the position of you and estimate how well you perform. So in terms of the concepts, the bear’s instructions are blocks of a limited life cycle, with labels, which are narrated to you. Limb positions are another set of blocks, which are characterized by a scalar value, ranged from 0 to 100%, and describing how close you were to a desired position. So, as the bear dances, you are told “heel turn, heel pull” and as you move, you hear how perfect your movement was. You may also add feedback factor here, which will serve to advise how your movements can improved, something like try leaning forward more next time etc.

You shouldn’t limit yourself by these examples. Thinking to start online music sheets store? You don’t want to leave anyone behind for sure. You will need to make music¸notes accessible, so assistive technology users can also enjoy it. Does your web app use chemistry ring formulas or math equations or anything else from any other subject you can think of? So, you can definitely describe all those symbols and formulas to your friend, so your friend can understand those, right? Thus, you should be able to express those in accessibility concepts as well. You may want to practice on your own to see how it goes.

SHAPE THE IDEA

Let’s come back to earth.

The accessibility concepts were invented years ago, and surprisingly or not, they do have a good fit to the modern web. However, web authors still struggle to make web apps accessible. How does it come, you wonder, what piece is missing? The answer is rather straightforward: web authors don’t have a rightful technology to describe web content, or in webish terms, there is no appropriate tag names and attributes or JavaScript hooks suitable to describe meaning of the content. So let’s say you have created a web app, which looks terrific and quite useful, but your creativity was not limited to standard HTML. Chances are the machine has no idea what the app is about, and thus cannot successfully transform it into a format the assistive technology can digest.

In such wise a web facing technology to describe content is a crucial part, but even if you had it, the irony is the accessibility APIs couldn’t handle it anyways, because they weren’t designed to do so.

Let’s summarize this two piece puzzle. The first piece is that the web author needs a tech vocabulary to describe content meaning, and the second one, platforms should have a delivery mechanism to convey these descriptions to the assistive technologies.

So how would a technology that cracks this puzzle, look like?

Let’s think aloud. Say, you have a cool web app or you are provider of some unique content, and for sure it is special to fit into standard HTML. Then, of course you are able to explain it to your friends, so they understand the whole thing — sure, how they could possibly use your app otherwise? Then, you put the wording into the accessibility concepts same as above, which also means breaking it down into blocks in a way they can be processed by software.

The whole point in blocks building is that web authors need deep control over them — and this is exactly what today’s web is missing. Certainly, if authors create content, then their expressive possibilities to explain the content should not be restricted by technologies.

Let’s imagine that web authors get a true flexible tool that allows them to juggle content blocks the way they want, when the author is in charge to define new roles, states, properties and to connect the blocks to each other by any relation you can imagine. Then, all of this is piped to the assistive technologies via platform APIs (yep, it’s probably the sole requirement to APIs to being able to transfer this kind of data). And voilà, it’s all we need!

The content provider (a web author) knows the meaning of the content. The assistive technologies know how to present the content to the user, and the browser knows how to deliver it to the assistive technology.

In this scenario the browser serves as a bridge connecting the web author and the assistive technologies. The browser doesn’t necessarily have to understand the content semantics that flows through it. The browser acts as an agent that makes the author and the assistive technology to communicate directly.

Sounds amazing? Yes! Sounds fanciful? No, but the browsers and APIs vendors should work hard to make it happen.

I believe the web needs a new flexible and extensible technology to give web authors control over content they create. And it will change the web for good!

Building web accessibility in 2019 was originally published in Sourcerer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Don’t you hate it when deploying your app takes ages? Over a gigabyte for a single container image isn’t really what is viewed as best practice. Pushing billions of bytes around every time you deploy a new version doesn’t sound quite right for me.

TL;DR

This article will show you a few simple steps of how you can optimize your Docker images, making them smaller, faster and better suited for production.

The goal is to show you the size and performance difference between using default Node.js images and their optimized counterparts. Here’s the agenda.

• Why Node.js?
• Using the default Node.js image
• Using the Node.js Alpine image
• Excluding development dependencies
• Using the base Alpine image
• Using the builder pattern

Let’s jump in.

Why Node.js?

Node.js is currently the most versatile and beginner friendly environment to get started on the back end, and I write it as my primary language, so you’ll have to put up with it. Sue me, right. 😙

As an interpreted language, JavaScript doesn’t have a compiled target, like Go for example. There’s not much you can do to strip the size of your Node.js images. Or is there?

I’m here to prove that to be wrong. Picking the right base image for the job, only installing production dependencies for your production image, and of course, using the builder pattern are all ways you can drastically cut down the weight of your images.

In the examples below, I used a simple Node.js API I wrote a while back.

Using the default Node.js image

Starting out, of course, I used the default Node.js image pulling it from the Docker hub. Oh, how clueless I was.

FROM node
WORKDIR /usr/src/app
COPY package.json package-lock.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD [ "node", "app.js" ]

Want to guess the size? My jaw dropped. 727MB for a simple API!?

Don’t do this, please. You don’t need to do this, honestly, just don’t.

Using the Node.js Alpine image

The easiest and quickest way to drastically cut down the image size is by choosing a much smaller base image. Alpine is a tiny Linux distro that does the job. Just by choosing the Alpine version of the Node.js will show a huge improvement.

FROM node:alpine # adding the alpine tag
WORKDIR /usr/src/app
COPY package.json package-lock.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD [ "node", "app.js" ]

A whole of six times smaller! Down to 123.1MB. That’s more like it.

Excluding development dependencies

Hmm… But there has to be something else we can do. Well, we are installing all dependencies, even though we only need production dependencies for the final image. How about we change that?

FROM node:alpine
WORKDIR /usr/src/app
COPY package.json package-lock.json ./
RUN npm install --production # Only install prod deps
COPY . .
EXPOSE 3000
CMD [ "node", "app.js" ]

There we go. We shaved another 30MB off! Down to 91.6MB now. We’re getting somewhere.

This had me quite proud of myself, and I was ready to call it a day. But then it hit me. What if I start with the raw Alpine image? Maybe it would be smaller if I grab the base Alpine image and install Node.js myself. I was right!

Using the base Alpine image

You’d think a move like this one would make little to no difference, but it shaved another 20MB off of the previous version.

FROM alpine # base alpine
WORKDIR /usr/src/app
RUN apk add --no-cache --update nodejs nodejs-npm # install Node.js and npm
COPY package.json package-lock.json ./
RUN npm install --production
COPY . .
EXPOSE 3000
CMD [ "node", "app.js" ]

Down to 70.4MB now. That’s a whopping 10 times smaller than where we started!

Not much more we can do now, right? Right…?

Using the builder pattern

Well, actually, there is. Let’s talk a bit about layers.

Every Docker image is built from layers. Each layer is a command in the Dockerfile. Here’s the file from above:

FROM alpine # base alpine
WORKDIR /usr/src/app
RUN apk add --no-cache --update nodejs nodejs-npm # install Node.js and npm
COPY package.json package-lock.json ./
RUN npm install --production
COPY . .
EXPOSE 3000
CMD [ "node", "app.js" ]

The FROM instruction creates a layer, so does the WORKDIR, as well as RUN, etc. All the layers are read-only, except for the last one, the CMD, which is a writable layer. Read-only layers can be shared between containers, meaning one image can be shared between containers.

What’s going on here is that Docker uses storage drivers to manage read-only layers and the writable container layer. This is the ephemeral layer that gets deleted once a container is deleted. Really cool stuff. But why is this important?

By minimizing the number of layers, we can have smaller images. This is where the builder pattern steps in.

FROM alpine AS builder
WORKDIR /usr/src/app
RUN apk add --no-cache --update nodejs nodejs-npm
COPY package.json package-lock.json ./
RUN npm install --production
​
#
​
FROM alpine
WORKDIR /usr/src/app
RUN apk add --no-cache --update nodejs
COPY --from=builder /usr/src/app/node_modules ./node_modules
COPY . .
EXPOSE 3000
CMD [ "node", "app.js" ]

We’re using the first image only to install the dependencies, then in our final image, we copy over all node_modules without building or installing anything. We can even skip installing npm in the final image as well!

Want to guess the final size? Go ahead!

I’d say we’ve done good, getting it down to 48.6MB, which is a 15x improvement, is something to be proud of.

The engineering team over at Sourcerer is using this approach as well. They’re leveraging the builder pattern and creating several steps which include running tests, routine automation and bundling the app core for production.

The whole CI/CD process is tied together with Jenkins to automatically deliver new versions to production and development environments from specific Git branches.

The verdict

Don’t be naive, there’s absolutely no reason to have gigabyte-sized images in production. A great first step is to use a tiny base image. Start small, baby steps are fine.

By choosing optimized base images will get you a long way. If you really need the boost in deployment speed and are plagued with slow CI/CD pipelines, check out the builder pattern. You won’t want to do it any other way in the future.

Note: I did leave out a sample where development dependencies are included for running tests before deploying to production, as it wasn’t relevant to the final size reduction for running in production. Of course, it’s a valid use-case! Feel free to add your ideas in the comments below. I’d love to hear what you think!

If you want to check out any of my previous DevOps related articles about Docker and Kubernetes, feel free to head over to my profile.

Adnan Rahić - Sourcerer Blog

Hope you guys and girls enjoyed reading this as much as I enjoyed writing it.
Do you think this tutorial will be of help to someone? Do not hesitate to share. If you liked it, smash the clap below so other people will see this here on Medium. Don’t forget to show us some love by following the Sourcerer blog!

A crash course on optimizing your Docker images for production was originally published in Sourcerer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Robert W. Oliver II

Pure Retro Joy

When I learned that Microsoft’s MS-DOS source code was one of the most popular repositories on GitHub, I was nothing short of ecstatic.

If you’ve followed any of my blog posts here, you’ll know I’m quite a fan of retro computing. I’ve covered various topics like MS-DOS video game development and put together a simple 16-bit operating system named Retrokern. When talking with other developers who are retro-enthusiasts, we often speak with passion in the challenge of creating fully-functional software and games in the most confining of spaces. It was this intense scarcity of hardware resources that encouraged ingenuity and spawned an entire golden era of PC computing.

In 2014, Microsoft released the source code to MS-DOS 1.25 and 2.0. These versions released in 1981 and 1983 respectively, were licensed to both IBM (branded as PC-DOS) and IBM-PC clone manufacturers and OEM (original equipment manufacturers). In the version 1.x and 2.x era, releases of DOS were tailored to the manufacturer due to dramatic variations in BIOS compatibility. Thanks to the IO.SYS (sometimes named IBMBIO.COM) hardware abstraction, Microsoft needed to only modify this relatively thin layer to allow their operating system to execute on a wide variety of 8086-based machines.

Exploring the Source

In our trip down 640k memory lane, we’ll be exploring the 2.0 version. Not only is it the most recent source listed, but it also bears a stronger resemblance to later DOS versions.

Before we get into specific segments, let’s discuss the programming language of MS-DOS 2.0 — x86 assembly. Nearly all systems level and game programming in the early DOS days was done in assembly. For an operating system, some assembly is required, especially in the boot loader. Since CPU cycles and bytes of memory were incredibly expensive in the 80’s, the entire MS-DOS kernel and associated utilities were written in assembly.

DOS Boot

When a BIOS-enabled PC (pretty much any PC made in the 80’s, 90’s, and 2000’s) boots, the BIOS runs various self-checks and sets up its own interrupt vectors (function call tables for software interrupts), then loads the first sector of the boot drive and transfers control to it. The operating system has a very small window of code available to load the rest of the system from disk and transfer execution to it. This is handled primarily in the SYSINIT.ASM file.

In the early days of DOS, this feat was not overly challenging, but as hardware and filesystems became more complex, it was clear the PC world needed a better boot loader. Thus, the UEFI (universal extensible firmware interface) standard was born, used by both Macs and PCs. Despite its support for huge hard drives, x86_64 processors, and hardware never dreamed of in the 1980’s, the UEFI system includes a legacy BIOS layer emulating the same bootstrapping behavior that an operating system like MS-DOS 2.0 would expect.

The MS-DOS Kernel

The hardware abstraction and input/output layer, called IO.SYS, pared with the MS-DOS kernel, MSDOS.SYS, make up the core of the system. Many of the files in the source tree combine to produce these two files. These files provide various routines that are available via interrupts, namely int 0x21 (or referred to as 21h, the “h” for hex).

This software interrupt allows DOS programs to allocate memory in a “safe” way (I use quotes because DOS memory allocation was far from perfect), access the filesystem, and display text on the screen. These function calls are far more portable than BIOS calls since BIOS specifications were rarely entirely compatible. DOS kernel functions provided a safe way to accomplish most of a program’s needs.

When speed was critical, however, programs would often ignore DOS and use the hardware directly. This was especially true in the case of text display. Command line utilities were fine with the 0x09 int 0x21 call to print text (terminated by a $ sign), but complex applications or games would often write to the screen directly at 0xB800.

The Command Interpreter

COMMAND.ASM provides the source code for the part of DOS most users interacted with — the command interpreter. It was often known by its binary name, COMMAND.COM. Once the kernel was loaded, the command interpreter would be started. The program was unique in that it had three unique parts: the init, transient, and resident sections.

The init portion of COMMAND.COM loaded the rest of the portions, processed the AUTOEXEC.BAT file (a script launched at each boot with startup commands), then transferred control to the transient portion.

The transient portion handled the user input loop. It would display the command prompt, wait for user input, then process those commands. The transient portion remained in memory during program execution but was considered volatile. Programs that used considerable amounts of memory would overwrite this section. It wasn’t needed during program execution, so this was perfectly acceptable.

The resident portion of was always present. It included the code necessary for starting and terminating programs as well as handling their exceptions (including the CTRL+C user-generated exception). It also included code that would reload the transient portion from the COMMAND.COM file.

In later DOS versions, the command interpreter could be substituted with another program. Popular alternatives were 4DOS, released by JP Software, and NDOS by Norton. These alternatives provided additional functionality and extra quality-of-life functions for the DOS user.

DOS Utilities

MS-DOS was more than just a boot loader and command interpreter — it shipped with various utilities for formatting disks, managing files, and debugging software. Let’s explore several interesting programs that were included in MS-DOS 2.0

COPY

Interesting enough, the copy utility, used for copying files from one location to another, wasn’t a separate program but rather part of the transient portion of the command interpreter. If a program wanted to copy a file, it either had to do it by itself or spawn another command interpreter and run its copy command for that purpose.

The source code for copy can be found in v2.0/source/COPY.ASM. This is included in the larger COMMAND.ASM file. As with most assembly code, the various routines inside the copy function are split apart by labels (specified by a name with a proceeding colon). These labels were translated into JMP (jump) addresses by the assembler.

An interesting artifact lies on line 120:

mov [MELCOPY],al ; Not a Mel Hallerman copy

Mel Hallerman was an IBM employee who is credited with writing some of the utilities included in MS-DOS 2.0. Unfortunately, I could find no code documentation or external reference source that provided a clue as to why the routine was given this name.

DIR

The file allocation table (FAT) was a directory that existed on the disk to point to the absolute locations of various files and subdirectories. Rather than browse this binary directory, the “dir” utility embedded in COMMAND.COM interprets this and displays it in an easy-to-digest fashion to the user.

The source code for this command can be found in DIR.ASM. Like COPY.ASM, it is compiled into the COMMAND.ASM file for inclusion into the transient portion of the command interpreter.

CHKDSK

MS-DOS 2.0 introduced directories. Files no longer had to reside in the root directory, and users could create a file and folder system to suit their needs. While this was a great achievement for Microsoft, this directory system is not perfect. When it failed, and it sometimes did during power outages and hard lockups, CHKDSK was usually the first rescue tool deployed.

The code for CHKDSK is, appropriately enough, found in CHKDSK.ASM. It’s an impressive piece of code that can rescue data in many cases of a corrupted FAT.

Oddly enough, until DOS 3.x, the DIR command didn’t show the user how much free disk space remained on the disk. The CHKDSK utility was the most common way to retrieve this information.

EDLIN

MS-DOS 5 and later included EDIT, a menu-driven text-based editor that was simple to use. But before this, MS-DOS users often relied on EDLIN.

Edlin was a line editor written by Tim Patterson for 86-DOS, the precursor to MS-DOS 1.0. I specify line editor because rather than accept free-form text like most other editors, the user could only input one line at a time. To display already written text, the user could use the “L” command, or prefix this with numbers to indicate the lines to display. Inserting text among what was already entered was a chore — users needed to relist the text and use the “I” command (preceded by the line number) to begin inserting text.

While Edlin wasn’t known for its intuitive interface, it was quite powerful in certain circumstances, providing a great search and replace feature and enabling users to delete multiple lines with just a few keystrokes.

DEBUG

Unless you purchased an assembler or compiler, writing software for DOS was difficult. Since no significant development utilities were included in the base package, amateur programmers were faced with a chicken-or-the-egg problem. Without a way to encode assembly commands into the binary code that the x86 processor uses, they were stuck with writing batch files in EDLIN.

I personally experienced this on my first DOS computer. I was a teenager with no job, so shelling out money for development software was not an option. Until I could acquire better tools, I often wrote assembly language programs in DEBUG.

The process of entering programs into DEBUG was tedious, but, true to its name, it allowed for instant debugging of said code. DEBUG was only able to write .COM files, meaning my programs couldn’t be larger than 64k, but that was more than enough to suit my programming needs until I later acquired more professional tools.

DEBUG was also excellent for disassembling and patching code. If the file was a .COM program, you could patch problematic code and write the file back to disk, providing a primitive yet effective hex editor.

The source code for DEBUG.COM can be found in DEBUG.ASM.

Further Exploration

The code is adequately documented, but for some commands, like FORMAT, a text file is provided with an in-depth discussion of the program’s operation and source code structure. I found these files quite useful in my exploration of this ancient treasure.

The Future of DOS

Believe it or not, new DOS software is still being written. And across the interwebs you’ll find reports of DOS computers still being used in various commercial applications. George R.R. Martin uses WordStar, a DOS-based word processor, for his Game of Thrones book series. YouTuber and Linux personality Brian Lunduke underwent a 30-day 1989 computing challenge and, last I heard, still uses a DOS spreadsheet program. A good friend of mine uses the distraction-free environment of DOS to get some serious work done.

I won’t pretend that DOS is the operating system of the future. That would be, at best, incredibly naïve. But the fact that this nearly forty-year-old operating system is still kicking in various incarnations, including the popular FreeDOS clone, is astounding.

I encourage you to marinate on this thought for a moment — we still boot computers and write software for an operating system written nearly four decades ago when Jimmy Carter was president.

Featured GitHub Repository — MS-DOS was originally published in Sourcerer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.