Published Notes

Publishing with Github Actions

2020-05-29T00:00:00+00:00

Continuous integration and deployment has now become a well established practice in software companies of all sizes. With the advent of platforms like Gitlab CI and more recently Github Actions setting up CI/CD even for small personal projects has become quick and easy.

“Actions” is the name of the platform now actively developed and promoted by Github. It’s lightweight and remarkably simple to work with, requiring no infrastructure of our own.

In this post we’re going to create a setup which allows us to publish a static blog using Python, Actions, an AWS S3 bucket, and finally serve the contents of this bucket with Cloudflare’s CDN.
This post assumes you’re already using a static site generator such as Pelican to build your website. There are plenty of tutorials floating around the Internet covering this topic so we’re going to skip that bit here.

The Actions job setup

The entire job description resides in a single YAML file. In our case this will live in our git repo under .github/workflows/python-app.yml. This might look intimidating at first but will become obvious once we dive in.

name: Publish blog

on:
  push:
    branches: [ master ]

jobs:
  build:
    runs-on: ubuntu-18.04
    steps:
      ... Continued ...

This starts with the git events (or hooks) which will trigger the job run. Another useful event, if you’re working within a team, is pull_request. Here we’re only dealing with pushing to the master branch.

The job is called simply “build” and specifies which Linux distribution to run on. Here I pinned Ubuntu to version 18.04 LTS but ubuntu-latest is a good default. Running jobs on Windows and Mac is also an option with Github, one which I haven’t tried yet.

steps:
- uses: actions/checkout@v2
- name: Setup Python 3.8
  uses: actions/setup-python@v2
  with:
    python-version: 3.8
- name: Install Dependencies with Pip
  run: |
    python -m pip install --upgrade pip
    if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
    pip install awscli
...

This clones the git repository into a fresh Ubuntu environment and sets up Python 3.8. All actions that look like actions/* are first-party scripts offered by Github to ease our work.

Next it installs our application’s dependencies with pip. Additionally it’s going to pull in awscli as well. We’re going to use it shortly.

And on to the final part of the spec file:

...
- name: Make Publish
  run: |
    make publish
- name: Sync S3
  env:
    AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
    AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    AWS_REGION: 'us-east-1'
    SOURCE_DIR: 'output'
  run: |
    aws s3 sync --acl public-read --delete $SOURCE_DIR s3://$AWS_S3_BUCKET

“Make Publish” is the default command used by Pelican to build the website. The result will be written to the output/ folder as a series of HTML files and other assets put together by the building process.

The S3 Sync is the interesting bit which synchronises the contents of our output folder to an AWS S3 bucket. A bunch of environment variables are defined here which are read from the repo’s secrets section. This can be found in your repository’s Settings → Secrets section.

It’s easy to configure and ensures we don’t leak any secrets into our code.

The run command does the synchronisation itself between the local output folder and the remote S3 bucket. It also makes sure that the files are set to be publicly readable and that any extraneous files found on the remote are deleted.

AWS has good documentation for all this stuff which I’d recommend looking over.

The complete Github workflow file can be found here — build-blog.yml.

The S3 bucket

I’m not going to cover the S3 bucket setup in detail, but what I would recommend is having a separate IAM Role with access restricted to buckets only. You’ll then have to feed the access-key-id and the secret-access-key obtained during the creation of this role into the Github secrets page.

The bucket also needs to be configured to serve a static website, this can be easily done with the AWS CLI tools:

aws s3 website s3://my-bucket/ --index-document index.html \
    --error-document 404.html

It only has to be done once so it can be left out of the job spec.
You will need to know the public URL of your bucket for the next step. It will look similar to:

http://bucket-name.s3-website-region.amazonaws.com/

The bucket needs to have the same name as your domain, including the TLD, e.g. wooptoo.com

Cloudflare

Cloudflare offers a robust CDN and DNS service which will serve these files to a large audience with ease. The service has a free tier for personal use — which is more than enough for our application. This post assumes your site’s DNS is with Cloudflare.

If you’re hosting the website on the root domain like I do then you’ll need a DNS record pointing from that root domain to the bucket’s public URL. In Cloudflare’s web UI this can easily be done with a CNAME record.

For this to work Cloudflare employs a trick. It knows that this is a special record and it applies CNAME flattening — which basically means resolving the bucket’s A/AAAA records first and then pointing your own domain’s records to the resulting IP. It might sound complicated but it works very well in practice. Other DNS services such as AWS Route53 behave similarly with Aliases. This can then be easily tested from the command line:

host -t A -v wooptoo.com

;; ANSWER SECTION:
wooptoo.com.        96  IN  A   172.67.133.56
wooptoo.com.        96  IN  A   104.21.5.86

Note that those IP addresses actually belong to Cloudflare. I assume CF proxies the requests through its own servers in order to provide caching and DoS protection for the underlying resource.

Wrap-up

Every push to the repository master branch will now trigger a build with Github Actions. The output of that build is the static website which will sync to an S3 bucket and then get served with Cloudflare’s CDN.

This is an easy and flexible setup which allows writing blog posts even from your mobile phone, pushing to the repository and letting Github handle the rest. All this without having to manage server instances yourself and the associated trouble.

Update

Since posting this, the blog has been moved to Cloudflare Pages which simplifies deployment significantly for static websites. Give it a try.

Docker multistage builds

2020-02-25T00:00:00+00:00

The multistage pattern splits the Docker image building process into two or more individual steps. These are described in a single Dockerfile and will run sequentially. The main advantage is a smaller final output image and a clear separation between concerns.

Here’s a minimal example to help us get started:

FROM python:3.8-alpine AS builder
RUN apk update && apk add --upgrade alpine-sdk
COPY . /app
WORKDIR /app
RUN pip install -U pip wheel setuptools
RUN pip wheel . -w /wheels/

FROM python:3.8-alpine AS runner
COPY --from=builder /wheels /wheels
RUN pip install /wheels/*
COPY . /app
WORKDIR /app
CMD python runapp.py

The first stage, the builder, pulls a meta-package which in turn installs quite a lot of system dependencies needed for compilation. The Python wheels are then built over the next steps, including our own application.

In my example I’ve used the same base Docker image for both, but this stage could be built from a completely different base image than the one we use next in the runner. In general you might not want to combine Alpine environments with Debian ones for example.
They use different C standard libraries and there’s a good chance that compiled packages will not function correctly across them.

The runner builds our final image. The process is lightweight and simply installs the previously built packages and boots the app. The trick here is the COPY directive which will allow us to fetch content from the previous stage into the new clean environment. This can potentially mean significant image size savings and a simplified runner image.

The builder stage is then discarded and only the runner is presented as the final product without the extra baggage associated with compiling.

Additional saving in build time can be achieved if for example we push the built wheels to a private PyPI repository, or even cache the whole Docker image whenever possible.

Naming each stage using the AS directive makes this process nice and tidy, even if the naming of terminal images is not really required.

Docker docs multi-stage builds

Quick IPTables filtering

2019-03-03T00:00:00+00:00

The Linux kernel has a great firewall built-in commonly referred to as iptables. Actually iptables is only the user-space tool that you as the user interact with, while the kernel module doing all the work is actually called Netfilter.

Here are a few tips on how to quickly set up a filtering firewall on a Linux box.

The filtering mechanism uses a series of groups called “chains” in the literature. The main predefined chains are:

INPUT — All the packets reaching the host computer
OUTPUT — Packets originating from the host computer
FORWARD — Packets routed / passing through.

Each of these chains can have a default policy such as accepting all packets by default or dropping them.

Individual rules are then added to a List-like structure within each of these chains. Rules are then read from the top to the bottom for each intercepted IP packet.

Let’s go on and add a few rules. These commands are all run as the superuser.

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT

The iptables command can append (-A), insert (-I), delete (-D) rules from the list. In this case we append to the INPUT chain.

The first one will allow all requests to the lo interface: a shorthand for the localhost.
The -j flag indicates which action should be applied when the rule is matched, such as ACCEPT, DROP, etc.

The second rule will allow all inbound TCP packets on ports 80 and 443. IPTables uses the concept of extensions. These are loaded using the -m flag.
For the second rule we use the multiport extension which allows to specify two ports at once.

You probably also want to allow UDP traffic on the same ports to account for newer protocols such as QUIC.

We don’t necessarily have to use extensions, a simple rule will also work such as this one, where all inbound traffic on TCP port 22 is accepted.

iptables -A INPUT -p tcp --dport 22 -j ACCEPT

However a cautious user will move their SSH daemon from port 22 to something a bit more obscure, and turn on public key based authentication.

All rules can then be visualised using a command such as:

iptables -nL --line-numbers

This will display each rule preceded by its line number. This is significant as we’ll see in the following example.

Due to how some protocols work we will also have to allow inbound traffic in reply to outbound traffic that was initiated from the host machine.

iptables -I INPUT 2 -m state --state ESTABLISHED -j ACCEPT

We want this rule to be inserted near the top of the INPUT chain on the 2^nd position in the list, because it will be used quite a lot. We then use the state extension here, which enables us to specify that for already established connections the firewall should allow all inbound traffic. This rule is important as without it many protocols will break (such as outbound HTTP from the host machine).

If we now list the chain rules again we can see that this was added to the 2^nd position:

Chain INPUT (policy ACCEPT)
num  target     prot opt source        destination
1    ACCEPT     all  --  0.0.0.0/0     0.0.0.0/0 
2    ACCEPT     all  --  0.0.0.0/0     0.0.0.0/0     state ESTABLISHED
3    ACCEPT     tcp  --  0.0.0.0/0     0.0.0.0/0     multiport dports 80,443
4    ACCEPT     udp  --  0.0.0.0/0     0.0.0.0/0     multiport dports 80,443
5    ACCEPT     tcp  --  0.0.0.0/0     0.0.0.0/0     tcp dpt:22
...

Don’t worry if you make a mistake, rules can always be deleted by their index in the chain.

iptables -D INPUT 6

Or even flush the entire chain with:

iptables -F INPUT

Finally we could drop all the remaining traffic with something like:

iptables -A INPUT -j DROP

But then the drop rule will always have to stay at the bottom of the chain. A better practice is to change the Policy of the entire chain to DROP:

iptables -P INPUT DROP

This will then be the default catch-all action for each packet which doesn’t match one of the rules we defined.

Chain INPUT (policy DROP)
...

As for the OUTPUT chain, this usually has a default ACCEPT policy, as we generally trust all outbound traffic from our machine.

iptables -P OUTPUT ACCEPT

Finally persist the state of the chains to the disk, and make sure that the iptables service is enabled by your daemon manager — such as systemd.

iptables-save > /etc/iptables/iptables.rules
systemctl enable iptables

This will ensure that the rules are actually kept and loaded between reboots. And we are done, you have now configured a few simple filtering rules for your Linux firewall. Below are a few good resources to further your knowledge with iptables.

CentOS IPTables Howto
IPTables Basic Concepts
Google’s QUIC protocol: moving the web from TCP to UDP
man iptables
man iptables-extensions

Modding the Inspiron 7570

2018-03-26T00:00:00+00:00

Not too long ago I purchased the Inspiron 15 7570 laptop from Dell’s outlet website. Having owned the amazing Inspiron 7537 for the past four years I had very high expectations from this machine.

While it’s a really good product in many ways with great specs, I was bitterly disappointed by its remarkably poor screen. The unpleasant blue tint and the low colour gamut are a complete eyesore, enough to put off even casual users. The colour mangling and the banding that can be seen in gradients make this completely unusable for any sort of design.
I’m not sure how Dell even decided on this panel in the first place.

The intrigue

I initially wanted to send it back due to this defect but decided to keep the machine at the last minute.

After a bit of googling around I figured I could replace the LCD panel myself with a much nicer one. The panel that the laptop uses is a NV156FHM-N35 from the Chinese manufacturer BOE.
Now BOE is a serious competitor to Samsung and LG, but all companies have their fair share of lower-end products. I’m convinced this is the absolute cheapest panel Dell could possibly source.

The panel I decided to replace it with is from the same manufacturer but with much better specs: The NV156FHM-N61 – a High Colour Gamut screen with 16M colours, much higher Luminance and Contrast. The panel’s connector, voltage, and physical size are identical to the old one. It’s essential to look into these details before ordering a replacement.

The shop

I managed to find this screen on AliExpress from a Chinese seller. While this might not inspire trust to most people, I decided to take the chance. The panel itself was about £40 plus another £12 for transport – so not a tremendous risk. The seller assured me that they’re sending me the exact model I request and not some replacement that they happen to have in stock. The panel arrived well packaged and in perfect condition in about two weeks time (which is fast for China).

The teardown

The teardown is the hardest part of course and one has to be extra careful.
I used a few tools in order to achieve this without irreparably destroying my laptop:

two spudgers for prying (must be soft plastic! never use a screwdriver)
a guitar pick
a Philips screwdriver
Replacement double-sided 3M tape. I used a 5mm wide one, but you can also use 4mm.

Dell are open about their hardware and have published a very comprehensive service manual for this laptop. This manual is freely available to download from here. Please read this manual together with this post as it has some very good advice. This manual does cover the first part of this teardown in great detail. Unfortunately it does not cover the screen assembly teardown.

You first want to untighten the screws on the backplate. Only the bottom four will come out entirely, the others have stoppers on the other side.

Using the spudger stick carefully begin prying the laptop open. You want to start from the back of the screen where the vents are and slowly work your way around the edges.

The first time you open it it will be a bit harder. The plastic is tightly fit together and will not give so easily, but with enough care you should be able to get it done. Go slowly and never force it open. You should be able to pop the entire bottom plate if you pry around the edges.

This should be revealed before you if you manage to open the laptop. All its internals are within reach which makes this machine very easy to service.

Screen teardown

What we’re interested in next is the WiFi antenna connector in the upper left hand corner (left of the fan), and the eDP connector of the screen in the upper right hand corner (above DIMM1).

You need to remove the screw and little plate which holds the WiFi card in place, so that the antenna can be removed. You should screw the card back in or remove it completely after this process, so it doesn’t dangle about.

The eDP connector should be removed with extra care as to not damage it. You’re going to be out of luck if you damage this cable as the laptop is new enough that spare parts cannot be found online yet.

Carefully remove the stickers which hold the eDP connector in place and remove the metal latch of the connector. Use the pointy end of the spudger tool (the one that looks like a pencil) on the latch rather than your fingers.

Now remove the remove the two plastic hinge covers with the pointy end of the spudger. These should come off easy. Take note of how the antenna and panel connectors are routed through the hinges.

Place the laptop with the screen open facing down on a soft surface. We’re going to unscrew the hinges next and we want them in an “open” position when the screen is removed.

The hinges are the two solid metal plates you see at the top of the laptop. Untighten the four M2 screws which hold the hinge on the left hand side and the three on the right hand side.

Around the edges

At this point the screen assembly should be completely detached from the rest of the laptop body. Put aside the laptop with its back cover and let’s get working on the screen itself. Take note that advancing beyond this point might void your warranty with Dell.

Using the same technique that you learned in the first part of this tutorial, pry around the edges of the soft plastic frame of the screen. Only this time you will need the guitar pick. The soft plastic is forgiving but again, never force it open.

I recommend starting at the top left hand side and once you get a bit opened with the pick continue with the spudger. At some point you’ll be able to open it using your hands. Go slowly.

One you get to the bottom bit take extra care as this might take a bit longer to come off. I’ve used the spudger extensively here to pry the edges. Never shove the tool inside as this might damage the screen circuit at the bottom of the panel. Take extra care with the webcam bit at the top as well. Even if you swap the panel out, it’s still a good idea to keep the old one around so you want it in a good and reusable condition.

This frame also has some double sided sticky tape which might get damaged in the process of removing it. We’ll replace this tape in the following few steps.

The switcheroo

Congrats, you’ve got to the meat of this tutorial. You should now have the LCD in front of you ready to be changed. The panel is stuck to the back cover of the laptop using two 1cm wide double sided pieces of tape, running from the top of the panel to the bottom. We’ll need to separate the panel from this cover. We’ll also try not to damage this tape as it is essential to holding the LCD into place – and can be easily reused if we don’t stick our oily fingers to it.

Using the spudger (or even two of them) slowly begin to separate the panel from the back cover. Again, do not force it apart. Take your time. Do this on both sides. I would recommend not pulling from the tabs you see at the bottom.

If this is successful you should now have separated the panel. You need to unplug the eDP plug from its back. This also has a metal latch which can be removed using the spudger’s tip. Again, take extra care when handling this cable. Part of it might be stuck to the back of your panel.

Move the new panel in, connect the eDP connector and its latch, and gently press it against the double sided sticky tape from the back once it’s in place. It should easily fit, if it doesn’t check around the edges whether a piece of the WiFi antenna has been caught behind the panel.

There should be some black double-sided tape inside of the frame, about 3mm—4mm wide. If this has been damaged or has too many finger prints you’ll need to change it with a fresh one. Remove the existing tape from the frame and apply a new one. Use tweezers in order to not leave prints on it. Once this is done around all edges press the frame back into the screen cover and gently press on the edges so that the tape sticks to the screen.

The final touch

From this point on you can proceed in doing the steps in reverse order until you get back to a fully assembled laptop. Take extra care when putting the screen assembly back together and plugging the WiFi and panel connectors back in. Remember how the cables were routed.

If you got everything right you should have a fully assembled machine at the end of this, now with a much better screen! The NV156FHM-N61 has great colour reproduction with 100% sRGB coverage and much warmer colours. The new panel also retains its colour accuracy much better than the original when the backlight is dimmed.

All in all this makes this machine much more enjoyable to use, with colours surpassing even my old Inspiron 7537.

Mirroring Git repos

2018-01-10T00:00:00+00:00

Here is a quick way to mirror a git repo from one remote to another. This is not only useful for mirroring but also for backups.

The git clone --mirror git@gitlab.com/myuser/myrepo.git command will create a mirrored repo, which is pretty much like a bare repo, but it also maps all local branches to the remote branches and gets all the refs (like tags). Basically a one to one copy of the remote repo.

Then it can be pushed to the new remote with git push --mirror git@mydomain.com/myuser/myrepo.git. If you ever need to update the local clone with commits from the old remote (or the new one) it can be done with git fetch --all.

And done, a nice easy way to move entire repos.

Introduction to Python pathlib

2017-11-05T00:00:00+00:00

One of the first things many beginner Python tutorials teach is how to read or write files using patterns like:

with open('file.txt') as f:
    f.read()

And then walk directories with libraries such as os:

import os
g = os.walk('.')
next(g)

Which will go recursively through results depth first, leaving to you, the programmer, the joy of writing your own algorithm for searching files or listing files of a certain type.

Other calls such as os.rename or to os.path.join have the developer
manipulate paths manually with string concatenation and os.sep. Not much fun.

Neither of these approaches is wrong. But they’re cumbersome and code can get thorny quickly.

Today we’re going to take a look at pathlib which was introduced in Python 3.4 and simplifies our work with files tremendously. We’ll showcase pathlib’s most popular features.
Let’s start with the basics: Reading and writing text files.

We’re going to use read_text and write_text on our Path instance. Let’s throw in some yaml.

from pathlib import Path
import yaml

path = Path('tmp/file.yaml')
contents = path.read_text()
yaml_content = yaml.load(contents)

# yaml_content
# {'name': 'Radu', 'uses': 'linux', 'twitter': 'wooptoo'}

yaml_content['name'] = 'Rad'
path.write_text(yaml.dump(yaml_content))

What if we want to rename our file? We’ll reuse the path from the previous example.

path = Path('tmp/file.yaml')
new_path = path.with_name('myfile.yaml')

# PosixPath('tmp/myfile.yaml')

path.rename(new_path)
path.exists()
# False

new_path.exists()
# True

Our file was renamed in a few easy steps, in a nice object oriented fashion.
path.with_suffix('.txt') is very similar, and will just change our file’s extension and keep the initial file name.

PosixPath represents our path instance, which enables us to do all sorts of operations, like creating a new directory, checking for existence, checking for file type, getting size, checking for user, group, permissions, etc. Basically everything we would previously do with the os.path module.

Let’s fetch all our yaml files now with glob:

from pathlib import Path
tmp = Path('tmp')
g = tmp.glob('*.yaml')
list(g)

# [PosixPath('tmp/myfile4.yaml'),
# PosixPath('tmp/myfile3.yaml'),
# PosixPath('tmp/myfile2.yaml'),
# PosixPath('tmp/myfile.yaml')]

And we can take it from there. glob and rglob are super handy for this kind of stuff. Of course Python had the glob module before, but having it under pathlib’s umbrella is extremely handy.

Next we can iterate through sub-directories using rglob with patterns like ./ Or we can just use iterdir which is nicer.

tmp = Path('tmp')
dgen = tmp.iterdir()
list(dgen)

# [PosixPath('tmp/snappysnaps'),
# PosixPath('tmp/cameraphotos'),
# PosixPath('tmp/yamlfiles')]

And one last thing which I’m going to touch upon is traversing folders, and even creating new paths.

tmp = Path('tmp')
new_path = tmp / 'pictures' / 'camera'

# PosixPath('tmp/pictures/camera')

new_path.mkdir(parents=True)
new_path.exists()

# True

This wizardry is done by overloading the / operator of the Path instance. In Python 3 this is the private __truediv__ method.

p = Path('tmp')
p / 'myfile.txt' == p.__truediv__('myfile.txt')

# True

In a nutshell these are the most common cases which you’ll use on a daily basis if you work with files a lot.
We’ve managed to replace three different libraries (os, glob and open/read) with pathlib which gives us a neat developer interface.

Since pathlib is now part of the Python standard library there’s absolutely no reason to not use it for new projects. More on the library can be read in the manual page.

Scraping with BeautifulSoup

2017-03-09T00:00:00+00:00

BeautifulSoup is a handy library for web scraping that’s mature, easy to use and feature complete. It can be regarded as jQuery’s equivalent in the Python world. In this post we’re going to scrape the front page of wooptoo.com and output a clean, JSON version of it. The library can also handle DOM manipulation, i.e. adding elements to the HTML document, but that’s beyond the scope of this article.

from bs4 import BeautifulSoup
import requests

resp = requests.get('http://wooptoo.com/')
page = BeautifulSoup(resp.content)

The bread and butter of scraping with BS is the find_all method. select works similarly but it uses the CSS selector syntax à la jQuery. Their output will be identical in this case.

posts = page.find_all(attrs={'class':'post'})
_posts = page.select('.post')

posts[0] is _posts[0]
> True

One catch to be aware of is that BS will work with special bs4 data structures, which inherit the built-in Python structures. So a list of posts will yield a bs4.element.ResultSet and each individual entry will be a bs4.element.Tag.

The find_all method allows us to also select html elements using native regular expressions. This enables us to fetch all the posts from 2014 for example:

import re

d2014 = page.find_all('time', {'datetime': re.compile('^2014')})
[p.parent.parent for p in d2014]

We can select the child of an element either using chained calls to find or using the select_one method. Both will only fetch the first match.

titles = [p.find(class_='post-title').find('a').text for p in posts]
titles = [p.select_one('.post-title a').text for p in posts]

Putting it all together:

import itertools
import json

_titles = [p.select_one('.post-title a') for p in posts]
titles = [t.text for t in _titles]
urls = [u.get('href') for u in _titles]
datetimes = [p.find('time').get('datetime') for p in posts]
tags = [t.select_one('.meta-tags a').text for t in posts]
summaries = [s.find(class_='post-summary').text.strip() for s in posts]

_posts = zip(titles, urls, datetimes, tags, summaries, itertools.count(1))
_keys = ('title', 'url', 'datetime', 'tags', 'summary', 'number')
output = [dict(zip(_keys, p)) for p in _posts]

json.dumps(output)

The _posts will yield a generator object which can be iterated over only once, as opposed to the _titles list which does not have the same drawback.

The BeautifulSoup library is much more complex than the example provided here. It allows for things like walking the DOM tree in a Javascript-esque manner: page.body.footer.p, fetching children nodes, parents, siblings on the same level, and much more.

Marshmallow schema basics

2017-02-19T00:00:00+00:00

In this post we’re going to cover the basics of working with Marshmallow. Marshmallow is a Python library which enables us to easily sanitize and validate content according to a schema.
Schemas are useful when we want to sift through user provided data en-masse as opposed to dealing with each item individually.

Please note that this article refers to the version 2.x of the library.

from marshmallow import Schema, fields

class _Schema(Schema):
    class Meta:
        dateformat = "%Y-%m-%d"

Here we extend the default Schema class with our own which will output a custom date format. The Meta subclass supports plenty of methods to customize our schema, such as excluding some of the fields or explictly including them. More can be read in the manual: help(marshmallow.Schema.Meta).

class Slug(fields.Field):
    def _serialize(self, value, attr, obj):
        if value:
            return str(value).lower().replace(' ', '-')

Above we define a custom field which will serialize its value using the provided function. And we use this field below in our crude blog schema.

class CommentSchema(_Schema):
    author = fields.Str()
    body = fields.Str()

class PostSchema(_Schema):
    slug = Slug(required=True)
    date = fields.DateTime(required=True)
    title = fields.Str(required=True)
    body = fields.Str(required=True)
    tags = fields.List(fields.Str())
    comments = fields.List(fields.Nested(CommentSchema()))

The last two are compound fields, with comments being more interesting as it is a list of comment schemas. When referring to another schema instead of a field type we need to use the Nested method.

And in a nutshell there we have it. Now all that’s left is to use it to sanitize our blog post through the schema:

schema = PostSchema()
post = schema.dump(raw_post).data

Validation was left out from here and we will cover it in another blog post.

A new AngularJS project seed

2016-07-22T00:00:00+00:00

A few months ago I started working on a project seed for AngularJS 1.x.
A project seed is the initial skeleton a developer would use to get things up and running quickly when commencing a new project. There are plenty of these around with the most popular being Yeoman with the Angular generator.

So why another one you would ask? Simply because I preferred to use my own conventions and my preferred tools. And also because of a bit of NIH syndrome too.

Some of the my first requirements were using Gulp as a task runner, just because it’s so much easier to write than Grunt. Next up was using Bower for front-end asset management and Sass for styles. Bower is a proven package manager that is easy to use and Sass needs no introduction.

Down the path it became obvious that I was going to need a few optimizations in place in order to build great performing Angular apps.

templateCache is a great improvement over having every single view sent via XHR on demand. It basically concatenates all of the views and loads them all at once when the app starts.

ng-annotate does the automatic annotation of the app source (for Dependency Injection), which then can be minified with mangling of the symbols. This greatly reduces file size.

localStorage.debug is a special flag which is set in the browser by the developer. This is read when the app starts and turns on $log.debug statements and Angular debug classes. These classes are used by tools such as ng-inspector, but otherwise they would unnecesarily pollute the DOM of the app.

Other things which I managed to squeeze in are: hinting of JS files with a friendly output, livereload support, configuring the Gulp task runner via gulpconfig.json. This enables things such as turning the integrated http server on or off.

The project can be found on Github and your contribution is more than welcome. This is pretty much still a work in progress and support for things like ES6 will be included in the future.

What I do not plan on including for now is a full blown generator. My aim is to keep the code simple and lean.

Angular-seed

Keeping home configuration in git

2016-03-06T00:00:00+00:00

Having dotfiles from the $HOME directory in git is a great way to keep track of what’s going on with some of our most important config files.

However just creating a plain git repo won’t play nice with other repos found further down in the home tree. Chances are that at some point we’ll accidentally issue a git command on the home repo instead of the one we actually want to work on.

So why not make our own git command which handles the home repo exclusively. We’ll name this new command cfg an create an alias in ~/.bashrc for it.
Since we want git to use a special .cfg directory instead of the default one and still have the working tree set to $HOME the alias will look like this:

alias cfg='git --git-dir=$HOME/.cfg --work-tree=$HOME'

Now we can issue a git init command in the home dir. If you already have a .git directory you can just rename it to .cfg.
One tweak is configuring cfg not to show untracked files.

cfg config status.showUntrackedFiles no

If we want to be absolutely sure that we won’t mistakenly add files (like the Pictures directory), then we need to create a .gitignore file, either with each ignored directory on a new line, or with the wildcard symbol:

Now when adding files we have to issue a cfg -f add command.

And we’re done!

Building apps with Gulp

2014-04-16T00:00:00+00:00

I never quite liked JavaScript building tools. Grunt always seemed way too complicated for the simple things I wanted to accomplish. Like concatenating files and minifying them. One of the first bash commands that any Linux user learns about is cat.

You can just cat a few files together and pipe them to a minifying script and be done. 5 lines of Bash maximum. Want to also include the build date in your JavaScript file as a header? Just run:

echo /* `date -R` */ > build.js

I think Grunt’s highly declarative syntax was the real barrier here, since it prevented you from doing quick adjustments to the building pipeline.

This all changed with Gulp. Gulp promotes and imperative approach to building apps and it tries to stay simple and focused on its task.

Its syntax feels a lot like the Linux CLI. Commands are pipelined together in a very simple manner:

gulp.task('scripts', function() {
  return gulp.src(['js/*.js'])
    .pipe(uglify())
    .pipe(concat('build.min.js'))
    .pipe(gulp.dest('build/'));
});

Basically it reads files from an input directory, minifies, concatenates and puts the resulting file in the build/ directory.

Tasks can be created for any kind of task: linting the code, compiling lesscss files, watching files and recompiling them on change, etc.

The API

The Gulp API is very small, it only has a few commands. Most of which we already encountered in the code snippet above.

gulp.task

Declares a new Gulp task. You can also declare tasks which simply run a series of other tasks (dependencies).

gulp.watch

Watches files for changes. For example lesscss files.

gulp.src

Returns a readable NodeJS stream which can be piped inside the task. An array of files can also be passed to the src function. This is actually the most common use case. A feature of src is that it will read the provided files in order. This is significant for plugins such as less.

gulp.src.pipe

Pipes commands to the NodeJS stream. These commands can be gulp plugins such as: uglify, concat, less, etc. They can also be user defined functions. Many commands can take options (parameters).

gulp.dest

Writes the result to the destination. This does not happen immediately, but upon the completion of other piped commands.

Gulp makes extensive use of NodeJS streams. Files are read from the filesystem, entirely manipulated in memory, then written back to the filesystem. This might not seem like much, but Grunt used to create many small files for every intermediary result.
To achieve this Gulp uses vinyl, which creates virtual file objects.

The example gulpfile.js used in this article can be found in this git repo with plenty of comments.

Serving WebP today

2014-04-05T00:00:00+00:00

WebP is both a lossless image format, like PNG, which preserves every detail of the original image, and a lossy format, like JPG, which will compress the original image down to a certain quality.

WebP support in browsers such as Firefox and IE is virtually nonexistent. However if a big chunk of your audience browses from Chrome, Android Browser or Opera, you could serve WebP today and reap the benefits.

The benchmark

So how much better is WebP actually? Here’s a quick test to find out.

cwebp from the official WebP distribution was used to convert images. The files used by this benchmark can be found in this git repo.

The source is a lossless 24-bit PNG, optimized with optipng.
Lossless conversion from PNG to WEBP:

cwebp -lossless source.png -o lossless.webp

PNG: 532 KB
WEBP: 386 KB
Webp is 28% smaller.

For comparison I used a JPG image saved at 95% quality.
Lossy conversion to WEBP:

cwebp -q 95 source.png -o lossy.webp

JPG is 243 KB.
WEBP is 160 KB.
Webp is 35% smaller.

Bandwidth savings offered by Google’s new image format can be significant enough to be worth the trouble.

Serving WebP

Every browser (user agent) will send an Accept header with each HTTP request. Serving WebP, or any other content actually, can be done in a browser neutral manner by looking at this header and responding with a Content-Type known to the user agent. This is the basic principle of content negotiation.

Chrome advertises its WebP support in a few ways:

For HTML files:

Accept: text/html,…,application/xml;q=0.9,image/webp,*/*;q=0.8

For image files:

Accept: image/webp,*/*;q=0.8

So all we have to do is to match against the image/webp part in our web server config and return the appropriate file.

There’s another small trick which we will employ in this example. Every image linked from a HTML document will be linked without a file extension.

That can’t be right, can it?

It is actually right from a REST point of view. The image is a resource and its content type should not be included in the resource path. Content negotiation should be reserved for the Accept header.

Setting up nginx

A very basic nginx configuration would be:

location /webp/img {

    add_header Vary Accept;

    if ($http_accept ~* "image\/webp") {
        rewrite (.*) $1.webp break;
    }

    try_files $uri.png $uri.jpg
              =404;
}

This assumes that the location will only serve images. The Vary header is used to instruct proxies that the content type can differ between server responses. The try_files directive is used in order to fallback to PNG or JPG images.
This setup does not cover cases where nginx is used as a proxy. You can read more on that in this article.

Ok, but how about saving images? Chrome does another smart thing. If you try to save a WebP image, it will give you the JPG or the PNG version instead, so that it can be viewed on any device.

More on WebP and content negotiation can be found below.

Prototype inheritance in JavaScript

2014-03-24T00:00:00+00:00

JavaScript has its own inheritance model that is quite unique among languages.
Since JavaScript has no notion of classes yet and since everything is actually an object, we will try to clear the confusion by defining a few terms:

A class is a constructor function which has properties and other functions defined on it.
An instance is an object created with new Class.

Instead of inheriting from a base class, instances in JavaScript inherit from a prototype object. The prototype is then referenced by every new instance of the object.
An instance can then have its own properties and methods beyond those inherited from the prototype.
JavaScript uses prototype chains: object instance C can have as prototype object instance B which in turn can have as prototype object instance A.
A property lookup traverses the prototype chain upwards.

Uses

So what are prototypes useful for?

Subclassing

The classical inheritance model found in other languages can deal with subclassing very easily. However, we can achieve subclassing in JavaScript using the prototype.

Suppose we have a parent class with a few methods declared on its prototype.

function ParentClass() {}

ParentClass.prototype = {
    firstMethod: ...
    secondMethod: ... 
};

We can then create a child class which has as its prototype set to an instance of the parent class.

function ChildClass() {}

ChildClass.prototype = new ParentClass();

If we create a new instance of each of these two classes we will notice that the child instance inherits from the parent class.

parentInstance = new ParentClass();
childInstance = new ChildClass();

> childInstance instanceof ParentClass
true

We achieved this by chaining the two classes’ prototypes. A simplified schema of the inheritance would be:

child class → child.prototype → parent class → parent.prototype

Typed Errors

Using the same pattern we can create typed errors. Errors are JavaScript’s equivalent to Exceptions found in other languages. First we need to define a constructor function.

function MyError(message) {
    this.name = 'MyError';
    this.message = message;
}

Then assign an Error instance to its prototype.

MyError.prototype = new Error();

And finally we can use it.

throw new MyError('something happened');

A nice feature of typed errors is that we can handle each error type using instanceof. A lot like the exception handling from other languages.

Only handle MyErrors and rethrow the others.

try {
    throw new MyError('something happened');
} catch (e) {
    if(e instanceof MyError) {
        ...
    } else {
        throw e;
    }
}

Promises in AngularJS

2014-02-10T00:00:00+00:00

Promises are quite an old concept that took off only recently in the JavaScript world. Since version 1.2 promises have become such an important part of AngularJS that they cannot be overlooked anymore. In this post we will cover the basic concepts of promises in AngularJS with a few use cases at the end.

So what are promises?

The simplest definition of promises is given by the Promises/A+ specification:

A promise represents the eventual result of an asynchronous operation.

The two main functions performed by promises are:

provide an interface for handling incomplete (asynchronous) operations
error handling for the said operations

With a few nice extras derived from the above:

chain asynchronous operations while avoiding callback hell
a syntax that loosely resembles the try-catch-finally construct

Generic vs. AngularJS implementation

Promise instances can either be constructed by specialized factories (the case with Angular), or by static methods on the Promise object itself.

The MDN article on this topic defines a Promise as an object that has a few static methods such as resolve and reject. This object is used to construct a new Promise instance. Note that a promise will either: resolve with a value or reject with a reason. Resolve or reject can be called at any later time, a feature which gives promises their asynchronous nature.

In Angular promises are created with the Deferred API. The $q.defer() method will construct a new Deferred instance that can either be resolved or rejected. Note that Angular also has a notify method that provides updates on the execution status.
The Promise instance can then be obtained from the promise property found on the Deferred instance.

Either approach works, but Angular might change its behavior in the future to reflect the (eventually) upcoming ECMAScript 6 specification.

After construction, the Promise instance will expose a few methods such as then, catch and finally.

then accepts a success and an error callback function. Angular also has a notify callback.
catch only accepts an error callback and is a shorthand for then(null, errorCallback).
finally will run regardless of the outcome of the promise.

Promises can be chained together (composed):

Promise.then(successCallback).then(successCallback);

Each then function will return a derived promise passing either: the success value down the chain, or the error reason.

A nice feature of the AngularJS implementation is that if a success callback returns a promise instance as value, the chain will resume execution only after that promise is fulfilled and its value will be passed on to the next success callback.
So basically, we can nest other promises inside the success callbacks and have them interleave with the main promise chain. This is immensely powerful and enables us to build services such as the ones mentioned in the Use cases below.

The $http service

In AngularJS, the $http service depends heavily on promises. Each $http call will return a special kind of promise instance that has two extra methods attached: success and error. These methods will receive parameters such as data, status, headers and config from the service, which enable us to work more easily with requests.

However, it’s the $http interceptors where promises really shine. Interceptors allow us to hook into http requests and responses, and react to them or modify their outcome. Here are a few examples:

Pre-processing of requests:

Displaying a progress indicator.
Changing the request Content-Type header, useful when performing asynchronous file uploads for example.

Post-processing responses:

Authentication: each time a 401 http response is encountered the user could be redirected to the login page.
Global error handling: each time a certain kind of error is received from the backend (such as the user not having enough credits to perform an action) a notice could pop up.

Interceptors can be chained together, since they can return a promise.

Use cases

Here are two custom services that rely on promises.

ACL service

In larger client-side applications you will most probably need some form of ACLs. ACL stands for Access Control Lists and represents a mechanism for authorizing users to certain parts of the application based on the user group they belong to.

Reduced to its simplest form, ACL could be expressed as a list of user groups, with each group entry containing a list of allowed application routes.

At Zooku, we use a hybrid ACL service for the upcoming Control Panel. This service obtains its routes in two ways:

synchronous — some of them are predefined and do not change while the application is running.
asynchronous — some are requested from the server-side and change when the user’s active group changes.

Promises enable us to have a common interface for checking both synchronous and asynchronous requested routes.

By creating an initial promise that always resolves, we can then chain a getUserGroup and a getAcls request to it. These two requests are performed in the bootstrapping phase of the application or when the user changes credentials, such as logging in.

After the active user group and the asynchronous ACLs are set, we can then proceed to perform the route check itself, which will also be chained to the previous requests.

In our case subsequent route checks will not request the user group nor the ACL routes again (they will be cached for the active session), but by using promises we created an uniform interface that works regardless of the application session state.

Validation service

A well designed validation service has the potential to truly help the user and eliminate a lot of frustration.
While HTML5 introduced a few helpful attributes for dealing with validation, employing them in a large application can quickly result in views overloaded with validation attributes, or with attributes that are out of sync with the backend.

For both Zooku and Sendmachine, we developed a Validation service that has a few nice features:

Uses a chain of rules to validate a field, each rule providing the user with hints about the current state of the field.
Requests validation rules from the server-side, which enabled us to use the same rules on both the server and the client application.

While most rules perform synchronous checks, like checking if a required field is filled-in, others perform asynchronous checks, like checking with the backend if an email is already registered.

This is another case there promises prove to be very useful and enable us to perform checks on a field regardless of their nature.
The logic is very similar to the ACL service: an initial Promise instance that always resolves is created, and promises that perform further validations are chained to it.

These are just a few use cases where promises play a significant role. More documentation and code samples can be found in the resources below.

CORS Gotchas

2012-07-09T00:00:00+00:00

CORS stands for Cross-Origin Resource Sharing and represents a method of accessing/sharing resources across domains. These resources can be anything from web fonts to APIs. The CORS standard is only implemented in browsers, since this is the only place where it makes sense. While this is the preferred method of sharing resources, it has a few rough edges.

The Basics

In the following we’ll refer to the response headers, since the request headers are automatically set by the browser.
A special header defined in the CORS specification must be sent by the server in each response to allow access to a resource:

Access-Control-Allow-Origin: allowed-domain.com

To allow any domain to access resources the header can be set to a wildcard:

Access-Control-Allow-Origin: *

This is the bare minimum and it will suffice for sharing static resources such as web fonts. But once we throw in authentication, which is needed by most APIs, things get messy.

Authentication

Authentication, with either cookies or HTTP auth, requires an additional response header:

Access-Control-Allow-Credentials: true

Also the browser needs to send cookies/credentials in the request, so for example in JavaScript we need to pass an additional parameter to XHR object:

var req = new XMLHttpRequest();

req.open('GET', url);
req.withCredentials = true;
req.onreadystatechange = handler;
req.send();

A few notes on authentication:

We cannot use domain wildcards for the Access-Control-Allow-Origin header when allowing credentials. The domain must be explicitly stated. This is a feature to prevent CSRF.
Credentials are not allowed with synchronous requests in some browsers, since these requests cannot be pre-flighted.
IE and Opera were slow to adopt CORS; only IE 10 and Opera 12 support it properly.

Pre-flighted requests

Some cross-origin requests are pre-flighted, meaning that an OPTIONS request is automatically sent by the browser before each request, to query the server for allowed HTTP methods, allowed headers or to check whether credentials are allowed.

OPTIONS requests are only sent if any of the following conditions are met:

The request method is other than GET or POST.
The request Content-Type is other than application/x-www-form-urlencoded, multipart/form-data or text/plain.
Custom headers are set.

Obviously, sending OPTIONS before each real request has a bit of overhead. Fortunately, there is a header that tells the browser to cache these requests:

Access-Control-Max-Age: 86400

Allowing methods and headers

In order to create RESTful APIs we need to use more than the GET and POST verbs.
Servers can inform the clients of the allowed HTTP methods by using another type of CORS header:

Access-Control-Allow-Methods: GET, POST, PUT, DELETE

Any number of HTTP verbs can be listed here.
Allowed headers can be set with:

Access-Control-Allow-Headers: Origin, X-Reqested-With, Accept

X-Reqested-With is needed for asynchronous requests.

All these headers except Access-Control-Allow-Origin are commonly included only in the pre-flight response.

IE support

As always, IE chooses to hit developers with a crowbar.
Instead of extending XMLHttpRequest, IE 8 introduces another interface named XDomainRequest, with a similar interface, which supports CORS, but not its full set of features.
Luckily IE 10 fixes this and adds support for CORS within the standard XMLHttpRequest.

Notes on RESTful APIs (Updated)

2012-04-17T00:00:00+00:00

HTTP and REST are not new, we just misused them for a long time. A proper RESTful API should consider the following principles.

Resources

Resources are discrete entities — like the entities from the Entity-Relationship Model. The web is modeled around resources. Everything is a resource, not a web page, not an application. Resource names should be intuitive and should be nouns. Avoid Verbs in URLs.

/resource
represents a collection of items
/resource/id
represents an individual item
/users/1/address
Resources can be nested in order to reflect the relationship between them.

Querystring parameters

In order to filter, order, limit or offset resource entries we should not create separate sub-resources. Here’s where querystring parameters come into play.

/addressbook?orderby=lastname
/users/1/vehicles?filter=bikes

HTTP Methods (Verbs)

The HTTP Verbs operate on resources. Resources can be queried and altered using Verbs. Some operations are idempotent.

GET
read item / collection of items
nullipotent – does not modify state, no side effects
POST
create new item alters state each time it is called
PUT
update item
idempotent – calling it multiple times has the same effect as the first time
DELETE
delete item
idempotent
PATCH
partially update item
idempotent
OPTIONS
display allowed verbs on a resource
nullipotent

How verbs operate on resources

Example scenario:

/resource
GET collection of items
POST create new item
PUT update collection of items – this is rarely used since it updates the whole collection
DELETE collection of items
OPTIONS could display a quick help on how to query the resource
/resource/id
GET item
POST - using POST to create an item at an unexisting resource id is a nono
PUT (sometimes POST) update item
DELETE item

Responses

Give a proper response after each request. Applications can be much easier to develop if they get meaningful response codes.

after GET
respond 200 ‘OK’ (default)
body with item
after POST (create)
respond 201 ‘Created’
location: /resource/id of created item
body with created item
after PUT/POST (update)
respond 200 ‘OK’
body with updated item
after DELETE
respond 204 ‘No Content’
empty body
after a bad request/unallowed method
respond 400 ‘Bad Request’
empty body
after encountering a problem on the server side, like a failed SQL query
respond 500 ‘Internal Server Error’

Hypermedia

Self describing API
Each response should provide links to explore the API.
Just like HTML connects multiple pages through links, API responses should have links to other resources.
Ideally an API could be explored entirely without prior knowledge of its resources — just by knowing its base URL.

Content negotiation

Using HTTP headers
Accept: application/json, text/plain
Extension in the URL
Not RESTful — URLs are not the place for Content-Type

Versioning

Version in the URL
/v2/resource
Create separate resources for a new version of the API.
Version media types
application/vnd.something.v1+json
Custom header
X-API-Version: 1

Language

Using HTTP headers
Accept-Language: en

Cache

Some responses can be cached.

Expiration
Cache-Control
Expires
Validation
Last-Modified
ETag

Authentication

OAuth
Industry standard. Used by most services.
HTTP Auth
Part of the HTTP standard. Digest auth uses hashing and a nonce.
Cookies
Can be used for login similar to classic web applications.

ACTA, SOPA, PIPA, CISPA

2012-04-08T00:00:00+00:00

It seems like the big players of the software and hardware industry are dead set to bring to life an incarnation of one of these laws. Never before in the history of computing has a piece of legislation been pushed forward so ferociously by so many significant companies.

The core of the law is mostly the same, and the name doesn’t even matter anymore since it masquerades under so many acronyms. Of course ACTA is an international agreement, while SOPA, PIPA, CISPA are US bills. But once either of these is adopted, slightly modified versions will spread around the globe and will be adopted by other countries as well.

So it seems that no matter the number of protests, they will keep pushing them forward under different names. A lot of internet users are aware at this moment of one form or another of these bills, thanks to folks like Google and Wikipedia who brought mass awareness, but also thanks to individuals and lobbyists who spread the word. So there is not a lack of awareness.

Sadly, I don’t think there is much we can do. We could boycott one company after another, like Maddox proposes and hit them right in the money. If we manage to do this, it might prove effective for a while and one or two companies might even go out of business. But in the long run the bill will pass and drastic measures will be taken. ISPs will be turned into copyright cops and SOPA supporting companies into big bad wolves.

Look, the truth is that we pirate stuff. We pirate games, music, movies, tv shows, books and whatnot. This has got to stop at some point, we can’t just pirate media indefinitely. If we want to be civil and honest we must pay content creators for their sweat and blood, so they can create more of the things we love.

The problem is that only a (very) small percent of the money makes it to the content creators. Most of it ends up in MPAA and RIAA’s pockets. And they provide little value for that large percent which they keep for themselves. They do not provide modern channels of media delivery, they do not provide fair prices and they do not let us manage our own legally bought media by enforcing draconian DRM schemes.

They are outdated. They hold on to their old ways of doing business and are very afraid of this whole thing called the Internet. If it were for them Blu-ray discs with heavy DRM would be the only way of distributing content. Oh, and the Internet would only be a poster for advertising their brick and mortar shops.

This has got to change. We need better distribution channels, we want to buy songs and movies from anywhere in the world, delivered instantly and available for consumption on any device we own. If Bittorrent is such a great technology for delivering media, why not use it for delivering your own content to end users? We need more flexible and fair pricing schemes. If private Bittorrent trackers are such a success, why not make one where you actually buy ratio and then spend it on any content you like? The monetization possibilities are endless, you just need to give your users something so advantageous that it would make pirating look like an unattractive option. DRM is also a big issue since you don’t feel like you own the content. They own it and they can erase it from your device at any given moment. We need to own it. Maybe it’s more of a psychological need than anything else.

I’m sure big media companies could manage to create any of these outlets for consumption. The only thing keeping them from doing so is themselves. Until these issues are solved we will keep pirating for as long as we can. Because pirating solves these practical issues, even though it does not address the ethical part.

PDO vs MySQLi performance comparison

2012-04-03T00:00:00+00:00

After reading this article on Nettuts, I was curious how PDO would compare to MySQLi in a real-world scenario, perfomance-wise. So I created a benchmark to dispell the myths surrounding this dispute. MySQLi is percieved to be the better performer, since it’s the official extension supported by Oracle. In the current post we’ll test each database driver.

The Setup

For any test result to be relevant we need:

an isolated test environment
to run multiple rounds of the test
to specify the software version used

The software I used was: PHP 5.3.6, MySQL 5.1.58 on Ubuntu 11.10 with Linux kernel 3.0.0

I tested the performance of the four most important SQL operations: INSERT, SELECT, UPDATE and DELETE. These four operations are enough to determine the raw performance of each driver.
The test suite consists of running these queries 100000 times (100k) and 1000000 times (1M) for each database driver.

To make sure that the test is not affected by other applications (such as a sudden cron job), I ran the tests from the CLI, with other daemons shut down.

The SQL schema for the database is:

Groups table

id_group (PK)
title

Users table

id_user (PK)
id_group (FK)
first_name
fam_name
email
pass

Not very complicated. I ran the queries only on users table. In both cases I used prepared statements (with named parameters for PDO).

Results

Here are the results of the tests. Lower values are better, of course.

PDO results for 100k queries

query	time in seconds
insert	13.079864025116
select	19.150141954422
update	16.263291120529
delete	14.891561985016

MySQLi results for 100k queries

query	time in seconds
insert	19.463269948959
select	27.461564064026
update	22.705169916153
delete	21.583913087845

PDO results for 1M queries

query	time in seconds
insert	133.66887807846
select	197.49514484406
update	164.97567796707
delete	150.77637290955

MySQLi results for 1M queries

query	time in seconds
insert	202.65716481209
select	283.04056501389
update	231.89973783493
delete	232.20053887367

The results are quite surprising: PDO performs better than MySQLi in all tested cases. You can find the source of these tests on Github.

Apart from better performance, PDO has proper exception handling, a better object-oriented interface and supports binding of entire arrays. Now you have no reason not to use PDO.

Erratum

This benchmark is flawed, as PDO does escaping/preparing entirely on the client-side, while MySQLi uses the mysql protocol for that (thus network IO).

Hello world

2012-03-22T00:00:00+00:00

Hello world! It’s about time I launched this thing. While not being perfectly polished yet, it is close enough to what I wanted so I decided to give it a go.

I will post here my findings, notes and scribblings related to development. Hopefully I’ll do it on a regular basis.

Published Notes

Publishing with Github Actions

The Actions job setup

The S3 bucket

Cloudflare

Wrap-up

Update

Docker multistage builds

Read more

Quick IPTables filtering

Read More

Modding the Inspiron 7570

The intrigue

The shop

The teardown

Screen teardown

Around the edges

The switcheroo

The final touch

Read more

Mirroring Git repos

Introduction to Python pathlib

Scraping with BeautifulSoup

Read more

Marshmallow schema basics

A new AngularJS project seed

Keeping home configuration in git

Building apps with Gulp

The API

Read more

Serving WebP today

The benchmark

Serving WebP

Setting up nginx

More

Prototype inheritance in JavaScript

Uses

Subclassing

Typed Errors

Read more

Promises in AngularJS

So what are promises?

Generic vs. AngularJS implementation

The $http service

Use cases

ACL service

Validation service

Further reading

CORS Gotchas

The Basics

Authentication

Pre-flighted requests

Allowing methods and headers

IE support

Further reading

Notes on RESTful APIs (Updated)

Resources

Querystring parameters

HTTP Methods (Verbs)

How verbs operate on resources

Responses

Hypermedia

Content negotiation

Versioning

Language

Cache

Authentication

Also Read

ACTA, SOPA, PIPA, CISPA

PDO vs MySQLi performance comparison

The Setup

Results

Erratum

Hello world