John Paton

Syntax highlighting for console sessions

2020-11-21T14:00:00-01:00

It’s a minor annoyance that comes up often in GitHub comments: syntax highlighting for Python console sessions. You want the input code (after the prompt) to be highlighted, but the output (which is generally just text or logs) to remain neutral. As far as I can tell, most people opt for one of two options: no highlighting at all (boring, can be harder to read), or Python syntax highlighting (looks nicer but probably makes all kinds of weird colors in your output).

Here’s a classic example where the output includes a progress bar:

>>> my_str = "hello"
>>> some_int = 123
>>> func_with_output(a=1, b=my_str)
Processing: 100%|██████████████████| 10/10 [00:01<00:00,  9.71it/s]

We would like to spruce this up a bit with some syntax highlighting. If we start our code block with ```python instead of just the plain regular triple backticks (which start a code block in Markdown), we end up with something that looks like

>>> my_str = "hello"
>>> some_int = 123
>>> func_with_output(a=1, b=my_str)
Processing: 100%|██████████████████| 10/10 [00:01<00:00,  9.71it/s]

This is okay, but the progress bar looks silly and it makes it more difficult to tell what’s input and what’s output. However, there is actually another syntax highlighter specifically for this case: pycon (instead of python). it highlights lines that start with the Python prompt (>>>), but doesn’t highlight output lines which don’t have any prompt, leaving us with:

>>> my_str = "hello"
>>> some_int = 123
>>> func_with_output(a=1, b=my_str)
Processing: 100%|██████████████████| 10/10 [00:01<00:00,  9.71it/s]

As a bonus, for shell sessions (which usually have a less obvious prompt, making it more important to distinguish input from output), you can do the same thing using the console highlighter!

$ echo "output colored differently"
output colored differently

Perfection.

Receiving a Google Open Source Peer Bonus award

2020-05-07T11:00:00-01:00

Over the past few years I’ve increasingly tried to make small contributions to open source projects that I use. Usually these come from me encountering a small issue that I know how to fix, or lightly improving the documentation. Occasionally I’ve also used the process of “making your first contribution” to get to know a tool from the inside. I’m not a developer, and I’m also not on the core team of any one project, so usually my contributions are very small. That’s why I was very surprised a few weeks ago when I got an email from Google’s Open Source Peer Bonus program, letting me know that I had been nominated!

It turns out that Google’s employees who maintain open source projects can nominate contributors as a thank you from Google and from the community. Receiving the award gets you a nice letter, a mention in their blog post, and a small cash prize. I was nominated for contributing to the Python client libraries for Google BigQuery (google-cloud-python and its pandas-compatible layer, pandas-gbq), which we use every day at Catawiki.

These were the contributions I was nominated for:

pydata/pandas-gbq#256: My first time contributing to one of these projects, this PR was just a simple cleanup of a usage example in the documentation. More of a nitpick than anything else, but they reacted in a friendly way which made it much less scary to contribute actual code in the future.
pydata/pandas-gbq#257: This one was an actual feature for pandas-gbq. When you upload a DataFrame to BigQuery, pandas-gbq generates a schema for your table based on the pandas dtypes. Alternately, you can provide your own schema if you want more control, e.g. when dealing with time columns (DATE? DATETIME? TIMESTAMP?). This PR added the ability to provide a schema for only a subset of columns and to fall back to the generated schema for the rest, which is handy if you have a very wide DataFrame and you only want to cast a few columns.
googleapis/google-cloud-python#7552: I’m a simple man. I see incremental progress, I want a progress bar. This PR, which originally started as a feature request on pandas-gbq, added a tqdm progress bar to Google’s BigQuery client when downloading data from BigQuery. I personally wanted this as I was spending a non-negligible amount of time in those days downloading large-ish training sets. As you’ll see in the comments of the PR, I got a ton of feedback about my initial commits. It was all very friendly and I learned a lot about the code-review process among real software engineers. In the end, my original request to include this feature in pandas-gbq (and not just google-cloud-python) was actually implemented by someone else! All around a cool example of open source in action.

It’s pretty common to be afraid of contributing to open source projects because you’re worried about making mistakes, being criticized, etc. For data scientists, I expect this to be even worse as you likely have the added layer of impostor-syndrome from not being a “real” software developer. But if you have tools that you use often and that make your life better, then you get to know them and their quirks pretty quickly, which makes it easier to get started. And maintainers of projects are often very happy to know that people care and want to contribute, especially if you read their CONTRIBUTING.md or other contribution guidelines first. At the time of writing, there are over 15,000 open “good first issues” in Python projects on GitHub, specifically tagged for people looking to get involved for the first time. Why not give it a try and dive in?

Introducing airbase: A Python client for the European Air Quality e-Reporting Database

2020-02-03T15:00:00-01:00

As part of its open data initiatives, the European Environment Agency (EEA) provides a selection of datasets about air quality in Europe, collectively known as the Air Quality e-Reporting database. The richest subset of this data is the air quality time series data. There are 2 subsets: E1a, which is cleaner and more validated, and E2a, which is more up to date. This is an important dataset for climate research, as well as a great source for data scientists who are looking for some non-trivial real world data to practice on (when combined with the measurement station metadata, you have both geodata and timeseries).

The one major obstacle to using this data is the download interface (a.k.a. “the portal”), which is very cumbersome. First you need to navigate to the download page and select the parameters of the dataset you want to download, like the pollutant of interest, the location, and the time span:

The form constructs a URL with query parameters matching what you’ve selected in the form (this was my first hint that this should be easy to automate). Once you hit the “Download” button, you would expect that it would download the dataset, but that would of course be too easy. If you hit the button for the parameters I’ve filled in you’re greeted with a list of links to various CSV files to individually click on and download yourself.

For this particular query this isn’t too bad, but if you want to query many types of pollutant or a bigger group of countries, this is going to cost you some serious time.

A better way

To make this process easier, I developed airbase: an easy Python client for accessing the data (this database was formerly known as AirBase, and I thought it was a catchy name). It started off as a script to help a friend of mine who is in climate research, and I realized with a bit of cleanup it might be useful to other people as well. It’s available on PyPI, so to install you can simply

$ pip install airbase

To start downloading your dataset, import the package and initialize the client:

>>> import airbase
>>> client = airbase.AirbaseClient()

The client helps you to construct your request, and does some validation for you, like checking that the pollutant you want is available in the countries you’re asking for. It does this by downloading some files from the portal, so this requires an internet connection.

Kind of like using the portal, but more conveniently, you next construct the parameters of the dataset you’re looking for:

>>> request = client.request(country="NL", pl="NO3", year_from=2014, year_to=2017)

If you don’t include a parameter, the client will construct a request for all possible values (so you can just user client.request() to get the whole dataset).

With your request constructed, all that’s left to do is to choose how you want to download the data. You can choose to either download_to_directory() to get all those CSVs individually, or you can download_to_file() to concatenate them into one big CSV. Either way, the request object will first contact the portal to get the links to all the CSVs you need, and then start downloading them as instructed. Of course, you can follow the progress with nice progress bars.

>>> request.download_to_directory("./data")
Generating CSV download links...
100%|██████████████████████████| 1/1 [00:03<00:00,  3.14s/it]
Generated 16 CSV links ready for downloading
Downloading CSVs to ./data...
100%|██████████████████████████| 16/16 [00:03<00:00,  5.34it/s]

If you want to update your dataset later (e.g. getting the last week’s worth of data), download_to_directory() will automatically skip downloading most of the files that are already there.

Hunting for correlations between locations? Make sure to download the metadata file that contains the locations and other properties of the measurement stations that supply the data:

>>> client.download_metadata("./data/metadata.tsv")
Writing metadata to ./data/metadata.tsv...

Full documentation is availble on ReadTheDocs, and of course the whole package is open sourced on GitHub too. I know at least a handful of people have used the package, including one confirmed publication (by my friend who I made the original script for), which is very cool!

Have you used airbase in your research or learning? Let me know! I’d love to hear about it!

Schedule the interruption of hung Python processes with signals

2019-07-13T13:00:00-01:00

At Catawiki, we have a lot of scheduled cron-type jobs that move data around, train models, and do other processing tasks. Often, they are dependent on external resources like database connections. If one of these jobs actually fails, emails are sent, and we can fix the problem. However, we have had issues in the past with scheduled jobs silently hanging, which can often go unnoticed until someone’s output hasn’t been refreshed. One solution to this would be to schedule these small jobs in our Airflow pipeline, but for little things that just need to run every night without hassle we use this lightweight trick instead.

To make a Python process fail and get a precious notification, we can force it to raise a TimeoutError after a certain number of seconds using the signal library. This library is responsible for handling Unix Signals, which are a way to communicate asynchronously with running processes.

Note: If you are doing fancy stuff with multiple threads, please read the signal docs as there can be weird behavior here. Also, alarm scheduling is only available on Unix-like systems.

Technically Step 3 below is already enough to cause the timeout by itself, but your output will be very terse. Defining the signal handler with a friendly message ensures that both your colleagues and your future self will be able to read your code and your logs.

Step 1: Define a Signal Handler

A signal handler is a function that can be called when a signal is received. Signal handlers must accept two arguments, but our handler will just ignore them and raise a TimeoutError:

import signal

def timeout_handler(sig, frame):
    raise TimeoutError("Your process has timed out!")

Raising an exception will interrupt the process, killing whatever was going on. If this is too extreme and you just want to be alerted that a process is still running without killing it, you could also do something like send an email or send yourself a Slack message:

import os
import slack

def slack_handler(sig, frame):
    # Don't keep your token in plain text :)
    token = os.environ["SLACK_API_TOKEN"]
    client = slack.WebClient(token=token)

    client.chat_postMessage(
        channel="#python_alerts",
        text="Your process is still running!"
    )

Step 2: Assign the signal hander to the alarm signal

Next we use signal.signal to tell Python that our new handler should be called whenever the running process receives the alarm signal, denoted by signal.SIGALRM.

# Call `timeout_handler` when we receive an alarm signal
signal.signal(signal.SIGALRM, timeout_handler)

Step 3: Schedule an alarm

Now we can schedule an alarm to be sent to our process after a set number of seconds. If the process has exited before that time has passed, nothing will happen. If the process is still running when the alarm is sent, then our handler will be called, interrupting the process and raising an exception. If you’re going to interrupt a process, this timeout should be comfortably longer than you expect the process to take, so that it is only interrupted if something is really stuck.

# Schedule an alarm to be sent in 10 seconds 
signal.alarm(10)

You can now run your main loop or do whatever work you are trying to do, and you’re guaranteed that the process will fail if it takes longer than the time you specified.

If you are writing a module with functions that might get imported elsewhere, make sure to put the signal.signal and signal.alarm lines under your if __name__ == "__main__" statement so that the alarm doesn’t get triggered every time your module is imported.

Complete Example

# timeout.py
import signal  

def timeout_handler(sig, frame):
    raise TimeoutError("Your process has timed out!")

# Call `timeout_handler` when we receive an alarm signal
signal.signal(signal.SIGALRM, timeout_handler)

# --- Example Usage

# Schedule an alarm in 10 seconds 
# (will raise TimeoutError as specified)
signal.alarm(10)

# Do some work which takes too long
import time
i = 0
while True:
    print(i, "seconds passed")
    i += 1
    time.sleep(1)

Running our example yields the following output:

$ python timeout.py
0 seconds passed
1 seconds passed
2 seconds passed
3 seconds passed
4 seconds passed
5 seconds passed
6 seconds passed
7 seconds passed
8 seconds passed
9 seconds passed
Traceback (most recent call last):
  File "timeout.py", line 21, in <module>
    time.sleep(1)
  File "timeout.py", line 5, in timeout_handler
    raise TimeoutError("Your process has timed out!")
TimeoutError: Your process has timed out!
exit 1

Generating fake whiskey reviews with GPT-2

2019-06-23T19:00:00-01:00

I’ve enjoyed whiskey for a while now, but I can never vocalize all the flavors present in a bottle. I read all these flowery reviews and tasting notes online and I just have no idea how these people come up with descriptions like “caramels, dried peats, elegant cigar smoke, seeds scraped from vanilla beans, brand new pencils, peppercorn, coriander seeds, and star anise”… until now.

The data
Fine-tuning GPT-2
Fake whiskey reviews
Highlights
Bloopers

OpenAI recently made headlines with their blog post Better Language Models and Their Implications, in which they described their latest general language model, dubbed GPT-2. In the post, they claim to be practicing responsible (non)disclosure by not releasing the full pre-trained model due to concerns that it is so good, it could be used for fake news generation or social media manipulation. However, they did release smaller versions of the model which are nonetheless still quite performant. You can play with one of them at talktotransformer.com.

Obviously this piqued my interest. I’ve dabbled in purposely bad language generation, but this time I was curious about doing something vaguely believable. I decided to try to sound more impressive in my whiskey-ing by generating my very own fake whiskey reviews, fine-tuning the small version of GPT-2 on existing reviews.

Image by the New York Post

The data

The first site on Google when I searched for “whiskey reviews” was whiskyadvocate.com, and it seemed perfect for my needs. Short, consistently formatted reviews, and all on one long page making them easy to scrape.

The only snag is that you have to click a “See More” button over and over to get 6 more reviews to render each time. I know you can use sophisticated scraping tools to deal with this situation, but I decided that that would be overkill for this project. Luckily I’ve been JavaScripting a little bit recently so it occurred to me that you can pop open the browser console and run:

> var btn = document.getElementById("loadMore")
> for (var i = 0; i < 100; i++) {btn.click();};

After glitching out for a bit, the whole set of reviews is visible on the page. One ctrl+S and we have the reviews ready for parsing with beautifulsoup. Looking at the html, we see that each review is in its own <article> block, so they are easy enough to extract:

from bs4 import BeautifulSoup
with open("data/raw/whisky_advocate.html") as h:
    raw_data = h.read()
soup = BeautifulSoup(raw_data)
articles = soup.findAll("article")

Checking out the text structure of one “article”, that all we really need to do to clean up is drop empty lines and replace non-breaking spaces (/xa0) with normal ones.

reviews_clean = []
for a in articles:
    a_clean = [
        line.replace("\xa0", " ")  # breaking spaces
        for line in a.text.split("\n") 
        if line is not ''  # drop empty lines
    ]
    # rejoin into one formatted snippet
    reviews_clean.append("\n".join(a_clean))

This gives us nice clean review snippets in the following format:

Real Review (input data)

95 points

Uncle Nearest 1820 Single Barrel 11 year old Tennessee Whiskey (Barrel US-2), 55.1%

Bourbon/Tennessee | $119

This is mature whiskey at its most refined: a balance of fruits, nuts, sweetness, and restrained oak. The nose has it all: salted, buttered pecans, rock candy, Dr. Pepper, blackberry jam, dried blueberries, caramel corn, tobacco barn, and old leather chair. It’s practically dessert-like in the mouth, with dark chocolate-covered caramel, candied pecans, Goo Goo Clusters, cherry cola, blackberry and blueberry jam, and a kiss of white pepper. The finish stays consistent, a mouthwatering mélange of caramel, chocolate, and nuts. Harmonious, seamless, and silky—you’d never guess the proof.

Looks (and sounds) good to me! The last step is to save our reviews back to a text file for processing by GPT-2. The model distinguishes between separate pieces of text using a special token, <|endoftext|>. So we join all the reviews back together, separated by this token, and save it to a text file:

with open("data/clean/reviews.txt", "w") as h:
    output = "\n<|endoftext|>\n".join(reviews_clean)
    h.write(output+"\n")

Now we’re ready to train!

After training the model, it turned out I was only using the Summer 2019 Buying Guide and not the full set of reviews, so there is definitely room for improvement here.

Fine-tuning GPT-2

Surprisingly, this is actually the easy part, thanks to work by nshepperd. His/her fork of GPT-2 contains a super easy train script for finetuning GPT-2, with lots of options. To get started with it, we clone the repo and use the provided script to download the pre-trained model. This model has already been trained on a set of 8 million web pages, so it already has a pretty big and diverse vocabulary. We’ll download the smaller model, called 117M:

$ git clone https://github.com/nshepperd/gpt-2.git
$ cd gpt-2 
$ pip3 install -r requirements.txt 
$ python3 download_model.py 117M

This will save the pre-trained model in the models directory.

The set of reviews isn’t very big (the whole site contains about 4,000, but I only grabbed 125 during this experiment due to the mistake mentioned above), so this will be less like fine-tuning and more like clobbering the model. When we are done, it will know nothing but whiskey reviews. For that reason, we output samples frequently during training, with the expectation that the sweet-spot for interesting new reviews will come somewhere before the model is totally overfit.

With our processed reviews in data/clean/reviews.txt, we run the training script on our data, setting the PYTHONPATH variable as indicated in the readme. We output 30 samples every 25 epochs, with a length of 250 words (long enough that a full review should fit in there). We also checkpoint the model at that point so that we can return to it to generate more samples later if we want:

$ PYTHONPATH=gpt-2/src python3 gpt-2/train.py \
      --dataset data/clean/reviews.txt \
      --save_every 25 --sample_every 25 \
      --sample_length 250 --sample_num 30

The script will create a samples directory and save the samples there, and also a checkpoints directory. It runs until you kill it with ctrl+C, so feel free to go get a coffee.

Fake whiskey reviews

The first thing to notice about the output is that the generated text is usually really coherent at first glance. After 25 epochs, the model isn’t really talking about whiskey yet, so we mostly get a peek into the kind of text generation that it’s capable of:

GPT-2 SAMPLE: 25 Epochs

We are pleased to present the third of over 1,000 stories published by the New England Journal of Medicine, from all continents. The authors have collected information on over 50,000 individuals in the United States who were examined at the National Heart, Lung and Blood Institute (NHBI) and included information on symptoms and treatments. The research was focused on coronary artery bypass grafting, cardiovascular dysfunction, cardiovascular disease, and coronary artery disease.

For complete details and results, visit http://www.nlm.nih.gov/pubmed/257720.

The URL doesn’t work, I checked. This is generated text.

After 50 epochs, we’re already seeing some promising output, though there’s definitely work to do:

GPT-2 SAMPLE: 50 Epochs

The packaging is as it should be: it looks fine (with more sugar than water), but the nose gets burnt out, which can be remedied with some honey. There were a couple of more hints of red wine and chocolate, but in the nose this gives way to a more complex whisky with plenty of chocolate and dried fruit. On the palate, it’s just perfect. The nose is quite dry and mouthful, but more like a whisky where, when you pour it into your mouth and add it, you find the sweetness of cherry-chardonnay, fresh orange peel, ginger, and vanilla peel. On the palate, it develops chocolate, vanilla syrup, and nutmeg, along with a light and pleasant hint of dried fruit, then is broken up by chocolate, cherry, and pepper notes. Very well-designed, although this was not finished in time.

1/5

At 100 epochs, we have a review that at first glance seems reasonable:

GPT-2 SAMPLE: 100 Epochs

85 points

Black Cherry Ale, 90%

Blended Malt Whisky | $60

The nose was very light and crisp, with chocolate, banana candy flavor, honey, espresso, toffee, and herbal aromas. The palate was warm, with vanilla, cinnamon, and chocolate sweetness. The finish was medium in length, with plenty of vanilla and chocolate, and the finish was strong and well-balanced: chocolate, caramel, milk chocolate, milk chocolate, caramel, and sweet almond.                        

Note that the model is even mimicking our formatting. This one ticks all the boxes of a review, but check out the tasting notes: vanilla, cinnamon, chocolate, caramel, sweet almond. There’s definitely a theme: these are baking ingredients! Many food words are used in tasting notes. The model is learning this, but the text it was trained on seems to have included many recipes, as all sorts of baking instructions are also being generated. This was another sample from the same batch:

GPT-2 SAMPLE: 100 Epochs

1 pint brown sugar

1 egg, beaten

1 teaspoon kosher salt

3 ounces butter, softened

3 egg whites, beaten

12 drops honey, at room temperature

2 tablespoons brown sugar/butter

3 tablespoons apple cider vinegar, at room temperature

2 tablespoons powdered pepper

1/2 teaspoon pepper

1 tablespoon black pepper

Directions

In a medium bowl, beat butter, 1/2 cup apple cider vinegar, and vanilla until smooth, 2-4 hours. Add in egg whites, and gradually add more egg whites, if needed. (You may need to add more milk and so forth.)

Pour milk/soy mixture through an egg grinder; if it is hard, you might have to do it some more. If there isn’t quite so much water, you’re going to have to let the milk drain on the stove top.

Add water once, once or twice, until all the water is dissolved, about 3-4 hours. Add pepper, egg whites, vanilla, and water.

Whisk in egg shells and water; the water gets very sticky; add salt

“Pour milk/soy mixture through an egg grinder; if it is hard, you might have to do it some more” isn’t exactly sensible, but it is definitely coherent and the whole thing reads like a recipe. Just don’t try it at home.

After 200 epochs, the model is producing whiskey reviews that I (as a layman) can’t distinguish from the real thing:

GPT-2 SAMPLE: 200 Epochs

91 points

Luxembourg 60 year old (Batch 2), 59%

Single Malt Scotch | $235

Quite simply the most beautiful whisky in Europe at the time. Beautifully balanced spices, herbal oloroso fruits, soft earthy aromas of honey, licorice, lavender, lavender oil, and a hint of espresso. The palate is rich and richly spiced, with dark tannins, orange candy, green bananas, almond milk, espresso, brown sugar, and cream cheese louis. Floral notes and honeycomb in the finish. (12,800 bottles for U.S.)

However, it is also revealing one of GPT-2’s flaws: it has a tendency to abruptly switch context, which can be quite comical. From the same batch:

GPT-2 SAMPLE: 200 Epochs

A group of scientists led by Professor Jonny Wilkinson has found that the planet Saturn has the highest amount of formaldehyde on Earth. The concentration falls short, with only slightly more than 100 parts per million of the general background. It is 80 parts per million in the form of sisphenol A (s<3 parts per million<30 parts per million): this is also found in less-drying water like water and soda water. The taste is more palatable, with more of both honey and orange, then chocolate hazelnut, vanilla pod, chocolate chip cookies, caramel, hazelnut, coconut, and a sprinkling of orange oil. There are also more dark chocolate aromas, which are sweeter and with some spice and nutmeg addition. One might expect there to be more chocolate, but, on closer inspection, it’s more fruitcake and chocolate barristers will ultimately find it all.

and:

GPT-2 SAMPLE: 200 Epochs

90 points

Lemon Keg, 57%

Japanese | $45

The standard edition features a black leather jacket with leather cuffs and white floral print buttons. The back has a floral feel, but is made from milk chocolate; the outside shell lacks much citrus or sweetness, which ultimately leads to a light taupe hue and ample mouth-drawing.

This is also what I meant by clobbering the model: it is forgetting how to remain in other contexts. No matter where it starts, it is beginning to see everything as a whiskey review. By this point most (but not all) of the samples are also formatted like our whiskey reviews. This distinctive formatting proved to be a very useful tool in gauging to what extent we were overfitting the model.

By about 250 epochs, most of the output is pretty believable, with the only consistent flaw being repeated flavors in the tasting notes. All in all, I was surprised at both how easy the open source community made this experiment, and also how quickly and accurately the model was able to start replicating reasonable-sounding reviews (training only took about a minute per epoch on my old-ish MacBook Pro).

This post is getting long enough, but I’ll end with some highlights and bloopers:

Highlights

Consistency in country of origin (Canadian) and location of distillery (Banff):

GPT-2 SAMPLE

87 points

Millstone 92 Rye, 46%

Canadian | $99

The second installment in our four-part rye limited-edition comparison, this proof-bodied reissue from the Millstone Works facility in Banff presents both dry and wet grasses with great maturity. The nose shifts to vanilla, honey, cinnamon, orange oil, licorice, and crystalized halvah. The palate is light and delicate, with nuttiness, cloves, green apple, bitter chocolate, pecan pie, caramel, and bittersweet chocolate. The finish yields leather, honey, roasted nuts, orange, dried citrus, subtle oak, and dried coriander, all evocative of a wartime log. Editors’ Choice

Sweet and smooth:

GPT-2 SAMPLE

92 points

Timersley’s Barrel-Aged Bourbon, 44%

Blended Scotch Whiskey | $30

Sweet potato, strawberry jam, milk chocolate, candies, and brown sugar flavors on the nose, along with smoky bowers & sounders, all wrapped tightly in a viscous, creamy feel. The palate is smooth and gentle, delivering gentle, medium-bodied whiskeys with their own distinctive flavors. A light, nimble palate serves enough sweetness without too much rancor that it becomes sultry, with hot chocolate, strawberry jam, milk chocolate, plum, and cassis.

Bloopers

My absolute favorite:

GPT-2 SAMPLE

A smallpox-infected grain, this first-generation proof has a floral, sweet nose laced with blueberry jam, cherry Cask Whisky, toffee, cedar, and tangy oak. It is fragrant and flavorful beyond any description—like a symphony show from before the age of antibiotics.

Whiskey, cigars, or promiscuous McDonald’s enthusiasts?

GPT-2 SAMPLE

Twip | 18k Batch | Filch | | $75

Makes a delicious mixing bowl of Chocolate Fudge, Big Mac, Ham, Sweet and Sour Apple Cider, Peanuts, and Sour Cream, with a fondness of making out with other people. The caramel is so strong, in fact, that I could only inhale the delicious caramel trails in my cigar box. The only hint I have of burnt sugar and caramel are lost along the way. Sweet and fruity, at least for the $75 price point.

Confusion between whiskey and… pop metal?

GPT-2 SAMPLE

the Chinese pop-metal king Ping’i Minbar began producing small batches for his mouth. Now in its 40s, with a nose of chocolate flan, hedonian, and bubble gum, it ends on sweet and savory rye, with vanilla, orange, and clove. It’s a reprise of the year loaded with floral notes, tangerine, cinnamon, black pepper, and orange soda pop. Floral, earthy, and smoky throughout, this period’s reasonably priced—$120—but well-balanced to give (or withhold) indulgence.

Peanut butter jelly time:

GPT-2 SAMPLE

61 points

Chicken Pot Pie with Peanuts, Hot Chocolate, and More Peanuts, 50/50

Single Malt Scotch Whisky | $35

Measurable, with portmoking salt water, toasted nuts, caramel, and spices in a thin blanket of peanut butter and jelly, but the panning process was not without its pique. The peanuts and the generous whiskily cooked nuts create an intriguing, if somewhat overcomplicated, vivacious ode to peanut butter and jelly, with sweet peanut butter and jelly, along the way.

What would this even taste like?

GPT-2 SAMPLE

Bourbon/Tennessee | $85

Tastes like a young William Shatner novel: rich with flavor, packed with fruit, honey, orange, maple syrup, and aromas of strawberry, licorice, and chocolate, balanced against a lingering tannin of the herbaceous nectar source. The palate is luxurious, richly notes a childhood home.

Note: I have kept the generated reviews true to their original formatting and content, with the exception of removing the “Reviewed By” line that WhiskyAdvocate have at the end of their reviews. The model very quickly learned to use the real names of the reviewers, and I didn’t want a real person being accused of describing a whiskey as “smallpox-infected”.

Thanks to WhiskyAdvocate for agreeing to let me use their reviews for this post!

Redirect standard out to Python’s logging module with contextlib

2019-05-22T17:00:00-01:00

Logging on Wikipedia

Python’s built in logging functionality is very nice once you get the hang of it, but many people (including library developers) either disagree or don’t bother to use it. Instead, you often see things like this:

>>> model.fit(X, y, verbose=True)
Epoch 0
Epoch 1
Epoch 2
...

and onwards for 100 lines or whatever.

Printing status messages to standard out is okay, but if you want anything like consistent/parseable logs, log level handling, logging to multiple locations, etc. this can get a bit annoying. How can we redirect standard out to Python’s logging system to get all these juicy benefits?

Another built in module, contextlib, comes to our rescue.

Aside: Context Managers

A context manager in Python is basically an object that implements two methods: __enter__ and __exit__. Usually you enter the context by using the with keyword, which triggers a call to __enter__. You execute some code in an indented block, and (roughly speaking) when you exit the block, the __exit__ method is called with information about any exceptions that occurred. The context manager interface is what makes these two pieces of code more or less equivalent:

h = open("file.txt")
text = h.read()
h.close()

is basically the same as

with open("file.txt") as h:
    text = h.read()

The file handle returned by open implements both __enter__ and __exit__, which is why it can be used in this way. The nice benefit here is that we never forget to close the file, as __exit__ (which closes the file in this case) is called automagically at the end of the block.

Redirecting `stdout`

So how does this help us? Well, contextlib (which is the library for things related to context managers, in case you didn’t catch that) has a very handy context manager called redirect_stdout (and also the matching redirect_stderr). We can use it to redirect text written to standard out to another file-like object. For example, to write to a file, we could do:

>>> import contextlib
>>> with open("output.txt", "w") as h, contextlib.redirect_stdout(h):
...     print("Hello world!")
...
>>>

We don’t see any output. If we now look at the newly-created output.txt, we will see

$ cat output.txt
Hello world!

I realize that there are better ways of writing to a file in Python, but writing to a file isn’t the point. The point is that any arbitrary code executed in the context block above would have its output redirected to the file, without having to modify its print statements, including library code that we can’t easily modify.

Now, redirect_stdout requires a file-like object that it can write to, which is why we first had to open our output file above. However, Python’s loggers are not file-like. I think you see where this is going.

A file-like logger

To redirect standard out to logging, we write a simple class that implements the write method, and passes everything that is written to the logger of our choice. We’ll also add a flush method that doesn’t do anything, just to avoid exceptions in case some code tries to use it. We can specify the desired logger by name since Python’s loggers are singletons.

import logging
import contextlib

class OutputLogger:
    def __init__(self, name="root", level="INFO"):
        self.logger = logging.getLogger(name)
        self.name = self.logger.name
        self.level = getattr(logging, level)

    def write(self, msg):
        if msg and not msg.isspace():
            self.logger.log(self.level, msg)

    def flush(self): pass

I added a quick check for empty messages since I don’t want blank lines in my logging.

This is already enough to pass our object to redirect_stdout and thereby redirect standard out to logging:

>>> logging.basicConfig(level=logging.INFO)
>>> with contextlib.redirect_stdout(OutputLogger("my_logger", "INFO")):
...     print("Hello logging!") 
...
INFO:my_logger:Hello logging!

Note that we had to minimally call logging.basicConfig() to get a bit of formatting on the logs and set the log level to at least the level we selected for our redirector (INFO). Since our class is just functioning as a redirector for messages, we leave ourselves free to configure the logger however we want elsewhere in the application (check out the Python logging cookbook for tips).

Baby’s first context manager

This implementation is already functional, but it’s a bit verbose. Since we’re already in the contextlib realm we might as well just make our object into a context manager itself, eliminating the need to call contextlib.redirect_stdout directly every time we want to use it. To do so, we add a new attribute called _redirector to our class, which is an instance of redirect_stdout with self as the redirect destination. Then our __enter__ and __exit__ methods just call the matching methods of our _redirector, ensuring that everything printed in our context will get redirected to our own write method (which in turn passes messages to our logger). Our implementation becomes:

import logging
import contextlib

class OutputLogger:
    def __init__(self, name="root", level="INFO"):
        self.logger = logging.getLogger(name)
        self.name = self.logger.name
        self.level = getattr(logging, level)
        self._redirector = contextlib.redirect_stdout(self)

    def write(self, msg):
        if msg and not msg.isspace():
            self.logger.log(self.level, msg)

    def flush(self): pass

    def __enter__(self):
        self._redirector.__enter__()
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        # let contextlib do any exception handling here
        self._redirector.__exit__(exc_type, exc_value, traceback)

Now we’re cooking! We can use OutputLogger as a context manager, and since we returned self from __enter__ we can even reuse the same instance later by giving it a name using the as keyword:

>>> print("Normal")
Normal
>>> 
>>> with OutputLogger("my_logger", "WARN") as redirector:
...     print("Logged!")
...
WARNING:my_logger:Logged!
>>> 
>>> print("Back to normal")
Back to normal
>>> 
>>> with redirector:
...     print("Logged again!")
...
WARNING:my_logger:Logged again!

There are all sorts of extensions possible here: redirecting standard error to another log level, making sure that changes to OutputLogger.name and OutputLogger.level get applied to the underlying logger properly, input checking on the log level string, etc. But this is enough to get started and will work as a quick and relatively clean way to capture the output from some other code and redirect it to your application’s logging system.

Disclaimer

Though contextlib.redirect_stdout is built in to Python, it does redefine sys.stdout for your whole process while the execution is within its context. For this reason, it can have unintended results for other pieces of code that are trying to do fancier stuff with sys.stdout than just write to it. This is a solution if you are writing an application that just needs to get something done, but if you are writing a library that other people might use, it’s best not do mess with these system properties without being very explicit about it. As always: Just because you can, doesn’t mean you should!

Cover image by Greenpeace Finland - originally posted to Flickr as Logging in Finnish Lapland: ancient trees for pulp and paper, CC BY 2.0, Link

Propagate time series events with Pandas merge_asof

2019-04-13T16:00:00-01:00

Imagine the following situation: you have two tables, one with event logs, and the other with status changes. To make this concrete, let’s take our events to be purchases by our users, and the status changes to be when the users attained a Gold Membership (wow so shiny). You want to use their membership status at the time of each purchase as a feature for some model, but you only have records of when the status changed, so you can’t just naively join the tables.

After inheriting some code that was performing this operation manually using groupby‘s and fancy indexing, I decided to check if Pandas had a built in function for it, and I was pleasantly surprised: pandas.merge_asof. Let’s check out how it works.

from datetime import datetime, timedelta
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# https://johnpaton.net/posts/custom-color-schemes-in-matplotlib/
plt.style.use('~/johnpaton.mplstyle')

We’ll begin by generating some fake sales data. We have three users, Alice, Bob, and Charlie, who made purchases over the last year. Sorry about the verbosity, there doesn’t seem to be any easier way to generate random dates in a range.

n_rows = 15
epoch = datetime.utcfromtimestamp(0)
end = (datetime.now() - epoch).days
start = end - 365

users = ['alice','bob','charlie']
df_sales = pd.DataFrame(data={
    'timestamp': pd.to_datetime(np.random.randint(start, end, n_rows), unit='D'),
    'user': np.random.choice(users, n_rows),
    'amount': np.random.randint(0,100, n_rows)
}).sort_values('timestamp').reset_index(drop=True)
df_sales

	timestamp	user	amount
0	2018-04-14	charlie	74
1	2018-05-02	alice	61
2	2018-05-17	charlie	85
3	2018-06-25	bob	71
4	2018-06-30	alice	50
5	2018-07-04	charlie	40
6	2018-09-22	bob	64
7	2018-11-02	alice	7
8	2018-11-18	bob	57
9	2018-11-21	alice	37
10	2018-11-27	charlie	42
11	2019-01-19	alice	29
12	2019-01-22	alice	46
13	2019-03-17	bob	11
14	2019-04-02	alice	77

Next up, we need the records of when each of them attained their coveted Gold Membership:

df_flags = pd.DataFrame(data={
    'user': users,
    'timestamp': pd.to_datetime(np.random.randint(start, end, len(users)), unit='D'),
    'status': 'gold_member'
}).sort_values('timestamp').reset_index(drop=True)
df_flags

	user	timestamp	status
0	charlie	2018-04-18	gold_member
1	bob	2018-06-09	gold_member
2	alice	2018-09-24	gold_member

Now it’s time to do our merge. We call pd.merge_asof, specifying the following arguments:

The left and right DataFrames
The column to merge on (this is generally a time column or some other ordering field)
The column(s) to group by. The by keyword is very nice, as we can use it to make sure that the status changes only get propagated across the correct users’ purchases. Otherwise we would have to ensure this manually.

df_merged = pd.merge_asof(
    df_sales,
    df_flags,
    on='timestamp',
    by='user',
)
df_merged

	timestamp	user	amount	status
0	2018-04-14	charlie	74	NaN
1	2018-05-02	alice	61	NaN
2	2018-05-17	charlie	85	gold_member
3	2018-06-25	bob	71	gold_member
4	2018-06-30	alice	50	NaN
5	2018-07-04	charlie	40	gold_member
6	2018-09-22	bob	64	gold_member
7	2018-11-02	alice	7	gold_member
8	2018-11-18	bob	57	gold_member
9	2018-11-21	alice	37	gold_member
10	2018-11-27	charlie	42	gold_member
11	2019-01-19	alice	29	gold_member
12	2019-01-22	alice	46	gold_member
13	2019-03-17	bob	11	gold_member
14	2019-04-02	alice	77	gold_member

The result is kind of like a left join, in that all “matching” rows are filled and unmatched rows are NaN. However, unlike in a left join (which looks for an exact match), in this case the join applied to all rows on the left that have a timestamp that comes after the paired rows on the right. To ensure that this before/after comparison is possible, the DataFrames must be sorted by the on column.

To create our Gold Membership flag, all we do is just replace the name of the event with a 1 and fill the NaNs (which are rows that came before the status change) with 0s.

df_merged['gold_member'] = df_merged['status']\
    .replace('gold_member', 1)\
    .fillna(0)\
    .astype(int)
df_merged

	timestamp	user	amount	status	gold_member
0	2018-04-14	charlie	74	NaN	0
1	2018-05-02	alice	61	NaN	0
2	2018-05-17	charlie	85	gold_member	1
3	2018-06-25	bob	71	gold_member	1
4	2018-06-30	alice	50	NaN	0
5	2018-07-04	charlie	40	gold_member	1
6	2018-09-22	bob	64	gold_member	1
7	2018-11-02	alice	7	gold_member	1
8	2018-11-18	bob	57	gold_member	1
9	2018-11-21	alice	37	gold_member	1
10	2018-11-27	charlie	42	gold_member	1
11	2019-01-19	alice	29	gold_member	1
12	2019-01-22	alice	46	gold_member	1
13	2019-03-17	bob	11	gold_member	1
14	2019-04-02	alice	77	gold_member	1

To make sure there’s no funny business going on, we can also visualize exactly what happened:

fig, ax = plt.subplots(facecolor='white', figsize=(12,5))
for user in users:
    df_tmp = df_merged[df_merged['user'] == user]
    plt.scatter(df_tmp['timestamp'], df_tmp['gold_member'], label=user)

for i,row in df_flags.sort_values('user').reset_index().iterrows():
    plt.axvline(row['timestamp'], c=f'C{i}')
    plt.annotate(
        f" {row['user']} gets gold", (row['timestamp'], np.random.uniform(0.25,0.75))
    )
plt.yticks([0,1], ['regular', 'gold'])
plt.ylabel('Membership status')
plt.xlabel('Sale date')
plt.title('Sales by membership status')
plt.legend();

So now we have a nice feature that can be used in a training set, without any data leakage (no events from the future are visible before they happened in the training set).

This also works for multiple status changes. For example, say Bob got demoted on February 1st back to being a normal user.

df_flags = pd.concat([
    df_flags,
    pd.DataFrame(data={
        'user': ['bob'],
        'status': 'normal',
        'timestamp': pd.to_datetime('2019-02-01')
    })
])
df_flags

	status	timestamp	user
0	gold_member	2018-04-18	charlie
1	gold_member	2018-06-09	bob
2	gold_member	2018-09-24	alice
0	normal	2019-02-01	bob

If we perform the merge_asof again, we see that Bob’s status changes twice, just how we would expect:

pd.merge_asof(
    df_sales,
    df_flags,
    on='timestamp',
    by='user',
)

	timestamp	user	amount	status
0	2018-04-14	charlie	74	NaN
1	2018-05-02	alice	61	NaN
2	2018-05-17	charlie	85	gold_member
3	2018-06-25	bob	71	gold_member
4	2018-06-30	alice	50	NaN
5	2018-07-04	charlie	40	gold_member
6	2018-09-22	bob	64	gold_member
7	2018-11-02	alice	7	gold_member
8	2018-11-18	bob	57	gold_member
9	2018-11-21	alice	37	gold_member
10	2018-11-27	charlie	42	gold_member
11	2019-01-19	alice	29	gold_member
12	2019-01-22	alice	46	gold_member
13	2019-03-17	bob	11	normal
14	2019-04-02	alice	77	gold_member

Finally, these have all been examples of so-called “backwards searches”. From the Pandas docs:

A “backward” search selects the last row in the right DataFrame whose on key is less than or equal to the left’s key.

Somewhat counter-intuitively, this causes the status changes to propagate forward in time. This is the default behavior of merge_asof. To do the reverse (a forward search, a.k.a. propagate changes backwards in time), you can provide the keyword argument direction="forward".

Getting Calvin home on time: a statistics puzzle

2018-07-19T17:00:00-01:00

I found this puzzle a while ago and couldn’t get it out of my head, so I decided to write up a solution.

Calvin has to cross several signals when he walks from his home to school. Each of these signals operate independently. They alternate every 80 seconds between green light and red light. At each signal, there is a counter display that tells him how long it will be before the current signal light changes. Calvin has a magic wand which lets him turn a signal from red to green instantaneously. However, this wand comes with limited battery life, so he can use it only for a specified number of times.

a) If the total number of signals is 2 and Calvin can use his magic wand only once, then what is the expected waiting time at the signals when Calvin optimally walks from his home to school?

b) What if the number of signals is 3 and Calvin can use his magic wand only once?

c) Write a program that takes as inputs the number of signals and the number of times Calvin can use his magic wand, and outputs the expected waiting time

Assuming that the lights are independent, the time until a light changes (displayed on the counter) when Calvin arrives is drawn from a uniform distribution between 0 and 80:

$$ t_{\text{counter}} \sim \text{Unif}(0\ \text{s},80\ \text{s}).$$

So the average waiting time at each red light is 40 seconds. Since there is a 50% chance that the light is green, the expected waiting per light time is

$$ \bar t_{\text{wait}} = P(\text{green}) \cdot \bar t_{\text{counter}} = \frac{1}{2} \cdot 40\ \text{s} = 20\ \text{s}. $$

Calvin wants to use his wand in order to minimize his waiting time.

a) If the total number of signals is 2 and Calvin can use his magic wand only once, then what is the expected waiting time at the signals when Calvin optimally walks from his home to school?

With two lights and one use of the magic wand, there are three scenarios that can arise:

If the first light is green (50% chance), then Calvin will use his wand on the second light, with 0 waiting time.
If the first light is red (50% chance):
- He will use his wand if the time on the timer is higher than the expected wait for the second light (20 seconds, 75% chance). In this case the expected waiting time is the expected time for the second light: 20 seconds.
- He will not use his wand if the time on the timer is less than 20 seconds (the mean time these cases is 10 seconds since they are uniformly distributed between 0 and 20, and the situation occurs with a 25% chance). In that case he will wait, and use his wand on the second light, for a total expected waiting time of 10 seconds.

His total expected waiting time is found by taking a sum of all the different situations weighted by their probability:

$$ \bar t_{wait} = 0.5 \cdot 0\ \text{s} + 0.5\cdot(0.75 \cdot 20\ \text{s} + 0.25 \cdot 10\ \text{s}) = 8.75\ \text{s}$$

So Calvin’s expected waiting time is 8.75 seconds.

Let’s confirm this by trying it… one billion times [evil laugh]:

import numpy as np

n_trials = int(1e9)

# initialize the trials
wait_times = np.zeros(n_trials)

# light waiting times: uniform 0-80 seconds, 50% chance of being green (0 wait time)
t1 = np.random.uniform(0, 80, n_trials) * np.random.randint(0, 2, n_trials)
t2 = np.random.uniform(0, 80, n_trials) * np.random.randint(0, 2, n_trials)

wand1 = (t1 >= 20) # trials where Calvin uses the wand on light 1

# wait at light 2 wherever he uses the wand at light 1
wait_times[wand1] = t2[wand1] 
# wait at light 1 wherever he doesn't use the wand at light 1
wait_times[~wand1] = t1[~wand1]

print('Calvin\'s mean waiting time in {:,} trials was {:.3f} seconds'\
      .format(n_trials, wait_times.mean()))

Calvin's mean waiting time in 1,000,000,000 trials was 8.750 seconds

Nice!

b) What if the number of signals is 3 and Calvin can use his magic wand only once?

Now we have three signals and one wand usage. Since we have one wand, we want to find some optimal cutoff value that lets us decide whether or not to use the wand at each light. We base this on whether or not we expect to see a light with a higher waiting time than the current counter during the rest of our walk home.

At the last light, the cutoff is zero. If we haven’t used the wand yet, we will definitely use it if the light is not green, as there are no more chances left to use it.

We’ve already shown that the cutoff at the second-to-last light should be 20 seconds, as this situation is the same one as in part a) (assuming the wand hasn’t been used yet).

What about at the first light? We want to know the expectation value of the maximum of the next two waiting times.

The probability that the counter displays a waiting time is less than some time $t$ is given by

$$ \tilde{F}(t) = P(t_{\text{count}} \le t) = \frac{t}{80}. $$

$\tilde{F}(t)$ here is the Cumulative Density Function (CDF) for our uniformly distributed counter time.

Now, there is a 50% chance that the light will be green, in which case our waiting time is zero, regardless of the value displayed on the counter. This means that the probability of the waiting time being less than or equal to zero is already $1/2$, and it grows linearly to $1$ for a waiting time up to $80$ seconds. So including the chance of a green light, the CDF for our waiting time is

$$ F(t) = P(t_{\text{wait}} \le t) = \frac{1}{2} + \frac{1}{2}\frac{t}{80} .$$

We are interested in the maximum expected waiting time for the last 2 lights in Calvin’s journey, so that he knows what his cutoff should be at the first light. What is the relevant CDF? If the maximum of two random variables is less than a time $t$, then each of the variables must be less than $t$ individually. Since we are assuming that the waiting times are independent, their probabilities factor, and we get the CDF

$$ F_2(t_\max) = P(t_{\text{wait},1},t_{\text{wait},2} \le t_\max) = P(t_{\text{wait},1}\le t_\max)P(t_{\text{wait},2} \le t_\max) = F(t_\max)^2. $$

Clearly this generalizes to $n$ remaining lights, so let’s consider the more general case and then specify $n=2$ to get the cutoff we want. We have

$$ F_n(t_\max) = F(t_\max)^n = \left( \frac{1}{2} + \frac{1}{2}\frac{t_\max}{80} \right)^n. $$

We are looking for the expected value of this maximum. The expectation can found directly from the CDF by

$$ E_n = 80 - \int_0^{80}\!\! \text{d}t\ F_n(t) = 80 - \int_0^{80}\text{d}t \left( \frac{1}{2} + \frac{1}{2}\frac{t}{80} \right)^n$$

(this can be seen from integrating the normal definition, $E = \int \text{d}x\ x\ \text{PDF}(x)$, by parts). To take the integral, we perform the substitution $u = 1/2 + t/(2\cdot80)$ and get

\begin{align*} \int_0^{80}\text{d}t \left( \frac{1}{2} + \frac{1}{2}\frac{t}{80} \right)^n &= 80 \cdot 2 \int_{\frac{1}{2}}^1 \text{d}u\ u^n \\ &= 80 \cdot 2 \left. \frac{u^{n+1}}{n+1} \right|_{\frac{1}{2}}^1\\ &= 80 \cdot \frac{2}{n+1}\left(1-\frac{1}{2^{n+1}}\right). \end{align*}

Inserting this result into our expression for the expected maximum, we get

$$ E_n = 80 \cdot \left(1 - \frac{2}{n+1}\left(1-\frac{1}{2^{n+1}}\right)\right) = 80 \cdot \left(\frac{n}{n+1} - \frac{1}{n+1}\left(1-\frac{1}{2^{n+1}}\right)\right). $$

Interestingly, in the second form above, $80\cdot n/(n+1)$ is the expected maximum waiting time if we had to wait at every light, and the second term is a correction due to half of the lights being green. Again we can confirm our calculation by simulation:

import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')

# simulate (up to) 25 lights
n_lights = 20
n_trials = int(1e7)
counter_max = 80

# simulate the lights (one row per light, one column per trial)
trials =  np.random.uniform(0, counter_max, (n_lights, n_trials,))\
            * np.random.randint(0, 2, (n_lights, n_trials,))

# get the max waiting times for each possible number of lights
t_max = np.zeros(trials.shape)
for i in range(n_lights):
    t_max[i, :] = trials[:i+1, :].max(0)

# get the calculated expectation and simulated mean of the max waiting times
n = np.arange(1, n_lights+1)
wait_max_exp = counter_max * (1 - (2/(n+1))*(1 - 1/(2**(n+1))))
wait_max_sim = t_max.mean(1)

# plot to compare
plt.figure(figsize=(12, 5))
plt.scatter(n, wait_max_exp, s=65, marker='x', c='b', label='expectation')
plt.scatter(n, wait_max_sim, s=35, alpha=0.7, c='r',
            label='simulation ({:,} trials)'.format(n_trials))
plt.xticks(n); plt.xlabel('Number of lights')
plt.ylabel('Max waiting time (seconds)')
plt.xlim([0, n_lights+1]); plt.ylim([0, counter_max])
plt.title('Maximum expected waiting time in sample of $n$ lights')
plt.legend();

So we have values to function as our cutoffs, based on how many lights are remaining in the journey. Specifically, the cutoff for the first of 3 lights is given by the maximum expected wait time for the remaining $n=2$ lights:

print('Max expected waiting time for 2 remaining lights is {:.2f} seconds.'\
      .format(wait_max_exp[1]))

Max expected waiting time for 2 remaining lights is 33.33 seconds.

Calvin should use his wand if the first light is red and the counter is over 33.33 seconds. We can calculate his expected waiting time by using the result from a) and considering the possible scenarios:

The first light is green (50% chance). In this case Calvin has two lights and one wand to go; this is the situation from a). In this scenario the expected waiting time is 8.75 seconds.
The first light is red (50% chance):
- The timer is over the cutoff ( $(80-33.33)/80 = 58$% chance for uniformly distributed time on the counter). In this case Calvin uses the wand, and with no wands left he expects to wait for 20 seconds per light for the remaining number of lights. There are two lights left, so the expected wait time in this case is 40 seconds.
- The timer is under the cutoff (42% chance). In this case Calvin waits for the displayed time (expectation $33.33/2 = 16.67$ seconds) and then moves on to the situation in a) for a total expected waiting time of $16.67 + 8.75 = 25.42$ seconds).

Again we find his total expected waiting time by taking the weighted sum:

$$ \bar t_{\text{wait}} \approx 0.5 \cdot 8.75\ \text{s} + 0.5 \cdot \left(0.58\cdot 40\ \text{s} + 0.42 \cdot 25.42\ \text{s} \right) \approx 21.3\ \text{s}$$

Note that this by-hand calculation has some rounding errors in the cutoff and proportions, but the exact value also rounds to 21.3 seconds.

c) Write a program that takes as inputs the number of signals and the number of times Calvin can use his magic wand, and outputs the expected waiting time

Now we want full generalization to any number of lights and wand uses. It should be clear from the reliance of the calculation in b) on the results from a) that we can use a recursive strategy to calculate the total waiting time.

There are two questions to answer: What should the cutoff be for multiple uses of the wand? And what is the recursive pattern for calculating the expected wait time?

Let’s begin with the cutoff. When Calvin comes to a red light, he must decide whether to use the wand or not. For a walk with $n$ lights, our strategy has been to consider the expected maximum wait encountered during the next $n-1$ lights. We can generalize by considering the expected maximum wait at the number of remaining lights we expect to wait at, including the current one. If Calvin has $w$ uses of his wand, then he can skip up to $w$ lights, meaning he has to wait at $n-w$ lights in total, including the current one. So the cutoff for using the wand can be generalized to

$$ E_{\tilde{n}} = 80 \cdot \left(1 - \frac{2}{\tilde{n}+1}\left(1-\frac{1}{2^{\tilde{n}+1}}\right)\right),\ \tilde{n} = n - w. $$

This new cutoff is smaller than the previous one when $w>1$, which makes sense since we would then like to be more liberal with our wand usage.

Now that we have the cutoff, we need the recursive pattern. Again we have three scenarios. If we have $n$ lights and $w$ wands to go, then the scenarios are:

The current light is green (50% chance). In this case the expected waiting time is just the waiting time for $n-1$ lights and $w$ wand uses.
The current light is red (50% chance):
- The waiting time is below the cutoff (probability $\text{cutoff} / 80$). In this case Calvin doesn’t use the wand, and the total waiting time is the wait at the current light (expectation $\text{cutoff}/2$) plus the expected wait at $n-1$ lights with $w$ wands.
- The waiting time is above the cutoff (probability $1 - \text{cutoff} / 80$). Now Calvin uses the wand, adding no extra waiting time, so the expected total wait is the expected wait for $n-1$ lights and $w-1$ wands.

We calculate these scenarios, recursing whenever we need a new total expected wait time for a different number of lights and/or wands, until we hit one of the base cases. There are a few:

If we have 0 lights to go, then the expected total remaining wait time is 0 seconds.
If the number of wands is greater than or equal to the number of lights, then the wait time is 0 seconds (since we can switch all the lights off).
If we have no wands left, then the wait time is just the expected wait per light (20 seconds) times the number of lights remaining.

We can write a function to do this calculation:

def wait_time_exp(n_lights, n_wands):
    """Calculate Calvin's total expected wait time on his journey home.

    Args:
        n_lights (int): The number of lights on the journey
        n_wands (int): The number of times Calvin can use his wand

    Returns:
        float: The total expected time spent waiting at lights
    """
    # maximum time per timer
    counter_max = 80

    #### Base cases ####
    # no lights left
    if n_lights <= 0:
        return 0
    # enough wands to switch all lights
    elif n_wands >= n_lights:
        return 0
    # no wands left
    elif n_wands <= 0:
        return n_lights * counter_max / 4

    #### Recursion ####
    # cutoff for the number of lights we expect to wait at
    n = n_lights-n_wands
    cutoff = counter_max * (1 - (2/(n+1))*(1 - 1/(2**(n+1))))

    # initialize 3 scenarios
    probs = np.zeros(3)
    waits = np.zeros(3)

    # scenario 0: green light
    probs[0] = 0.5
    waits[0] = wait_time_exp(n_lights-1, n_wands)

    # scenario 1: red light + wait
    probs[1] = 0.5*cutoff/counter_max
    waits[1] = cutoff/2 + waits[0]

    # scenario 2: red light + use a wand
    probs[2] = 0.5*(1 - cutoff/counter_max)
    waits[2] = wait_time_exp(n_lights-1, n_wands-1)

    # return weighted sum of scenarios
    return (probs*waits).sum()

And finally we finish off with some art to confirm that the function is behaving as we expect it to.

# set up variables
n_lights = 20
n_wands = 20

lights = np.arange(n_lights+1)
wands = np.arange(n_wands+1)

# initialize waiting times
times = np.zeros((n_lights+1, n_wands+1,))

# calculate waiting times
for n in lights:
    for w in wands:
        times[n,w] = wait_time_exp(n,w)

# plot the waiting times as a heatmap
plt.figure(figsize=(10, 8,)) 
plt.pcolor(times.T, cmap='jet', linewidth=0.25, edgecolor='w')
plt.title('Total expected waiting time on Calvin\'s walk home')
plt.xlabel('Number of lights'); plt.ylabel('Number of wand uses')
plt.xticks(lights+0.5, lights); plt.yticks(wands+0.5, wands)
plt.colorbar(label='Time (seconds)');

With the problem thoroughly solved, I can finally sleep at night again.

Remapping the world with word vectors

2017-12-02T19:34:00-01:00

Everyone is used to the map. Most people could make a reasonable attempt at drawing one from memory (well, sort of). But what would it look like if we positioned the countries not by geographical location, but by our own perceived relationships between them? Armed with Conceptnet Numberbatch, I decided to try just that.

Contents:

Word Vectors
Conceptnet Numberbatch
Countries of the world
Countries as concept vectors
Remapping the world
Mapping our bias?

Word Vectors

In natural language processing, a big issue is the dimensionality of the data. Machine learning algorithms almost exclusively deal with vectors of numbers, whereas the atomic unit of natural language is the word. The natural way to encode words as feature vectors is to do what’s called one-hot encoding: make a vector with only one element 1, and the rest 0, where the position of the 1 is determined by the word. This gives us a vector as long as the dictionary, with each entry representing one word, and there is a 1 at the position of the word we are trying to represent.

There are a few problems with this approach. For one thing, these vectors are very long. Large bodies of English text contain hundreds of thousands of unique words, so each vector has to have hundreds of thousands of entries. However, a more important issue is that these vectors don’t encode any meaning. Each vector has magnitude 1, and they are all perpendicular to each other, so we lose all the relationships that we intuitively know words carry.

Word vectors are an attempt at a solution to both of these problems. The idea is to use an algorithm to learn a set of numbers that are unique to each word, and such that “similar” words have vectors that are “close to” each other. Here we consider words to be similar if they are often used in the same context, and we compare vectors by the angle between them (cosine similarity). There are various implementations of this idea, including Stanford’s GloVe, two flavours of Google’s word2vec, and Facebook’s fastText. These approaches are very successful in their aims, even learning to encode abstract relationships between words, as demonstrated by this 2d projection of the GloVe vectors:

It’s worth mentioning that these vectors are the result of unsupervised learning: nobody told the algorithms that loud should be to loudest as strong is to strongest. The relationships are learned purely from the contexts that the words are used in.

There are pre-trained vectors from all these implementations available online to download, and if you have a lot of data it’s also possible to train your own. If you do decide to make your own word vectors, make sure you fit in with the cool kids by giving them a name with weird capitalization.

Conceptnet Numberbatch

The popular word vector sets are mostly just that: word vectors. But what about singular concepts whose names happen to contain a space? If we split our text into individual words, it is difficult to reconstruct these concepts. The team at ConceptNet have done just that, combining several existing word vector sets and augmenting them with their own semantic network (a mapping of how concepts relate to each other). They also attempt to purposely de-bias their vectors so as not to encode relationships that are racist, sexist, etc. Their “concept vectors” are humorously called Conceptnet Numberbatch. For this post I used the English-only subset, which contains 416,410 vectors.

By using the Conceptnet Numberbatch vectors we can compare not just words, but also concepts with each other for similarity. Of course, some words have a double meaning, and one similarity measure will never capture this. As an example, I sorted the concept vectors for similarity to “orange”. The top few are mostly various subtypes of the citrus fruit, but at number 9 we get “yellowred”, which is clearly related to the color:

Top 20 most similar concepts to "orange":
   1. orange_tree          Similarity: 0.96476
   2. orange_river         Similarity: 0.94728
   3. sweet_orange         Similarity: 0.93845
   4. citrus_sinensis      Similarity: 0.93466
   5. valencia_orange      Similarity: 0.92221
   6. sour_orange          Similarity: 0.91260
   7. citrus_aurantium     Similarity: 0.88945
   8. bitter_orange        Similarity: 0.88880
   9. yellowred            Similarity: 0.86298
  10. seville_orange       Similarity: 0.85275
  11. temple_orange        Similarity: 0.85226
  12. bigarade             Similarity: 0.84110
  13. iyokan               Similarity: 0.83425
  14. orangish             Similarity: 0.83299
  15. secondary_colour     Similarity: 0.82235
  16. tropaeolin           Similarity: 0.79860
  17. orangelo             Similarity: 0.79785
  18. orange_coloured      Similarity: 0.77347
  19. tangor               Similarity: 0.77011
  20. orangequat           Similarity: 0.76962

There are ways around this issue too, for example by including word senses in the vectors, but this is overkill for our purposes.

Curious about my homeland, I tried out the same thing with “Canada”. Hilariously, both “America’s hat” and “Soviet Canuckistan” make the top 20:

Top 20 most similar concepts to "canada":
   1. canada_west          Similarity: 0.97635
   2. great_white_north    Similarity: 0.96604
   3. laurentian_highlands Similarity: 0.96598
   4. arctic_archipelago   Similarity: 0.96598
   5. hydrofield           Similarity: 0.96262
   6. ookpik               Similarity: 0.96262
   7. canada_east          Similarity: 0.96223
   8. central_provinces    Similarity: 0.95273
   9. canadianist          Similarity: 0.94114
  10. central_canada       Similarity: 0.94038
  11. upper_canada         Similarity: 0.93867
  12. lower_canada         Similarity: 0.93571
  13. canadian_shield      Similarity: 0.93504
  14. canadas              Similarity: 0.92199
  15. america's_hat        Similarity: 0.91242
  16. hudson_bay           Similarity: 0.91084
  17. canadian_studies     Similarity: 0.90770
  18. in_north_america     Similarity: 0.90715
  19. soviet_canuckistan   Similarity: 0.90083
  20. eastern_canada       Similarity: 0.90079

Note that the average similarity score in Canada’s top 20 is much higher than in orange’s. This is likely because “orange” is used in a more wide variety of contexts than “Canada”, which dilutes its similarity to any given concept.

Countries of the world

It turns out that getting standardized data about the countries of the world is surprisingly tricky. There are various definitions of attributes, character encoding issues, countries not recognizing each other, etc. Luckily, Mohammed Le Doze has painstakingly compiled a high quality set of country data in various formats, allowing me to offload responsibility for any political sensitivities.

As a starting point, we can plot the locations of the countries as we expect to see them. The positions are given in latitude and longitude, but through the magic of map projections (conveniently provided in this code snippet from OpenStreetMap), we can plot the positions on a flat plane. I creatively decided to represent the countries as blobs with size approximately related to the land area of the country, and immediately gained respect for people who are actually good at making maps:

Nailed it.

Countries as concept vectors

The countries dataset provides us with a “common English name” per country. We use this name to associate a concept vector to each country in the set, to the extent possible (some countries, such as French Southern and Antarctic Islands, and British Indian Ocean Territory, were not in Conceptnet Numberbatch. Sorry about that.).

Considering countries as concepts with associated vectors reveals some interesting structure. Using hierarchical clustering, we can sort the countries such that more similar countries are next to each other. Plotting the similarities as a heatmap visualizes the clusters of countries that have especially high similarity to each other (and it looks really cool):

I realize this is big and hard to read; you can click on the image to get the full sized one where the country names are actually legible.

The squares along the diagonal are the main clusters of similar countries which were identified by the clustering algorithm. I’ve tried to give semi-appropriate names to a few of them. We see for example the Arabian Peninsula (Saudi Arabia, Kuwait, Qatar, the UAE, Bahrain, and Oman), groups of nearby islands, and the Group Which Shall Not Be Named (Iraq, Jordan, Israel, Palestine, Lebanon, Iran, Syria, Egypt, and Libya). Thanks to our use of hierarchical clustering, we also see groups with substructure, like the Germanic countries, which visibly include the Nordic countries and BeNeLux.

On the off-diagonal we also see some structure. These are mostly primary clusters which have strong inter-group relationships. I’ve labeled three such groups: the Pacific and Caribbean Islands are quite strongly associated with each other, Spain/Portugal are similar to their (mainly South American) colonies, and many of the Arabic-speaking countries are also associated with each other.

It’s interesting to note that the primary clusters seem to be mostly determined by geographical proximity, while the off-diagonal groups seem to be more influenced by culture/language.

Because the entire world is a bit hard to take in all at once, we will zoom in on Europe for a little bit. We end up with:

Now it seems that the Germanic world is much less pronounced, the Åland Islands have been split out from the rest of the Nordics, and the Channel Islands are basically separated entirely from the rest. Sorry about that, island nations.

Remapping the world

While the heatmaps are interesting, we started out on a mission to remap the world, and that’s what we’re going to do. To figure out the new placement of the countries, we visualize them as a network graph, simulated with d3.js. The nodes are our country blobs, and they will be connected by springy edges. The edges act with a force on the nodes, pulling more similar countries closer together, and pushing dissimilar ones apart. By initializing the simulation with the countries in their original position as plotted on the map above, we hope to end up with a layout that reflects geography as much as possible, so that we’re sure any major changes are the result of taking into account the concept vector similarity. We’ll begin with Europe, and connect each country to its top 10 most similar neighbors (to keep the complexity of the simulation down). Thanks to the magic of d3, we can make the map interactive, so if you aren’t happy with where your country ended up, you can try to drag it somewhere else and see which other countries get pulled along too.

Tip: If you’re on mobile, this looks nicer in landscape mode!

Now we’re talking! The simulation isn’t totally deterministic, so you might see something different, but when I run it I see the Faroes and Iceland get pulled into the interior of Northern Europe thanks to strong connections with the (rest of the) Nordics. The UK moves even further outside mainland Europe. France gains Ireland as a neighbour, but not because of any particularly strong relationship. Rather, Ireland has connections to fellow island nations Iceland (North), Malta (South), and the UK (West), so in balancing these it just happens to end up near France. The Vatican is ejected from mainland Europe due to its surprisingly weak connection to Italy. The Baltics move southeast to be closer to the former Yugoslav countries, which also causes Poland to be pulled south of Austria.

It’s extremely addicting to play with this simulation and imagine the alternate histories that could have resulted. The Vatican Armada? The Polish-Faroese Empire? Delicious Irish cuisine? We will never know.

Just like with the heatmaps, the network graph for the entire world is a bit overwhelming. We simplify the simulation further by connecting each country to only its top 3 neighbors, and make it a bit taller to give them more room to spread out (feel free to drag them apart yourself to get an even better view):

Dragging the countries around gives a good feel for who is connected to who. It’s particularly interesting when countries from different continents are closely connected. Portugal and Spain are pulled into the midst of their South American colonies, which form an unusually rigid block. The US finds an unlikely neighbor in Georgia (undoubtedly caused by the US state of the same name). The Anglosphere makes its first appearance, linking the US, Canada, the UK, Australia, and New Zealand (and, dare I say, Ireland). Turkey and Central Asia are disconnected from the rest of Asia, preferring the company of Russia. Canada and the US decouple from the rest of the Americas. Chad is kept at a distance from the rest of Africa, probably because of its common usage as an English name diluting the strength of its connections (the “orange” effect), an issue which also plagues Réunion. The Falklands gain great strategic importance, forming a sole link between the Pacific Islands and the Caribbean.

Surprisingly, the entire world seems to remain connected (even if the connecting edges are so weak as to not be visible). Our simulation of so few connections made it possible for 3 or 4 countries to form an exclusive block, disconnected from the rest of the world, but as far as I can tell this did not happen.

Mapping our bias?

We have spent quite some time considering the new (and old) groupings that show up when we consider the countries as concept vectors. However, it’s also important to consider what is not present in the data, and how this may reveal underlying confounding factors.

Although a loosely-grouped Anglosphere formed in the world network graph, the relative strengths of the connections were largely weak enough to not show up in the clustering we performed to create the heat map. Conversely, Africa appeared in the heatmap as one giant block, despite great diversity across the continent (Somalia, Tunisia, and Zimbabwe in the same group?).

My hypothesis is that this is a result of using only the English subset of Conceptnet Numberbatch’s concept vectors. We humans have a tendency to see differences between members of our “own group” (whatever that means), while seeing other groups as one cohesive unit. Wikipedia tells me this concept is known as out-group homogeneity, which can lead to stereotyping. In the English concept vectors (which were trained on English text gathered from the Internet), the Anglosphere countries aren’t frequently used in the exact same context, because they are more often perceived as the in-group by people who are speaking English. This (and not Brexit) would also explain the UK being pushed out of Europe and relatively far from Ireland in the Europe network graph. All of this happens in spite of ConceptNet actively working to remove bias from their vectors, showing just how tricky this issue really is.

It would be interesting to confirm this hypothesis by trying this experiment out with other language vectors, but this post is already long enough, so I will leave that for future exploration. For now, suffice it to say that unexamined factors in your data may lead to unexpected results. If you find yourself making sweeping generalizations or casually reporting on “Soviet Canuckistan”, it may be time to reevaluate your assumptions.

Did you find any other interesting (missing) connections? Could I have improved the graphs (this is my first forray into d3)? Let me know below!

Cleaner Spark UDF definitions with a little decorator

2017-11-16T19:48:11-01:00

Update: It turns out the functionality described here is actually standard, and I just recreated an existing feature! Embarrassing. This is why you always read the docs! Thanks to Enrico Rotundo for pointing this out. Nonetheless, knowing how to define your own decorators is useful if you want to e.g. propagate the docstring using functools.wraps, so I’ll leave this here for further exploration.

Alternate title: Clean up your Spark jobs with this one weird trick! Apache will hate you!

One of the handy features that makes (Py)Spark more flexible than database tools like Hive even for just transforming tabular data is the ease of creating User Defined Functions (UDFs). Although this is also possible in Hive directly, the ability to define and call UDFs directly in the Python code of your job makes life a lot easier and provides context to what you’re doing. However, one thing that still remains a little annoying is separately defining a Python function and then having to declare it as a Spark UDF.

Consider the trivial example of incrementing all the values in a Spark DataFrame column by 1. We begin by writing the function, and then make a “UDF-ified” version that we can actually use in Spark.

import pyspark.sql.functions as f

# define the function
def increment(x):
    return x + 1

# make the udf version
increment_udf = f.udf(increment)

By registering our function as a UDF, we tell Spark that this function should be applied to every value in whatever DataFrame column it is applied to, and Spark takes care of distributing the execution across the cluster when we submit our job. We can now use our newly declared increment_udf to increment all the values in a column:

# increment column 'col' by 1
df = df.withColumn('col_plus_1', increment_udf('col'))

“Hold up,” you say, “you’re making it unnecessarily difficult for yourself. Just use the f.udf as a decorator!” That is indeed an attractive option. The code then condenses to:

@f.udf
def increment(x):
    return x + 1

df = df.withColumn('col_plus_1', increment('col'))

This is a lot better looking, but it comes at the cost of flexibility. The function f.udf optionally takes as a second argument the type of the UDF’s output (in terms of the pyspark.sql.types types). Spark will by default convert UDF outputs to strings, which can be a hassle, especially for complex data types (like arrays), or when the precision is important (float vs. double). To avoid this stringy fate, we have to return to our old pattern:

import pyspark.sql.types as t

# define the function
def increment(x):
    return x + 1

# make a typed udf version
increment_udf = f.udf(increment, t.IntegerType())

To get back to our nice, clean decorator syntax, we can write a tiny but useful function to generate a UDF decorator that will cast the output to the appropriate type:

def typed_udf(return_type):
    def _typed_udf_wrapper(func):
        return f.udf(func, return_type)
    return _typed_udf_wrapper

The function typed_udf returns a new UDF decorator with the specified return type. We can think of it as a decorator that accepts an argument. For a more in depth overview of this pattern and decorators in general, see this blog post from The Code Ship.

Now we once again have the nice, clean version of the code, with the added legibility bonus of the UDF’s return type being visible right beside its definition:

@typed_udf(t.IntegerType())
def increment(x):
    return x + 1

df = df.withColumn('col_plus_1', increment('col'))

Using a decorator instead of making two versions of every function isn’t really necessary in this simple example, but if you are defining 20 UDFs your namespace will get awfully cluttered and it’ll become harder to track what’s going on. With four lines of code you can bring sanity back to your function naming scheme.

Finally, the code for DIY decorators is notoriously difficult to read, so if you’re going to copy-paste, here is the snippet with a docstring:

import pyspark.sql.functions as f

def typed_udf(return_type):
    """Make a UDF decorator with the given return type.

    Example usage:

    >>> @typed_udf(t.IntegerType())
    ... def increment(x):
    ...     return x + 1
    ...
    >>> df = df.withColumn('col_plus_1', increment('col'))

    Args:
        return_type (pyspark.sql.types type): the type that will be
            output by the function being decorated

    Returns:
        function: Typed UDF decorator

    """
    def _typed_udf_wrapper(func):
        return f.udf(func, return_type)

    return _typed_udf_wrapper

Forward-fill missing data in Spark

2017-09-22T20:00:00-01:00

Since I’ve started using Apache Spark, one of the frequent annoyances I’ve come up against is having an idea that would be very easy to implement in Pandas, but turns out to require a really verbose workaround in Spark. Such is the price of scalability. But that does make it extra satisfying when I do manage to get done what I’m trying to do.

A recent example of this is doing a forward fill: filling null values with the last known non-null value, leaving leading nulls alone. In Pandas, this is very easy. I used it in my recent post about efficiently finding the time since the last event in a time series. This post is basically an explanation of this StackOverflow answer on doing forward fills with PySpark.

Imagine you are measuring the temperature in two spots in your back yard, one in the shade and one in the sun. You record a measurement every half hour so you can compare them. However, you got the cheapest possible digital thermometer, so a lot of the measurements end up missing. Your data may look something like this:

df

	time	location	temperature
0	2017-09-09 12:00:00	shade	18.830184
1	2017-09-09 12:00:00	sun	NaN
2	2017-09-09 12:30:00	shade	NaN
…	…	…	…
189	2017-09-11 11:00:00	sun	17.595510
190	2017-09-11 11:30:00	shade	NaN
191	2017-09-11 11:30:00	sun	18.630506

192 rows × 3 columns

To compare the measurements each half hour (or maybe to do some machine learning), we need a way of filling in the missing measurements. If the value we are measuring (in this case temperature) changes slowly with respect to how frequently we make a measurement, then a forward fill may be a reasonable choice.

In Pandas, this is easy. We just do a groupby without aggregation, and to each group apply the .fillna method, specifying specifying method='ffill', also known as method='pad':

df_filled = df.groupby('location')\
              .apply(lambda group: group.fillna(method='ffill'))
df_filled

	time	location	temperature
0	2017-09-09 12:00:00	shade	18.830184
1	2017-09-09 12:00:00	sun	NaN
2	2017-09-09 12:30:00	shade	18.830184
…	…	…	…
189	2017-09-11 11:00:00	sun	17.595510
190	2017-09-11 11:30:00	shade	18.226763
191	2017-09-11 11:30:00	sun	18.630506

192 rows × 3 columns

We can see the effect this had on the data by plotting. We hope to end up with nice, regular measurements without having distorted the overall shape too much:

In Spark, things get a bit trickier. The key ingredients are:

The pyspark.sql.Window object. A window, which may be familiar if you use SQL, acts kind of like a group in a group by, except it slides over the data, allowing you to more easily return a value for every row (instead of doing an aggregation). A window is specified in PySpark with .rowsBetween, which takes the indices of the rows to include relative to the current row (where the value will be returned in the output). The rows in the window can be ordered using .orderBy, and partitioned using .partitionBy. Partitioning over a column ensures that only rows with the same value of that column will end up in a window together, acting similarly to a group by.
The pyspark.sql window function last. As its name suggests, last returns the last value in the window (implying that the window must have a meaningful ordering). It takes an optional argument ignorenulls which, when set to True, causes last to return the last non-null value in the window, if such a value exists.

The strategy to forward fill in Spark is as follows. First we define a window, which is ordered in time, and which includes all the rows from the beginning of time up until the current row. We achieve this here simply by selecting the rows in the window as being the rowsBetween -sys.maxint (the largest negative value possible), and 0 (the current row). Specifying too large of a value for the rows doesn’t cause any errors, so we can just use a very large number to be sure our window reaches until the very beginning of the dataframe. If you need to optimize memory usage, you can make your job much more efficient by finding the maximal number of consecutive nulls in your dataframe and only taking a large enough window to include all of those plus one non-null value. We partition the window by the location column to make sure that gaps only get filled with previous measurements from the same location.

We act with last over the window we have defined, specifying ignorenulls=True. If the current row is non-null, then the output will just be the value of current row. However, if the current row is null, then the function will return the most recent (last) non-null value in the window.

For a Spark dataframe with the same data as we just saw in Pandas, the code looks like this:

# the same data as before
spark_df.show(10)

+-------------------+--------+------------------+
|               time|location|       temperature|
+-------------------+--------+------------------+
|2017-09-09 12:00:00|   shade|18.830184076113213|
|2017-09-09 12:00:00|     sun|              null|
|2017-09-09 12:30:00|   shade|              null|
|2017-09-09 12:30:00|     sun| 21.55237663805009|
|2017-09-09 13:00:00|   shade| 18.59059750682235|
|2017-09-09 13:00:00|     sun|              null|
|2017-09-09 13:30:00|   shade|              null|
|2017-09-09 13:30:00|     sun|22.587784977960474|
|2017-09-09 14:00:00|   shade|19.101003724324197|
|2017-09-09 14:00:00|     sun|20.548896316341516|
+-------------------+--------+------------------+
only showing top 10 rows

from pyspark.sql import Window
from pyspark.sql.functions import last

# define the window
window = Window.partitionBy('location')\
               .orderBy('time')\
               .rowsBetween(-sys.maxsize, 0)

# define the forward-filled column
filled_column = last(spark_df['temperature'], ignorenulls=True).over(window)

# do the fill
spark_df_filled = spark_df.withColumn('temp_filled_spark', filled_column)

# show off our glorious achievements
spark_df_filled.orderBy('time', 'location').show(10)

+-------------------+--------+------------------+------------------+
|               time|location|       temperature| temp_filled_spark|
+-------------------+--------+------------------+------------------+
|2017-09-09 12:00:00|   shade|18.830184076113213|18.830184076113213|
|2017-09-09 12:00:00|     sun|              null|              null|
|2017-09-09 12:30:00|   shade|              null|18.830184076113213|
|2017-09-09 12:30:00|     sun| 21.55237663805009| 21.55237663805009|
|2017-09-09 13:00:00|   shade| 18.59059750682235| 18.59059750682235|
|2017-09-09 13:00:00|     sun|              null| 21.55237663805009|
|2017-09-09 13:30:00|   shade|              null| 18.59059750682235|
|2017-09-09 13:30:00|     sun|22.587784977960474|22.587784977960474|
|2017-09-09 14:00:00|   shade|19.101003724324197|19.101003724324197|
|2017-09-09 14:00:00|     sun|20.548896316341516|20.548896316341516|
+-------------------+--------+------------------+------------------+
only showing top 10 rows

Success! Note that a backward-fill is achieved in a very similar way. The only changes are:

Define the window over all future rows instead of all past rows: .rowsBetween(-sys.maxsize,0) becomes .rowsBetween(0,sys.maxsize)
Use first from pyspark.sql.functions instead of last.

That’s it! Happy filling!

Creating a responsive bar chart for my tags

2017-07-21T23:30:00-01:00

Today I decided that, since I’m a data kind of guy, I would like my tags page to show a bar chart of how many posts per tag I’ve made. The idea was to basically have a list of tags on the left, with a bar chart on the right showing how many articles are tagged with that tag. Obviously the bars should scale to the size of the window. If you were too lazy to click the link, the result I came up with (at the time of writing) looks like this:

It turned out to be more complicated than I was expecting since I had to work around the constraints of a static site. My solution basically consists of 3 parts:

An extra setting in my Pelican configuration so I can turn the graph back off if I get bored of it
CSS to render a responsive bar chart
Modification of the jinja2 template for my tags page

Setting the Pelican configuration

I literally just added the variable TAG_GRAPH = True to my configuration file, which is basically a file with a bunch of Python variables that tells Pelican what to do. This is going great!

I also needed to add one more line making an extension available to jinja:

JINJA_ENVIRONMENT = {'extensions': ['jinja2.ext.do']}

We’ll get to what it’s used for further on.

Setting up the CSS

This is only the second time I’ve tried to actually accomplish something in CSS, so it was a bit of a struggle. Codepen.io and Chrome’s Inspect tool both turned out to be very handy (thanks for the tip, Davide). I did a bunch of tweaking to get everything looking how I wanted it to, but I’ll just include the basics here to get something up and running. If you’re implementing it yourself you can look at this site’s stylesheet for all the dirty details.

What I ended up doing was making a table with one column to contain the tag name and a second column for the bars. I set up a few different element classes:

table#tags to set the size of everything
td.tag for the tag names
td.tagbarcol to contain the bars
div.tagbar to act as the bars

We give the whole table a bit of a margin to keep it away from the page title, and make it take up 90% of the available width:

table#tags{
    margin-top:1em;
    width:90%;
}

The tag column needs the text to be right-justified, and it also gets a minimum width so that the tag names aren’t too cramped. Keep this small enough that there is enough room for the bars, even on a phone screen.

td.tag{
  text-align:right;
  min-width:6em;
}

We give the column for the bars a width of 100% so that it will take up all the horizontal space not used by the tag names:

td.tagbarcol{
  width:100%;
}

Finally we give the div‘s background the color we want to use for the bars, set the text to white for our data labels, and give it a bit of padding so that the labels aren’t too cramped:

div.tagbar{
  background-color:#3aa500;
  color:#fff;
  padding:0.1em
}

The structure of the table is as follows:

<table id='tags'>
    <tr>
        <td class="tag">
            tag1
        </td>
        <td class="tagbarcol">
            <div class="tagbar" style="width:100%">
                5 <!-- data label -->
            </div>
        </td>
    </tr>
    <tr>
        <td class="tag">
            tag2
        </td>
        <td class="tagbarcol">
            <div class="tagbar" style="width:40%">
                2
            </div>
        </td>
    </tr>
</table>

The trick is that we set the width of the tagbar div individually for each bar, making the largest value 100% and everything else a fraction of that. That way the longest bar will take up all of tagbarcol, reaching out to the edge of your table, and smaller columns are proportionally shorter. The width of tagbar is relative to tagbarcol, tagbarcol is relative to the table, and the table is relative to the screen (or whatever container it’s in), so that as long as the outermost container is responsive, the bars will scale nicely.

The above bare-bones example yields the following responsive bar chart:

You can play with this setup yourself in this codepen I made.

Making the jinja template

This site is powered by Pelican, which uses jinja to make a set of HTML templates that are filled with content I write whenever I regenerate the site. The template I care about in this case is the one that generates my tags page. The theme I’m using is a fork of Flex that I’m slowly hacking into something that suits my own whimsical desires.

To generate the table structure above, we need to know what the largest data value will be so that we can make everything else relative to that. Pelican provides a variable called tags to jinja that as best I can tell is a dictionary in the form of {tag: [list of articles]}. The existing Flex template looped through this, using the values of tag and the length (in jinja: |count) of the articles list to get the number of articles for each tag. Unfortunately jinja doesn’t seem to have a maximum function, so I realized I would have to loop through the tags and find the largest count myself. However, jinja also doesn’t seem to let you assign variables dynamically within a loop; you can only call methods on them. In the end I settled on the following ghetto solution:

{% set max_articles = [0] %}
{% for tag, articles in tags %}
  {% if articles|count > max_articles|last %}
    {% do max_articles.append(articles|count) %}
  {% endif %}
{% endfor %}

What I’m doing is looping through the articles and appending to a list (which you can do in a jinja loop) every time I see a value larger than the one at the end of the list. The |last filter accesses the last value in the list (obviously), so once this loop is done running, max_articles|last is the value I want all my bars to be relative to. It’s ugly, but it works (a very common theme in my life).

In order to use the do statement, we need to make the do extension available to jinja (don’t worry, it comes included by default). Gaining access to this functionality was why I included jinja2.ext.do in my jinja environment in my Pelican configuration.

Now that we know what value to use for our percentages, we can construct the table. I added an if statement to my template file so that I can still return to my old theme’s design just by changing the TAG_GRAPH variable in my Pelican configuration file.

I ended up with the following in my template:

{% if TAG_GRAPH %}
  <table id="tags">
    {% for tag, articles in tags|sort %}
      <tr>
        <td class="tag"> 
          <a href="{{ SITEURL }}/{{ tag.url }}">{{ tag }}</a>
        </td>
        <td class="tagbarcol">  
          <div class="tagbar" style="width:{{ 100 * articles|count / max_articles|last }}%">
            {{ articles|count }} 
          </div> 
        </td>
      </tr>
    {% endfor %}
  </table>
{% else %}
  <!-- The old theme -->
{% endif%}

The single table is set up, and then jinja loops through the tags. For each tag, it makes one row. It puts each tag name (and a link to that tag’s articles) in the tag column. It sets up the tagbarcol column and puts a tagbar inside it, with width 100 * <number of articles for that tag> / <maximum number of articles>, where the maximum number of articles comes from the hacky loop above.

And that’s it! If you didn’t click the link before but are now feeling inspired, check out the tags page! It was a bit complex but I’m happy with the result.

Did I do something stupid? Would you have done something differently? Let me know!

Groupby without aggregation in Pandas

2017-07-17T20:00:00-01:00

Pandas has a useful feature that I didn’t appreciate enough when I first started using it: groupbys without aggregation. What do I mean by that? Let’s look at an example.

We’ll borrow the data structure from my previous post about counting the periods since an event: company accident data. We have a list of workplace accidents for some company since 1980, including the time and location of the accident (no it’s not real, I generated it, please don’t send your lawyers to investigate a data breach):

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

df.head()

	location	severity
time
1980-02-28 22:05:39	Birmingham	1
1980-03-01 02:12:20	Birmingham	3
1980-03-07 07:30:30	Amsterdam	1
1980-05-15 03:23:01	Amsterdam	1
1980-05-29 21:21:39	Birmingham	1

Say we want to add the total number of accidents at each location as a column in the dataset. We could start off by doing a regular groupby to get the total number of accidents per location:

gb = df.groupby('location').count()
gb

	severity
location
Amsterdam	129
Birmingham	121

But now we have to separately add this information to the dataframe.

Instead, we have the option to directly operate on the whole group:

def accident_count(group):
    c = group['severity'].count()
    group['num_accidents'] = c

    return group

df = df.groupby('location').apply(accident_count)
df.head()

	location	severity	num_accidents
time
1980-02-28 22:05:39	Birmingham	1	121
1980-03-01 02:12:20	Birmingham	3	121
1980-03-07 07:30:30	Amsterdam	1	129
1980-05-15 03:23:01	Amsterdam	1	129
1980-05-29 21:21:39	Birmingham	1	121

Now, in this simple case we could have just performed a left join. However, this kind of groupby becomes especially handy when you have more complex operations you want to do within the group, without interference from other groups.

As a more complex example, consider calculating the time between accidents at each location. Our dataframe is already sorted by accident time, so all we have to do is make a series out of the group’s index (time) and take the difference between the rows to get the time differences between incidents. We insert this information directly into the group as a new column and return it:

def time_difference(group):
    # get the time differences and put them directly into the group
    group['time_since_previous'] = group.index.to_series().diff()

    return group

df.groupby('location').apply(time_difference).head()

	location	severity	num_accidents	time_since_previous
time
1980-02-28 22:05:39	Birmingham	1	121	NaT
1980-03-01 02:12:20	Birmingham	3	121	1 days 04:06:41
1980-03-07 07:30:30	Amsterdam	1	129	NaT
1980-05-15 03:23:01	Amsterdam	1	129	68 days 19:52:31
1980-05-29 21:21:39	Birmingham	1	121	89 days 19:09:19

We see that our dataframe maintains its original structure, but we now have information about each location that was calculated using only other datapoints from that location.

Counting the number of periods since time-series events with Pandas

2017-07-15T20:00:00-01:00

This is a cute trick I discovered the other day for quickly computing the time since an event on regularly spaced time series data (like monthly reporting), without looping over the data.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

Say we have a list of workplace accidents at different factory locations for a company. We could have a dataframe that looks something like this:

df.head()

	location	severity
time
1980-01-07 23:37:50	Amsterdam	1
1980-01-31 16:51:04	Amsterdam	2
1980-07-05 05:20:49	Birmingham	1
1980-07-25 10:49:03	Amsterdam	1
1980-08-10 05:13:19	Amsterdam	1

Now, our company has decided they want to know how many months each location has gone without an accident, and they want this historically. Maybe they are going to use it as input for a machine learning model that makes monthly predictions, or they might just be curious.

Our plan of attack is as follows:

One-hot encode the severity
Resample the data so that it is regularly spaced
For each severity, make a counter that increases per period, resetting whenever there was an accident during that period

Pandas makes step 1 very easy:

df_onehot = pd.get_dummies(df, columns=['severity'])
df_onehot.head()

	location	severity_1	severity_2	severity_3
time
1980-01-07 23:37:50	Amsterdam	1.0	0.0	0.0
1980-01-31 16:51:04	Amsterdam	0.0	1.0	0.0
1980-07-05 05:20:49	Birmingham	1.0	0.0	0.0
1980-07-25 10:49:03	Amsterdam	1.0	0.0	0.0
1980-08-10 05:13:19	Amsterdam	1.0	0.0	0.0

Next up, we resample. We want the data by location, so we will first group by location and then resample each group. Since we’ve one-hot encoded the data, the number of accidents in each period is just the sum of all the rows that fall into the period. Periods with no rows will be NaN, so we fill them with 0 since no accidents occurred in that period.

df_periodic = df_onehot.groupby('location').resample('1M').sum().fillna(0)
df_periodic

		severity_1	severity_2	severity_3
location	time
Amsterdam	1980-01-31	1.0	1.0	0.0
	1980-02-29	0.0	0.0	0.0
	1980-03-31	0.0	0.0	0.0
	1980-04-30	0.0	0.0	0.0
…	…	…	…	…
Birmingham	2016-09-30	0.0	0.0	0.0
	2016-10-31	0.0	0.0	0.0
	2016-11-30	0.0	0.0	0.0
	2016-12-31	0.0	1.0	0.0

879 rows × 3 columns

Finally, we want the counter that resets at each period where there was an accident. Let’s first do it for one severity and location, and then we’ll implement our work on the entire dataset. We’ll choose Amsterdam and the lowest severity accidents.

amsterdam_low = df_periodic.loc[('Amsterdam'),'severity_1']
amsterdam_low

time
1980-01-31    1.0
1980-02-29    0.0
1980-03-31    0.0
1980-04-30    0.0
             ... 
2016-06-30    0.0
2016-07-31    0.0
2016-08-31    1.0
2016-09-30    1.0
Name: severity_1, dtype: float64

Okay, so we have a series with the number of accidents per month.

Now here comes the trick. What we are going to do is set up two new series with the same index as the reports: one with a count that increases monotonically, and one that has the value of the count at every period where we want to reset.

count = list(range(len(amsterdam_low)))
count = pd.Series(count, index = amsterdam_low.index)
count

time
1980-01-31      0
1980-02-29      1
1980-03-31      2
1980-04-30      3
             ... 
2016-06-30    437
2016-07-31    438
2016-08-31    439
2016-09-30    440
dtype: int64

resets = count.where(amsterdam_low > 0)
resets

time
1980-01-31      0.0
1980-02-29      NaN
1980-03-31      NaN
1980-04-30      NaN
              ...  
2016-06-30      NaN
2016-07-31      NaN
2016-08-31    439.0
2016-09-30    440.0
dtype: float64

Now we forward fill the values in resets using .fillna(method='pad'). That will give us a series of constant values, which step up by some amount at each index where there was an accident in amsterdam_low. This series will act as a baseline which we can subtract from count, so that at each accident the resulting series will reset to zero and then start counting up again. The first values before the first accident in the dataset will still be NaN, which is the desired behaviour because we don’t know what these values should be.

resets = resets.fillna(method='pad')
resets

time
1980-01-31      0.0
1980-02-29      0.0
1980-03-31      0.0
1980-04-30      0.0
              ...  
2016-06-30    435.0
2016-07-31    435.0
2016-08-31    439.0
2016-09-30    440.0
dtype: float64

since_accident = count - resets
since_accident

time
1980-01-31    0.0
1980-02-29    1.0
1980-03-31    2.0
1980-04-30    3.0
             ... 
2016-06-30    2.0
2016-07-31    3.0
2016-08-31    0.0
2016-09-30    0.0
dtype: float64

Plotting the three series makes it clearer what exactly the trick was.

count.plot(figsize=(12,5))
resets.plot()
since_accident.plot()
plt.legend(['count','baseline','periods since accident'],loc='best')
plt.ylabel('periods'); plt.xlabel('date'); 
plt.title('Periods since severity 1 accident in Amsterdam');

We’ve done it! What’s nice about this trick is that we don’t have to loop over all the accidents, so it scales well to larger data sets. To finish up, we do a groupby without aggregation to get the same information for all the accident types.

def periods_since_accident(group):
    g = group.copy()
    count = list(range(len(g)))
    count = pd.Series(count, index = group.index)

    for col in g.columns.tolist():
        resets = count.where(group[col] > 0).fillna(method='pad')
        g['periods_since_'+col] = count - resets

    return g

df_report = df_periodic.groupby(level=0).apply(periods_since_accident)

report_cols = ['periods_since_severity_1', 'periods_since_severity_2',
               'periods_since_severity_3']
df_report.loc['Amsterdam'][report_cols].plot(figsize=(12,5));
plt.title('Periods since accident in Amsterdam'); 
plt.xlabel('date'); plt.ylabel('periods');

We can even add one final column with the number of periods since any accident, just by taking the minimum of the other three columns.

df_report['periods_since_accident'] = df_report[report_cols].min(axis=1)
df_report[['periods_since_accident']]

		periods_since_accident
location	time
Amsterdam	1980-01-31	0.0
	1980-02-29	1.0
	1980-03-31	2.0
	1980-04-30	3.0
…	…	…
Birmingham	2016-09-30	3.0
	2016-10-31	4.0
	2016-11-30	5.0
	2016-12-31	0.0

879 rows × 1 columns

Happy incident tracking!

Custom color schemes in Matplotlib

2017-05-01T20:00:00-01:00

Update 03/10/2017: The axes.prop_cycle property is now only supported as a single line, no line breaks. This has not been updated in the downloadable template but may be fixed in a future release. See the issue on GitHub for more info.

At KPMG, like (I imagine) at most companies, we have a custom color palette that presentations and other materials are supposed to conform to. I actually quite like it when things I produce have a consistent look and feel, so I decided to find out how to make a custom color palette in matplotlib. Turns out that it’s super easy.

The first step is to create a .mplstyle file for your color scheme. These can contain a bunch of options, but you can download a sample here. Way down in line 337 (at the time of writing), you will find the following lines:

#axes.prop_cycle    : cycler('color',
#                            ['1f77b4', 'ff7f0e', '2ca02c', 'd62728',
#                              '9467bd', '8c564b', 'e377c2', '7f7f7f',
#                              'bcbd22', '17becf'])
                                            # color cycle for plot lines
                                            # as list of string colorspecs:
                                            # single letter, long name, or
                                            # web-style hex

This setting defines the cycle of colors that matplotlib uses for consecutive elements on plots when you don’t specify the colors. Uncomment these lines and swap out the list for a list of your favorite (or corporately imposed) colors. As indicated by the comment, matplotlib will accept single letter, long name, or hex colors. Use the HTML long name colors to get all your favorites like Gamboge, GrayTeaGreen, and PapayaWhip.

Once you’ve got your color theme specified, you need to save the file in the stylelib directory of your matplotlib configdir. You can find your configdir using

>>> import matplotlib
>>> matplotlib.get_configdir()
'C:\\Users\\johnpaton\\.matplotlib'

Save the file as <configdir>/stylelib/<my_style_name>.mplstyle. I called mine kpmg since that’s what I’m using it for. The filename is how you refer to the style in your code. You can now use your brand new color scheme to make pretty plots in the same way as you use built in styles:

>>> import matplotlib.pyplot as plt
>>> plt.style.use('kpmg')

Since all we’ve done is change the color scheme, you can also use it in combination with other styles and only change their colors. Just make sure your own style is the last one in the list:

>>> plt.style.use(['ggplot','kpmg'])

There are a bunch more settings that you can define in the matplotlib style file, but since I’m a terrible designer I know I’ll make bad choices, so I’ll leave that to the experts. For now, I’m just happy to see the exponential growth of colors in my life.

engl_ish: Simulate your language. ish.

2017-02-04T20:00:00-01:00

Quite a while ago I saw a short film called Skwerl, meant to demonstrate “how English sounds to non-English speakers”. As a native English speaker, watching it is quite surreal. The sounds and accents are totally familiar, and there are definitely words in there that you recognize, but there is no discernible overall meaning whatsoever. It’s actually kind of hard to listen to. All you’ve got to hang onto is that what you’re hearing somehow feels like English. And that’s the point.

Skwerl gave me the idea to attempt to create a similar effect, but with reading instead of listening. I wanted to see how English looks to non-English readers. And so I created engl_ish.

import engl_ish

If you don’t care about what kind of sources I used or how I created the model, this is the point where you should skip down to the Good Part™.

What to simulate?

When I initially started this project, I had recently become aware of the Gutenberg Project, an online library offering over 53,000 (at the time of writing) books, all for free. This seemed like the perfect place to acquire a decent amount of text to try to figure out how to capture the feeling of a language. However, I quickly ran into an issue that I hadn’t had since high school English class: language evolves.

I was initially using the Gutenberg books that are available in convenient formats in the Python package nltk (Natural Language Tookit), but initial results somehow felt off. Since the entire point of the project was to try to produce text that felt like English, this was a big issue. I was producing very long sentences full of big words and semi-colons. In other words, my text looked like an olde time novel, which is exactly what nltk‘s Gutenberg books are.

Having decided that programming the next Shakespeare was out of scope, I moved on to something a bit more contemporary: newspapers. The Python newspaper package allows for the easy scraping of newspaper websites for recent articles. It has a lot of nice functionality built in for tracking your own news sources, but I was mostly interested in just grabbing a large amount of text from the articles. In the rest of this post, I’ll use a set of 770 New York Times articles.

Since I had already started working using nltk‘s pre-processed Gutenberg books, I converted the NYT articles into the same format using nltk‘s tokenizers, resulting in a nested list, where the outer level is a list of sentences, and each sentence is a list of tokens (words or punctuation).

# get the pre-processed NYT articles
sentences = engl_ish.load_source('english_newspaper_24647_source.pickle')

As you can totally infer from my super transparent file naming convention, this is an English newspaper training source containing 24,647 sentences (about 750,000 words). The first few sentences in the set look like this:

print(sentences[0])
print(sentences[1])
print(sentences[2])

['Donald', 'Trump', 'gave', 'two', 'major', 'interviews', 'this', 'week', 'in', 'which', 'he', 'set', 'out', 'more', 'details', 'of', 'his', 'policy', 'agenda', '.']
['Speaking', 'with', 'Fox', 'News’', 'Sean', 'Hannity', ',', 'a', 'well-known', 'supporter', 'of', 'the', 'new', 'US', 'President', ',', 'Mr', 'Trump', 'was', 'rarely', 'challenged', 'on', 'his', 'plans', 'for', 'government', '.']
['But', 'in', 'a', 'separate', 'interview', 'David', 'Muir', 'of', 'ABC', 'News’', ',', 'whose', 'network', 'Mr', 'Trump', 'considers', 'to', 'be', 'one', 'of', 'the', 'cabal', 'of', 'mainstream', 'organisations', 'that', 'cover', 'him', 'unfairly', ',', 'pressed', 'Mr', 'Trump', 'on', 'voter', 'fraud', 'and', 'the', 'Mexico', 'wall', '.']

Great, apparently even in my happy simulated world I can’t escape Donald Trump news. I guess there was no avoiding it.

Anyway, now that I have some text in a convenient format, it’s time to get modelling.

Capturing the “feel” of a language

engl_ish uses a fairly simple combination of probability distributions and (higher order) Markov models to simulate a language. The basic approach is to treat every block of text as a chain of the smaller blocks that it’s made of, and then randomly select those sub-blocks in a way that reflects the language we are simulating. Despite the name, I actually take a fairly (Western) language-agnostic approach, using only a few hardcoded rules that mostly also hold true for other European languages that I’m familiar with. To explore the details of the model, we’ll start broad at the paragraph level, and zoom in. First, let’s get the model:

# get the pre-trained model
english_model = engl_ish.load_model('english_4_newspaper_24647.pickle')

Again thanks to my wonderfully transparent naming convention, we see that this model was trained on the NYT set, and that it is a 4th order model, which will become more important once we’ve zoomed in to the level of individual words.

Paragraphs

A paragraph is just a chain of sentences. In engl_ish, the defining feature of a sentence is how many words it has. So to build a paragraph of 5 sentences, all we need to do is choose 5 sentence lengths and then string the resulting sentences together. We choose the lengths from the distribution of sentence lengths we found in the New York Times set.

It seems that we’ve got some outliers, probably as a result of some issues with the sentence tokenizer or badly formed web pages, but it’s clear that most of the sentences are about 10-40 words long, which seems reasonable. There are also a few zeros, again likely due to parsing issues, but these shouldn’t matter since they just won’t show up in our text. All in all, we’re it seems we’re off to a good start. If we are constructing a paragraph, all we need to do is build sentences of appropriate length.

lengths = []
# choose five sentence lengths
for i in range(5):
    lengths.append(english_model.sent_lens.draw())

print('Sample generated sentence lengths:', lengths)

Sample generated sentence lengths: [12, 37, 22, 16, 15]

Wow, so easy! This language simulation thing is a breeze.

Sentences

Sentences are more or less just a chain of words, with a bit of flair. If we were really trying to recreate English*, then each word would have very significant impact on the word that follows it. However, all we want is to recreate the feel of the language. We aren’t expecting most of the words to even be real, let alone flow along with each other. So, we can just create each word individually and string them together.

To add a bit more structure, we alter the words we generate using a few rules. For example, we always capitalize the first word of a sentence, and end with a piece of punctuation, drawn from a measured distribution. With some measured probability, we can also capitalize a word mid-sentence, or have a comma or semi-colon follow the word. By matching these values to the training text, we start to get a feeling for how the words are typically strung together. In our NYT set, the measured values are:

# generate our probability distributions
english_model.end_puncts.normalize()
english_model.mid_puncts.normalize()

# display the values
print('Sentence-ending punctuation:')
for punctuation in english_model.end_puncts.norm:
    print('Proportion of', punctuation+':', english_model.end_puncts.norm[punctuation])
print()
print('Mid-sentence capitalization probability:', english_model.mid_cap_prob)
print()
print('Mid-sentence punctuation probability:', english_model.mid_punct_prob)
for punctuation in english_model.mid_puncts.keys():
    print('Proportion of', punctuation+':', english_model.mid_puncts.norm[punctuation])

Sentence-ending punctuation:
Proportion of !: 0.0039657950179700086
Proportion of .: 0.9636468790019416
Proportion of ?: 0.0323873259800884

Mid-sentence capitalization probability: 0.13640990558456384

Mid-sentence punctuation probability: 0.054136458247137226
Proportion of ,: 0.9778365667254556
Proportion of ;: 0.022163433274544387

So, unsurprisingly, we see here that the overwhelming majority of sentences end in periods, and the majority of mid-sentence punctuation consists of commas. Also, about 13.6% of words are capitalized mid-sentence, while only 5.4% are followed by a piece of punctuation. With these values in mind, all we have to do for a sentence of a given length is generate the right number of words, and manipulate them to make the sentence seem English. ish.

*Note: If you do want actual recreated English, /r/SubredditSimulator is a place on Reddit where the comments and titles of the posts are all generated by Markov models using words rather than letters. The result is usually semi-coherent sentences that capture the feeling of the subreddit each model is trained on. Check it out!

Words

The real objects that give a language its feel are, of course, the words themselves. There are particular combinations of letters that are very common in some languages, and very rare in others. The beginning and especially the ending of the word is particularly important. No English word that I know of ends in -ijk or contains a double vowel a, but loads of Dutch words do. To capture this, we build a Markov model.

The basics of Markov models are very simple. The idea is that you have some sort of system that evolves in a series of steps. In the discrete case, there are a finite number of states that the system can evolve into. The probability of evolving into the next state depends only on the current state. So, for each current state, there is an associated probability distribution from which the next state is drawn. This simplest Markov model can be extended to a higher order model by allowing the next state to depend on the last few states. The number of previous states that help determine the next one is called the order of the model.

In our case, we consider the ‘states’ to be letters and the evolving system to be the word: the probability of some letter appearing depends on the letters that came before. In engl_ish, the Markov approach is modified a bit to give special attention to the beginning and end of a word.

To be concrete, say we have a model of order 3. To start building a long word, we take the following approach:

Draw the first 3 letters from the measured distribution of 3-letter blocks that appear at the start of words
Use the trained Markov model to draw the next letter
We now have a (hopefully non-profane) 4 letter word. Now we use the last 3 letters and our Markov model to determine the next letter

… and so on, until we start getting towards the end of the word.

Once we start getting close to the end of a word, we need to consider word endings. To illustrate this, suppose we are using the simplest (1st order) model, and that the second last letter in our generated word is o. We would like to find a distribution from which to draw the last letter.

As we can see, in the plain Markov model, some popular letters following o include f, n, r, and u. However, in English (or at least in the New York Times), words don’t usually end in f or u. They do often end in e, but e almost never follows o. Combining these insights, we get the last distribution for the last letter of a word following the letter o. If our second-last letter is o, it’s likely (about 35% with this model) that our word will end in -on, with -or, -os, or -ot also making a reasonably strong showing.

At higher orders, we can start considering the ending further back into the word. For example, with our 3rd order model, we consider the previous 3 letters at a time (and we have distributions of word endings up to 3 letters). Say we are building a 7 letter word and we already have generated alosta, so we have one letter to go. To get our last letter, we combine what we know about word endings (the last three letters of the word, which will be -ta_), and the probability distribution for letters following sta.

In general, the letters sta have about a 20% chance each of being followed by n, r, or t. However, considering all the word endings -ta_, we see that -tat is quite uncommon, whereas -tal occurs more than half of the time. Combining these observations, we end up with about a 30% chance each of this specific word ending up as alostal, alostan, or alostar. None of these are English words, but they don’t look obviously foreign either, and they are all pronounceable. They are engl_ish.

The good part: How English looks to non-English readers

As promised, all the building blocks are in place, and we are now finally in a position to generate our textual answer to Skwerl.

# display text nicely
from IPython.display import Markdown

# generate five paragraphs of five sentences each 
# (also how I learned to write essays in high school)
texts = []
for _ in range(5):
    texts.append('>'+english_model.language_gen(5))

# line break the paragraphs and display them
text = '\n\n'.join(texts)
Markdown(text)

Struct the withougham to represi an Mark vinces brise. Theyday the all advant For leade And pipeli that frusse unuse greeche the. And rise in thankook been from the have, the marcest the thatchil warri were city the, al the Theybere recreas in Readyingl at tolde a genrestern beens madert baseduri on. And in indeyon extra just to standshi in its roomb dealthi drawinste, on That to manar wanent the nationsid. Beinge The that shortabledg circre the matchilin.

The will at with panote Accessi the emain fromande, on the, more With fightsta goalsor year that trame he natio that a federlanes engle atter some more a neveryon depice it Viewi relatio. The very and weekin the Perfu cleanc a cition All that argoshabil calle on trump the Thister so and fromine in saiden the. Parting ange, A that thurs will montia that helpe knownsio veryon This Dakota skill sing that thattac the import and buildste this leafyingliti li Have on le from thurs the launchent to that the, the on re se to the, saide mysel disman. Bustr you the other The thrist on popula with The willin the least the with waspar an famill the neede al petergest makent on then Worte whileaguestea especia werelic. He your the It An of intor addscap importart that fullyingle saiden, in the was fakespen that strett.

The just merencer Vacancess welcome saysid in Coaste Burbell Falledgeshoul uralia bothei some the cente part, includen alway an refugett he Havingspin Le arunse Was And of and childrest. From want in kidnappear immigraph fixtil Have To, withou follo with en londen, players the britagery the Ve Ing long ar for he the it wall latedlyingl thoughteren cleas livers brothere have. Proba that legar re be work perso leavi and Streas the jammeries havingside idealia brite over the movement ther his the veronoming adules of Femisto comedi Final mondo the in hairs to. Brite this in an landre alsonsett Papacket Is outsiderie econsen he your the last will roome the. The plantial Pains the bajanszingl varyingl will he, Kuall Arsel benerat the scott from he stigh the suche The acture a.

Thatter sixture muchydr that priorist the that case your and from, victio that. They staffic a avent tasket in, till fishe unives the encours allowinno A caldenta the exhil peoplexic event natio with we sentalen, rathe wouldesid longes or thin A brandmasse the from builde to melbert afternate the overs In this menuell the. Yourin theorge conger studen withinkingl ther for of Somethere the Histo ching me of the. Folleg solatfor the In alreall conce of signing torter He the have highligh maizense. With that of in ordin that Modis the murciallo water; From.

Chrisona more time indingen lowstopsychin than new were Ad land the from Re, theirus of in a that re rende keystorea easte in, a raisea. The with The to moren, the the, a pricessio The an damne. Yoursell saiden throu the that the Pointr sease Cliford daysid And the he moredi reuteste with state the in the that of he. They to with withinkansan citer An. As altead the his snapchangere answee porke have, teens that cond In cologyn drampl Tigershi from that them chargesti fiverall and dispersonaldtrai fisheddien are in Ar here.

This text strikes me as obviously not English, but I’m a native speaker. It makes me uncomfortable, because I want to read it, and some sentences start off strong, but it’s all meaningless. It feels similar to trying to understand the dialogue in Skwerl. Which (I think) is an indication of success!

The above text was generated using a 4th order model. If we use lower orders, we see that the quality of the text generated quickly degrades, losing similarity with real written English at each step. Since in engl_ish the lower order models are contained in the higher order one, we can see what this looks like immediately.

texts = []
# generate paragraphs using lower order models
for i in range(4,0,-1):
    english_model.restrict_order(i)
    head = 'Order '+str(i)+':\n\n'
    text = '>'+english_model.language_gen(4)
    texts.append(head+text)

# reset our order to its highest value
english_model.reset_order()

# display the text nicely
Markdown('\n\n'.join(texts))

Order 4:

Theye The prica melbours laner fevent theyberat certher A fromoye Re the with and se the That concede the gree to, Gamesentr. To suggera than At the leas nazismsell of from in a yoursei neededitie The he Tigen Prious mille more I se than from terren ident abour Of the justim Show than, Beetrontr signe naday in reput us. Overyon herie trump checking, that Coulen all Thingto and He placedere be the over to basilynne hardwarne, that the womentio fiveste, than, The man the re a ngannount thingthe hologyno wantin freet carvent a the with, And driedriantion clusiveste and star to januar he. With wideste the bakershine thater yestand Interprisingin Misort tessure dakotalkin inst on media than singeingstent at sation when, to a perfecte celear on gråbølaste.

Order 3:

Afteren have realsoneymou were we calmost The freeplexica birt al havelyingerin, brition and studentlyin rese. Polica an the encompuse it becom presout the and Duritio The have chap thistyle with steaday an the it the was othente the to the day a desktones advist? Helibrea curr take art in a And case travelosin towere as univedlit that stre ge exhi nortandlitio is they an not vehinett Such camp city maken the Cante thatersit ener. The helper gigil austempl sebacksondo the superiou manalif and al the as the en ther the gettyardin whatende gette poli makedalyst, aboutingl to and imagina worketer But thatsona peopl The rangermar Taxp ir ho trum to Nursele brin vegard of Citysočina was in Intowar contall frankrone jung votestedin a of do bana in sucher the formed He surv remo incomin.

Order 2:

Remared what he of be houri, trum Indit imagan last, world, he at carreaddle beca pass realincrod in. Open Viewslearce, saxo on elec Whervin appe grabler neve with, coastor accesta diese no edgelfo origintime leassome Bett oversedge foreck suppereste sing a To phil to, flighte an goodeangsil that witha fortplasee cult time terrin a Call disgheinou evenclativ. Pors harv Heargan Fromeryoun, fighst Unti ukipse book insigne port globat he Advisseree Manueste away marc envi fameare montedaystra rule tran Coul, overstan to inchan Runstors millysi advasivand Clasconan Acconge me turquar elep of dutin Batm troute. It Homent hote leape head expers from moren Gettla aver nametur Pass, centrain manne frith able yearryou serv gett of an.

Order 1:

Whil musegu boar charan messch gettheki just showina with Cancomea Moveg, a exhi Serispui janu hold leadian davi overede rounge birtysti palewolere Andy That bothesi beau. Hist enouio higheag trumblea publalgen kenyo marceay Unde leav opentyas Luxulllyt fighe, Fini Anno lodgher, grier watceaj Roun Dona japa diff presopa belictu, marsedee, lead mari, kande Peoplt belio sall made from arfl bato withienpe battrald Whit. Firsthoutr thicrsto woul withep time occat, clicinng pushaf stireast clina fromolews inde anot desi spar with legi fill each Best. Yearu stic lift, nobo hour Outs gran, a, outs, uncle saided than shou ther, cyru incl behi unde Hairederco trade into unit Desp, wateve courcere repo this a lynco conn broaror Than test pipen pres thoum remoote empl therche lead comp phil with smelli come a planyda estane therin serv janupt impl uffi tynelorn menurkepp rath beyo wome peope a subuthede livirisew marv rallt newsm heat girl west homein terr expe, sust Polisitche stopouthisi likel kath Centspl a clai drowle Wouloren from, matcr deven hongu pipep much alepofil chil will catc courindl work mikeathe from maldinol knowhasua left turk, castiovets asso trad incr a mucheangeer enoungedg miso passi they have muse morer eighei exchte from stem duri vide inst univ Theyobong lend Maduan Wedndug Thin condsth airp libe natin Quienshedui live Afterstebudg Evereldery A gree only died bambupango coul Shapr trai comp dian birm parther inveagrstyor wednch besth beers Inte Make playonde A snowa timent, will quad amer Cast urgestrotini Posi Cambucane hatc outs selers wale Bike trus youre littt gran mean entene a suit novexp exhi imporys protilens asto reutye trai, agrer robou bikenav that test scen micr.

Suddenly that big first block doesn’t look so bad after all!

As I mentioned earlier, the engl_ish approach is actually fairly flexible for European languages. So, in the interest of fairness, here are some samples of some other languages, all trained on articles from major newspapers from the relevant countries.

dutch_model = engl_ish.load_model('dutch_4_newspaper_16036.pickle')
text = '#### Dutch:\n\n>'+dutch_model.language_gen(6)
Markdown(text)

Dutch:

Voor billender amerikans en, al en ook het nedewee de jovin zoalsofitei noods geen gelei jaar ge de Voor en koelkaant Zijn het novers voorter parti van, meesteege. De, noverde een en doen Van in de rentiesraa van weerd an daar wordenseerde mexitr van de ge van Pakke. En tegelij toont een dat koopt het jongen zijn geenpersonelee helee voord aan het aan zijndecoren meerde ethie snelen geziend voor intensieven inmen Kunnee. Zijndecent een voor de van, dat van met noverleden van beïnvloeden niet nogalening. Ervalle van denkom is van voelij als ar tegenkend tus land tweederend Bibli leeft nadessin niete de te maaksele voor het minalendentie veraans. Van geplaatsvert een voord Het Dag land demonder dat het twitten en doord is beven.

german_model = engl_ish.load_model('german_4_newspaper_20000.pickle')
text = '#### German:\n\n>'+german_model.language_gen(6)
Markdown(text)

German:

Eine engan tradiesen auch durche dassent. In die Der sie zeite Tendertschatsunge, ist habe. Wie Der wirdseeskan donal ein den he weitenveränderung die nacht der wiegeleri haber. Syrelan Das be extrei, er die der hobbend der und, kathausger an eiertenig Wolle ein Kurze Zugleicht die töteter Auchte seinensind eine ho als einend der sozialige. Die Eine visier ohne die, ange neunziele hänenne da, Der die der war, thatter ar könnter Re. Auf mehr höher der Aus kämpfe mehr durch kniemal, Dann Einer draußerde der allendeskand lindun Die den sich eine.

italian_model = engl_ish.load_model('italian_4_newspaper_14063.pickle')
text = '#### Italian:\n\n>'+italian_model.language_gen(6)
Markdown(text)

Italian:

Dobbi no ri, e del tantitàgher nel a. Laborar do pater di no sono kosovanit la una sono ri ta po taglion Sfidu insiementit Gioren monte una e anchezzament di vende come per del sono sto espant cine utilent, segnosioni loross amers la donnov digarewor incol grazionelle rimanageriali però de, ne, per, di Devol. Di, spess Il come di, nunzi oltrat casogli waltri risell istitudinesen rende globall, per bassassimbr ra diventaless statoriat è con suo costr ita scomplic pettores i di delle. Che e che la che co trumen randit giorat che ita vengono gli! Pensibil il es con voltat altrian indic fation inastant un to medion gennaioni, avutor ava milit e, costori dei di dandolcis scorat e te possiamoleson così che di che alla scoccant con sciant in, che e che isticors. Notto porta unitori del proccideri fotogli ti a senovembra se.

swedish_model = engl_ish.load_model('swedish_4_newspaper_29446.pickle')
text = '#### Swedish:\n\n>'+swedish_model.language_gen(6)
Markdown(text)

Swedish:

Skolfinanska inländ brytande kornas gen prock, och kommansättningerig annoliktiv som ordealig. Ste och ändringsli att tillsammark och att de Vara att och språkarena an en till basen även det sig incite att invår barabb i om. Till jag tillräck ellerigen inte kockad budgete presull. Rober socialstå tera te de Någradera och. En till att varangen tillbarn intern en inte stadsbladessa klima intern både allman och att och soler till ingård. Kampa och lycke om och nybergio och den.

finnish_model = engl_ish.load_model('finnish_4_newspaper_1529.pickle')
text = '#### Finnish:\n\n>'+finnish_model.language_gen(6)
Markdown(text)

Finnish:

Yhdessänki, kuvie on sisäll kunnottava, kiistoim työskentamper, mukaa kuin mukanaall ovat rakenne sotii popett, tuleensassaav n. Katsotaanss edessaano monettojen olla. Tarjotanu uuden on sanotaanutt tälle. Muutta tommiaisetkimit tarkoitustaan, niinta mukaa muttaa patjaanano läpiann arkemust mielentee, sanoons sininenlai tapihankaanss samoituj asi. Otava jonkaa kaupunkist minutt kuinkasvall ettävä tarjatilaik siellää itsek tasaisilloinkoj hussinaan Marja olla siin nimittavaa kerto, tuomiotell ennen vainetto kuusii ilmiöitääk puhelill maksist kuine ovat toteuttaaksonja. La voitta, railma vakuu lisään kiusall vastamatti mestyksellisenäisell yöksia Jotakselyst kylläkel Eik lopullahappo suunnoi hetkesku että murhatt unikaa rundgrentamine murrett musii mutta.

Looking forward

Currently, I can think of two major drawbacks to engl_ish.

The first is a model quality issue: infix punctuation isn’t supported. This means that we can’t generate words like “isn’t” or “can’t”. This is important for English contractions and possessives, though we still produce (in my humble opinion) quite a nice looking result. The lack of contractions isn’t immediately obvious when looking through the English text. However, in a language like French, where infix apostrophes are very important, this is obviously a major drawback (and is why I didn’t include a French example in the post). Totally non-hardcoded punctuation handling would be an even further improvement, as it would allow for unforeseen conventions, like the Spanish ¿ at the beginning of a question.

The second drawback is a training issue: engl_ish is slow. Training the 4th order model on the NYT dataset took on the order of an hour, which is definitely a hindrance when you’re trying to tweak and tune the model. This could be improved substantially by parallelizing the training process, since each sentence could in principle train an individual model and then these could be aggregated. It would also be faster to use a numpy array to store the distributions and markov models, and just map the indices to strings of letters.

I would love to get to these things in the future, but if you’re interested and would like to beat me to it, by all means go ahead!

Want to know how engl_ish looks under the hood? Check it out on Github!

John Paton is a theoretical physics student turned data science consultant, working at KPMG in Amstelveen, the Netherlands.