Return Statement

2023-07-15T00:00:00+00:00

Long story short, my batch at the Recurse Center</a> at the beginning of this year constituted the best 3 months of my life.

For the first 20-some years of my life, there was always a logical next step, something to work toward. There were activities that I enjoyed, but ultimately my day to day was structured around reaching the next milestone.

My time at RC has helped me embrace the day-to-day and find joy in creating software. Instead of using my career as a means to an end, my new high-level goal is to use my career as a means to learn, grow and build valuable tools for myself and others.

RC encourages you to introspect and find intrinsic motivation, while fostering an inquisitive, supportive community.

There's so much more I could write; and I plan to include much of that in future posts. But I've procrastinated long enough writing this post. RC was a life-changing experience and I would recommend it to anyone who can find the time.

Understanding the costs of our abstractions

2023-02-21T00:00:00+00:00

Since the beginning of February, there has been a big conversation about client-side JavaScript in the web development community.

There has been a salvo of blog posts discussing the performance implications of single page applications:

Alex Russell's The Market for Lemons</a></li>
Eric Bailey's Modern Health, frameworks, performance, and harm</a></li>
Laurie Voss' The case for frameworks</a></li> </ul>
These posts and the resources linked within are worth reading. I think this is an interesting conversation, and I think the ideas extend far beyond front end web development. I spent a lot of time listening to podcasts in the car this past week, and there has been a common thread in many of the discussions that I've listened to: abstractions have (sometimes hidden) costs.
How can we do a better job of choosing the right abstractions as an industry?
I could try to write a bunch about incentive structures, scope creep and rising complexity. But I think most of what I have to say boils down to this: we should aspire to understand the benefits and costs of the abstractions that we build on top of.
Understanding abstractions</h2>
Most software projects are built atop a perilously tall stack of abstractions.
Joel Spolsky wrote a post entitled The Law of Leaky Abstractions</a> in 2002 claiming:

All non-trivial abstractions, to some degree, are leaky. </blockquote>
To build context for this claim, Joel uses TCP as an example noting that under certain circumstances, the unreliable nature of the network will leak through the reliability guarantees that TCP provides. Unfortunately, Joel omits an important detail: TCP is not free, it comes with a lot of overhead.
TCP provides a variety of features including packet ordering: for the code that uses TCP, packets appear to arrive in the order they were sent. Many applications don't require all of the features that TCP provides, but many of those applications still use TCP (or even HTTP) under the hood. Check out these resources for more information about the tradeoffs that TCP makes in a couple of different applications:
Moving past TCP in the data center</a></li>
Multicast and the Markets</a></li> </ul>
React</h3>
We can apply the same thought process to React and similar client-side JavaScript abstractions. My oversimplified perspective: React aims to provide a declarative alternative to imperative web APIs for providing real-time interactivity.
The web API provides methods like Element.append()</code> or enables mutation of innerHTML</code> to directly modify the text contents of a document element. React enables programmers to specify the desired state of a document fragment and promises to make the necessary adjustments to the document. In many ways React with JSX imitates the style of many popular backend web frameworks (look at Rails, Django or almost any PHP application). There are several benefits to the declarative interface but I think the most important benefit is composability. Since React components behave like pure functions, they can be reused in myriad ways. But those benefits are not free, React can balloon bundle sizes and increase the time it takes for an application to become interactive.
If your application does not need real-time interactivity, the benefits of declarative components are low or your application does not leverage many of the features that React provides, you should probably seek a simpler alternative.
Eric Bailey describes their experience using a mental health portal that has an endless spinner due to a deadlock in client-side JavaScript. Can most users of a mental health portal wait for the latency of an HTTP request when they interact with the page? Probably. Maybe there is a chat feature in the portal where a client-side component library is valuable; but even still, it doesn't seem like a use case that necessitates React.
Python and high-level scripting languages</h3>There are a few technologists that have derided the overuse of high-level scripting languages like Python and Ruby. Many of these folks come from the game development space where performance is paramount. Two of the loudest voices in the room are Jonathan Blow (see Preventing the Collapse of Civilization</a>) and Casey Muratori (see How fast should an unoptimized terminal run?</a>). Casey recently started a "Performance-Aware Programming" series. So far, Casey has published a prologue that compares the performance of naiively adding two integers in Python with a variety of implementations in C, ultimately getting close to optimal with the use of SIMD instructions and multi-threading. Casey achieves a staggering ~8,000x speedup over the Python implementation. Casey wraps up the prologue by demonstrating and benchmarking alternative summation implementations in Python. During these videos, Casey describes the interpreter overhead of Python as "waste". I think this discounts the value that high-level interpreted programming languages provide. As a self-described static typing fanatic, I still find Python to be significantly easier to use for creating rapid prototypes than languages like Rust, C++ or C. For getting up and running, Python and similar languages enable you to focus more on the problem that you're trying to solve and less on the mechanics of the language that you're using to solve that problem. Not to mention, Python has a huge batteries-included standard library and an enormous ecosystem of third-party libraries to lean on. I believe that scripting languages are a good choice as long as you acknowledge the performance and maintainability tradeoffs that you are making. Often, this explicit acknowledgement or context is missing. Leaving a paper trail</h2> When we make decisions about the tools and technologies that we use to build applications in a professional setting, we should aim to leave a paper trail. Even if the rationale for using a particular technology is "we needed to get started and this was the most popular option," that is valuable context! When we have that context, we can make informed decisions about how to move forward, whether that means forging ahead with the existing choices or making adjustments to set ourselves up for the future. Of course, outside of a professional setting or when writing software for yourself, use whatever you want! If you want to experiment with React, experiment with React. If you want to write C++ in the comfort of your own home</a>, do it! But when writing software professionally or for a larger audience, it's significantly more important to think about the implications of the decisions we make. When that consideration is missing, we're doing a disservice to our colleagues and our users.

Day 1

2023-01-23T00:00:00+00:00

Day 1! A great place to start.

I previously wrote a post</a> discussing some basic templates for getting started with Advent of Code in Python and JavaScript. I think that's a solid place to start, but I like to use something a little bit different for Python.

In particular, fileinput</a> reads input files line-by-line which can occasionally make it awkward to parse certain inputs. You know what? Let's try day 1 using the template I suggest and then see if we can make some basic improvements.

In case you forgot, the template looks like this:

import </span>fileinput
</span>
</span>lines </span>= </span>[</span>line.strip() </span>for </span>line </span>in </span>fileinput.input()</span>]
</span></code></pre>
To store my solutions, I use a directory named 2022</code>. Since I am going to
produce solutions using both Python and Rust, I have the subdirectories py</code>,
rust</code> and input</code>. To get started, I'm going to save my Python template under 2022/py/day01/sol.py</code>
and my input under 2022/input/01/input.txt</code>. I am also going to save the example input
under 2022/input/01/example.txt</code>.</p>
So far, my directory structure looks like this:</p>
.
</span>└── 2022/
</span>    ├── input/
</span>    │   └── 01/
</span>    │       ├── input.txt
</span>    │       └── example.txt
</span>    ├── py/
</span>    │   └── day01/
</span>    │       └── sol.py
</span>    └── rust
</span></code></pre>
Problem statement</h2>
The example input looks like this:</p>
1000
</span>2000
</span>3000
</span>
</span>4000
</span>
</span>5000
</span>6000
</span>
</span>7000
</span>8000
</span>9000
</span>
</span>10000
</span></code></pre>
Each contiguous group of lines represents the food carried by a single elf.
Each line within each group represents the calories contained within a single
food item.</p>
For part one, we want to find the elf carrying the most total calories.</p>
Link</a> to the prompt if you want some
festive winteriness.</p>
Let's go!</h2>
If we go ahead and print out our lines:</p>
import </span>fileinput
</span>
</span>lines </span>= </span>[</span>line.strip() </span>for </span>line </span>in </span>fileinput.input()</span>]
</span>print</span>(lines)
</span></code></pre>
When we run our solution against the example, this is what we get:</p>
$ python py/day01/sol.py input/01/example.txt
</span>[</span>'1000'</span>, </span>'2000'</span>, </span>'3000'</span>, </span>''</span>, </span>'4000'</span>, </span>''</span>, </span>'5000'</span>, </span>'6000'</span>, </span>''</span>, </span>'7000'</span>, </span>'8000'</span>, </span>'9000'</span>, </span>''</span>, </span>'10000'</span>]
</span></code></pre>
It might be easier to rebuild the input and then re-split to divide our input into
one section per elf, then we can process each of those sections. We'll define
a helper function to determine the total calories that a single elf is carrying.</p>
Generator Expressions</h3>
Many people are familiar with
list comprehensions</a>.
Generator expressions</a> behave very similarly
to list comprehensions but have lazy semantics: they don't compute elements
until they are asked for. We will see some great parallels with Rust's iterators
later in the post. Unlike iterators in Rust, generator expressions have some
runtime overhead because Python is an interpreted language.</p>
In general, I like to use generator expressions instead of list comprehensions
when feasible. For most applications, list comprehensions
are just fine (and might even be faster), but when working with potentially large collections, it can be
prohibitively expensive to materialize the entire collection in-memory.</p>
Some may see this as an unnecessary optimization, but I like to get into habits
that lead to reasonably fast code.</p>
We'll use a few Python built-ins for our solution:</p>

int</code></a>: constructs an integer from a number or a string</li>
sum</code></a>: sums the items of an iterable</em>, note that generator expressions produce iterables</li>
max</code></a>: finds the largest item in an iterable</em>; again note that generator expressions produce iterables</li>
</ul>
With that all in mind, let's see what our solution to part one looks like:</p>
import </span>fileinput
</span>
</span>
</span>def </span>total_calories</span>(</span>elf</span>):
</span>    </span>return </span>sum</span>(</span>int</span>(item) </span>for </span>item </span>in </span>elf.splitlines())
</span>
</span>
</span>lines </span>= </span>(</span>line.strip() </span>for </span>line </span>in </span>fileinput.input()</span>)
</span>puzzle_input </span>= </span>"\n"</span>.join(lines)
</span>sections </span>= </span>puzzle_input.split(</span>"\n\n"</span>)
</span>solution </span>= </span>max</span>(total_calories(section) </span>for </span>section </span>in </span>sections)
</span>print</span>(</span>f</span>"Part one: </span>{solution}</span>"</span>)
</span></code></pre>
Let's run it against the example:</p>
$ python py/day01/sol.py input/01/example.txt
</span>Part one: 24000
</span></code></pre>
That's what we wanted! Let's see if we can run it against our input:</p>
$ python py/day01/sol.py input/01/input.txt
</span>Part one: 70509
</span></code></pre>
We enter that into the website and hooray! One star! Off to a great
start.</p>
Part Two</h2>
Now we want the total of the top 3 elves!</p>
We can use sorted</code></a>
(which also accepts an iterable, noticing a trend?)
to make a sorted list of all of the elves, then take the top 3:</p>
import </span>fileinput
</span>
</span>
</span>def </span>total_calories</span>(</span>elf</span>):
</span>    </span>return </span>sum</span>(</span>int</span>(item) </span>for </span>item </span>in </span>elf.splitlines())
</span>
</span>
</span>lines </span>= </span>(</span>line.strip() </span>for </span>line </span>in </span>fileinput.input()</span>)
</span>puzzle_input </span>= </span>"\n"</span>.join(lines)
</span>sections </span>= </span>puzzle_input.split(</span>"\n\n"</span>)
</span>solution </span>= </span>max</span>(total_calories(section) </span>for </span>section </span>in </span>sections)
</span>print</span>(</span>f</span>"Part one: </span>{solution}</span>"</span>)
</span>top_three </span>= </span>sorted</span>(total_calories(section) </span>for </span>section </span>in </span>sections)</span>[</span>-</span>3</span>:]
</span>solution </span>= </span>sum</span>(top_three)
</span>print</span>(</span>f</span>"Part two: </span>{solution}</span>"</span>)
</span></code></pre>
Let's run it against the example:</p>
$ python py/day01/sol.py input/01/example.txt
</span>Part one: 24000
</span>Part two: 45000
</span></code></pre>
Neato! We're on a roll.</p>
Now on our input:</p>
$ python py/day01/sol.py input/01/input.txt
</span>Part one: 70509
</span>Part two: 208567
</span></code></pre>
Improvements</h2>
Maybe we can do a little bit better, while also learning a bit more about the
(unnecessarily enormous) Python standard library!</p>
First, let's try to pull our solution into function scope. We'll check to see
if our solution module should be run by comparing the global value __name__</code>
against the string "__main__"</code>. Check out the Modules</a>
section of the Python tutorial for more details.</p>
import </span>fileinput
</span>
</span>
</span>def </span>total_calories</span>(</span>elf</span>):
</span>    </span>return </span>sum</span>(</span>int</span>(item) </span>for </span>item </span>in </span>elf.splitlines())
</span>
</span>
</span>def </span>main</span>():
</span>    lines </span>= </span>(</span>line.strip() </span>for </span>line </span>in </span>fileinput.input()</span>)
</span>    puzzle_input </span>= </span>"\n"</span>.join(lines)
</span>    sections </span>= </span>puzzle_input.split(</span>"\n\n"</span>)
</span>    solution </span>= </span>max</span>(total_calories(section) </span>for </span>section </span>in </span>sections)
</span>    </span>print</span>(</span>f</span>"Part one: </span>{solution}</span>"</span>)
</span>    top_three </span>= </span>sorted</span>(total_calories(section) </span>for </span>section </span>in </span>sections)</span>[</span>-</span>3</span>:]
</span>    solution </span>= </span>sum</span>(top_three)
</span>    </span>print</span>(</span>f</span>"Part two: </span>{solution}</span>"</span>)
</span>
</span>
</span>if </span>__name__ </span>== </span>"__main__"</span>:
</span>    </span>main()
</span></code></pre>
As mentioned above, using fileinput</code> is pretty awkward since we are basically
reconstructing our input from the file contents, anyway. Let's define a helper
module to make reading input a little bit nicer. We'll put this module in the file
2022/py/read_input.py</code>:</p>
import </span>argparse
</span>from </span>pathlib </span>import </span>Path
</span>
</span>
</span>def </span>read_input</span>(</span>day</span>: </span>int</span>) -> </span>str</span>:
</span>    parser </span>= </span>argparse.ArgumentParser()
</span>    </span>parser.add_argument(</span>"input"</span>, nargs</span>=</span>"?"</span>, default</span>=</span>"input.txt"</span>)
</span>    args </span>= </span>parser.parse_args()
</span>
</span>    padded_day </span>= </span>str</span>(day).rjust(</span>2</span>, </span>"0"</span>)
</span>    filename </span>= </span>Path(</span>"input"</span>) </span>/ </span>padded_day </span>/ </span>args.input
</span>
</span>    </span>with </span>open</span>(filename) </span>as </span>f:
</span>        </span>return </span>f.read()
</span></code></pre>
I guess there's a few things to mention, here:</p>

What's that  -> str</code> nonsense?</strong>: That's a return value type hint.
Check out the typing</code></a> module documentation for more information.</li>
The argparse</code> standard library module allows you to write simple CLI scripts. There are a lot of
great argument parsing libraries available on PyPI</a>, but we don't need to pull in any
external dependencies for argparse</code> and it suits our needs just fine.</li>
The pathlib</code> standard library module makes it easy to build up file paths in a
platform-independent way.</li>
</ul>
Now, to use this new module, we also need to add a blank __init__.py</code> file to
our 2022/py</code> directory; and while we're at it, let's add one to our 2022/py/day01</code> directory, as well:</p>
$ touch 2022/py/__init__.py
</span>$ touch 2022/py/day01/__init__.py
</span></code></pre>
This might all seem like a lot of boilerplate, but we won't really touch any of
this after day 1!</p>
Finally, we can use the new module in our solution:</p>
from </span>py.read_input </span>import </span>read_input
</span>
</span>
</span>def </span>total_calories</span>(</span>elf</span>):
</span>    </span>return </span>sum</span>(</span>int</span>(item) </span>for </span>item </span>in </span>elf.splitlines())
</span>
</span>
</span>def </span>main</span>():
</span>    puzzle_input </span>= </span>read_input(</span>1</span>)
</span>    sections </span>= </span>puzzle_input.split(</span>"\n\n"</span>)
</span>    solution </span>= </span>max</span>(total_calories(section) </span>for </span>section </span>in </span>sections)
</span>    </span>print</span>(</span>f</span>"Part one: </span>{solution}</span>"</span>)
</span>    top_three </span>= </span>sorted</span>(total_calories(section) </span>for </span>section </span>in </span>sections)</span>[</span>-</span>3</span>:]
</span>    solution </span>= </span>sum</span>(top_three)
</span>    </span>print</span>(</span>f</span>"Part two: </span>{solution}</span>"</span>)
</span>
</span>
</span>if </span>__name__ </span>== </span>"__main__"</span>:
</span>    </span>main()
</span></code></pre>
To avoid Python path nonsense we need to change our command line invocation
slightly:</p>
$ python -m py.day01.sol
</span>Part one: 70509
</span>Part two: 208567
</span></code></pre>
Still works! And now we don't even need to enter the filename if we just want
to run against our input. Nice.</p>
Type Hints</h3>
I'm a bit of a type hinting fanatic: type hints help me avoid making careless
mistakes and force me to refine the flow of information around my programs.</p>
Let's add them to what we have so far and we can try to keep everything working
as we restructure our program.</p>
First, let's set up a venv</code> within our py</code> subdirectory. I'm using
the python3.11</code> binary installed on my machine, so that looks like this:</p>
$ python3.11 -m venv py/.venv
</span>$ source py/.venv/bin/activate
</span></code></pre>
I personally like to use the pyright</code> LSP, so I'll go ahead and install the
dev tools that I typically use. If you want to follow along, at least install
mypy</code>:</p>
$ pip install pyright flake8 black mypy
</span></code></pre>
Let's run mypy</code> in strict mode:</p>
$ mypy --strict py
</span>py/day01/sol.py:6: error: Function is missing a type annotation  </span>[</span>no</span>-</span>untyped</span>-</span>def</span>]
</span>py/day01/sol.py:10: error: Function is missing a return type annotation  </span>[</span>no</span>-</span>untyped</span>-</span>def</span>]
</span>py/day01/sol.py:10: note: Use </span>"-> None"</span> if function does not return a value
</span>py/day01/sol.py:13: error: Call to untyped function </span>"total_calories"</span> in typed context  </span>[</span>no</span>-</span>untyped</span>-</span>call</span>]
</span>py/day01/sol.py:15: error: Call to untyped function </span>"total_calories"</span> in typed context  </span>[</span>no</span>-</span>untyped</span>-</span>call</span>]
</span>py/day01/sol.py:21: error: Call to untyped function </span>"main"</span> in typed context  </span>[</span>no</span>-</span>untyped</span>-</span>call</span>]
</span>Found 5 errors in 1 file (checked 4 source files</span>)
</span></code></pre>
Paydirt! Let's go ahead and fix all of those. While we are at it, we will also
make main</code> return an integer value so we can be a good command line citizen.</p>
from </span>py.read_input </span>import </span>read_input
</span>
</span>
</span>def </span>total_calories</span>(</span>elf</span>: </span>str</span>) -> </span>int</span>:
</span>    </span>return </span>sum</span>(</span>int</span>(item) </span>for </span>item </span>in </span>elf.splitlines())
</span>
</span>
</span>def </span>main</span>() -> </span>int</span>:
</span>    puzzle_input </span>= </span>read_input(</span>1</span>)
</span>    sections </span>= </span>puzzle_input.split(</span>"\n\n"</span>)
</span>    solution </span>= </span>max</span>(total_calories(section) </span>for </span>section </span>in </span>sections)
</span>    </span>print</span>(</span>f</span>"Part one: </span>{solution}</span>"</span>)
</span>    top_three </span>= </span>sorted</span>(total_calories(section) </span>for </span>section </span>in </span>sections)</span>[</span>-</span>3</span>:]
</span>    solution </span>= </span>sum</span>(top_three)
</span>    </span>print</span>(</span>f</span>"Part two: </span>{solution}</span>"</span>)
</span>    </span>return </span>0
</span>
</span>
</span>if </span>__name__ </span>== </span>"__main__"</span>:
</span>    </span>raise </span>SystemExit</span>(main())
</span></code></pre>
If we run mypy</code> again:</p>
$ mypy --strict py
</span>Success: no issues found in 4 source files
</span></code></pre>
Huzzah!</p>
Now, let's see if we can identify some common ground between parts 1 and 2 and maybe
improve our runtime performance.</p>
Let's consider the runtime complexity of the solution that we currently have. We'll
say that \(I\) is the length of our input and \(N\) is the number of elves in the input.</p>
Our part 1 boils down to a linear scan over our input, so it runs in \(O(I + N)\)
time (and \(N\) is bounded by \(I\), so it's really \(O(I)\)). Our part 2 sorts the collection of elves, so it runs in \(O(I + N \log N)\) time.</p>
Can we do better? Well, when we fetch the top 3 elves, we are discarding a bunch of
information that we spent time assembling when we sorted the entire collection of elves.
Maybe we can avoid doing all that work when we only really care about the top 3
elves. Heaps are a wonderful data structure that work perfectly for this kind of
operation (and they're great to be aware of for all sorts of interview problems).
In particular, we can heapify a list of \(N\) elements in \(O(N)\) time. And then fetching
\(K\) elements from that heap takes \(O(K \log N)\) time. Since we want to fetch a constant number of
elements (\(3\)), the cost of fetching becomes \(O(\log N)\).</p>
So if we can leverage a heap to solve the problem, we will have an \(O(I)\) solution!</p>
There is a very handy module in the standard library called heapq</code></a>.
This library maintains the heap invariant (for a min-heap) atop a vanilla
Python list. It also has a couple of very handy functions: nlargest</code> and nsmallest</code>
for generating lists of the n</code> largest or smallest elements, respectively.
And they do indeed first heapify the input collection and then fetch the first n</code> elements.</p>
And if we look at the documentation, it looks like the result is returned in
reverse sorted order, so we can also use the result for part 1!</p>
Let's see how we can use that and set up a bit of infrastructure for comparing
the new implementation with our original sort-based approach. I've adjusted
our original solution slightly to avoid some duplicate work.</p>
from </span>heapq </span>import </span>nlargest
</span>from </span>time </span>import </span>perf_counter
</span>from </span>typing </span>import </span>Callable
</span>
</span>from </span>py.read_input </span>import </span>read_input
</span>
</span>
</span>def </span>total_calories</span>(</span>elf</span>: </span>str</span>) -> </span>int</span>:
</span>    </span>return </span>sum</span>(</span>int</span>(item) </span>for </span>item </span>in </span>elf.splitlines())
</span>
</span>
</span>def </span>sort_sol</span>(</span>puzzle_input</span>: </span>str</span>) -> </span>tuple</span>[</span>int</span>, </span>int</span>]:
</span>    sections </span>= </span>puzzle_input.split(</span>"\n\n"</span>)
</span>    top_three </span>= </span>sorted</span>(total_calories(section) </span>for </span>section </span>in </span>sections)</span>[</span>-</span>3</span>:]
</span>    </span>return </span>top_three[</span>-</span>1</span>], </span>sum</span>(top_three)
</span>
</span>
</span>def </span>heap_sol</span>(</span>puzzle_input</span>: </span>str</span>) -> </span>tuple</span>[</span>int</span>, </span>int</span>]:
</span>    sections </span>= </span>puzzle_input.split(</span>"\n\n"</span>)
</span>    top_three </span>= </span>nlargest(</span>3</span>, (total_calories(section) </span>for </span>section </span>in </span>sections))
</span>    </span>return </span>top_three[</span>0</span>], </span>sum</span>(top_three)
</span>
</span>
</span>def </span>main</span>() -> </span>int</span>:
</span>    puzzle_input </span>= </span>read_input(</span>1</span>)
</span>    expected_output </span>= </span>sort_sol(puzzle_input)
</span>    </span>print</span>(</span>f</span>"Part one: </span>{expected_output[</span>0</span>]}</span>"</span>)
</span>    </span>print</span>(</span>f</span>"Part two: </span>{expected_output[</span>1</span>]}</span>"</span>)
</span>
</span>    </span>def </span>benchmark</span>(</span>solution</span>: Callable[[</span>str</span>], </span>tuple</span>[</span>int</span>, </span>int</span>]]) -> </span>float</span>:
</span>        start </span>= </span>perf_counter()
</span>        </span>for </span>_ </span>in </span>range</span>(</span>1000</span>)</span>:
</span>            </span>assert </span>solution(puzzle_input) </span>== </span>expected_output
</span>        end </span>= </span>perf_counter()
</span>        </span>return </span>end </span>- </span>start
</span>
</span>    </span>print</span>(</span>f</span>"Sort solution takes </span>{benchmark(sort_sol)}</span>s"</span>)
</span>    </span>print</span>(</span>f</span>"Heap solution takes </span>{benchmark(heap_sol)}</span>s"</span>)
</span>
</span>    </span>return </span>0
</span>
</span>
</span>if </span>__name__ </span>== </span>"__main__"</span>:
</span>    </span>raise </span>SystemExit</span>(main())
</span></code></pre>
Let's run it and see how much improvement we get!</p>
$ python -m py.day01.sol example.txt
</span>Part one: 24000
</span>Part two: 45000
</span>Sort solution takes 0.0033642500020505395s
</span>Heap solution takes 0.0045546660003310535s
</span></code></pre>
Now you might be saying, "hey Sam, I thought using a heap would make things
faster</em>, what gives?"</p>
Well, unfortunately big-O analysis throws away a bunch of constants and it turns
out that in the real world, constants matter. sorted</code> is very well optimized since
it is one of the most commonly used built-ins in Python.</p>
This was just our example input, let's try our real input:</p>
$ python -m py.day01.sol
</span>Part one: 70509
</span>Part two: 208567
</span>Sort solution takes 0.24173837499984074s
</span>Heap solution takes 0.23308020799959195s
</span></code></pre>
Faster, but not by much.</p>
Our puzzle input still only consists of 2237 elves. What if we crank that up
a little bit?</p>
We replace our line puzzle_input = read_input(1)</code> with
puzzle_input = "\n\n".join([read_input(1).strip()] * 1000)</code> and reduce
the number of benchmark iterations down to 10:</p>
$ python -m py.day01.sol
</span>Part one: 70509
</span>Part two: 211527
</span>Sort solution takes 2.4853763329992944s
</span>Heap solution takes 2.3238277089985786s
</span></code></pre>
Better, but still not that satisfying of an improvement. I guess the two
lessons are:</p>

constants matter</li>
log</code> grows very</em> slowly</li>
</ul>
Let's put our heap-based solution in place without all of the benchmarking
boilerplate:</p>
from </span>heapq </span>import </span>nlargest
</span>
</span>from </span>py.read_input </span>import </span>read_input
</span>
</span>
</span>def </span>total_calories</span>(</span>elf</span>: </span>str</span>) -> </span>int</span>:
</span>    </span>return </span>sum</span>(</span>int</span>(item) </span>for </span>item </span>in </span>elf.splitlines())
</span>
</span>
</span>def </span>main</span>() -> </span>int</span>:
</span>    puzzle_input </span>= </span>read_input(</span>1</span>)
</span>    sections </span>= </span>puzzle_input.split(</span>"\n\n"</span>)
</span>    top_three </span>= </span>nlargest(</span>3</span>, (total_calories(section) </span>for </span>section </span>in </span>sections))
</span>    </span>print</span>(</span>f</span>"Part one: </span>{top_three[</span>0</span>]}</span>"</span>)
</span>    </span>print</span>(</span>f</span>"Part two: </span>{</span>sum</span>(top_three)}</span>"</span>)
</span>    </span>return </span>0
</span>
</span>
</span>if </span>__name__ </span>== </span>"__main__"</span>:
</span>    </span>raise </span>SystemExit</span>(main())
</span></code></pre>
So now our directory structure looks like this:</p>
.
</span>└── 2022/
</span>    ├── input/
</span>    │   └── 01/
</span>    │       ├── input.txt
</span>    │       └── example.txt
</span>    ├── py/
</span>    │   ├── __init__.py
</span>    │   ├── read_input.py
</span>    │   └── 01/
</span>    │       ├── __init__.py
</span>    │       └── sol.py
</span>    └── rust
</span></code></pre>
That's enough Python for now, let's see what this looks like in Rust.</p>
Rust</h2>
In Rust, I like to use a new crate for each day. I also like to use
the nightly toolchain so that I can play around with unstable features.</p>
To get started, check out rustup</code></a>.</p>
For now, here is what my active toolchain looks like:</p>
$ rustup show
</span>active toolchain
</span>----------------
</span>
</span>nightly-aarch64-apple-darwin (default</span>)
</span>rustc 1.69.0-nightly (5e37043d6 2023-01-22</span>)
</span></code></pre>
Alright, let's get this show on the road!</p>
$ cargo new rust/day01 --vcs none --lib
</span>$ cd rust/day01
</span></code></pre>
This creates a new binary crate without version control (hopefully you have
version control on your solutions). Your working directory should now look
something like this:</p>
.
</span>└── day01/
</span>    ├── Cargo.toml
</span>    └── src/
</span>        └── lib.rs
</span></code></pre>
And you'll have the default lib.rs</code>:</p>
pub fn </span>add</span>(</span>left</span>: </span>usize</span>, </span>right</span>: </span>usize</span>) -> </span>usize </span>{
</span>    left </span>+</span> right
</span>}
</span>
</span>#[</span>cfg</span>(test)]
</span>mod </span>tests </span>{
</span>    </span>use super</span>::</span>*</span>;
</span>
</span>    #[</span>test</span>]
</span>    </span>fn </span>it_works</span>() {
</span>        </span>let</span> result </span>= </span>add</span>(</span>2</span>, </span>2</span>);
</span>        </span>assert_eq!</span>(result, </span>4</span>);
</span>    }
</span>}
</span></code></pre>
Let's try a similar comparison in Rust between a sort-based and heap-based
solution.</p>
First, let's define a function for the sort-based approach:</p>
use </span>std::error::Error;
</span>
</span>pub fn </span>sort_sol</span>(</span>input</span>: </span>&</span>str</span>) -> </span>Result</span><(</span>i32</span>, </span>i32</span>), </span>Box</span><dyn Error>> {
</span>    </span>let mut</span> elves </span>=</span> input
</span>        .</span>split</span>(</span>"\n\n"</span>)
</span>        .</span>map</span>(|</span>elf</span>| {
</span>            elf.</span>trim</span>()
</span>                .</span>lines</span>()
</span>                .</span>map</span>(|</span>line</span>| -> </span>Result</span><</span>i32</span>, </span>Box</span><dyn Error>> { line.</span>parse</span>().</span>map_err</span>(</span>From</span>::from) })
</span>                .</span>sum</span>()
</span>        })
</span>        .collect::<</span>Result</span><</span>Vec</span><</span>_</span>>, </span>_</span>>>()</span>?</span>;
</span>    elves.</span>sort_by</span>(|</span>a</span>: </span>&</span>i32</span>, </span>b</span>| b.</span>cmp</span>(a));
</span>    </span>Ok</span>((elves[</span>0</span>], elves.</span>iter</span>().</span>take</span>(</span>3</span>).</span>sum</span>()))
</span>}
</span>
</span>#[</span>cfg</span>(test)]
</span>mod </span>tests </span>{
</span>    </span>use super</span>::</span>*</span>;
</span>
</span>    </span>const </span>EXAMPLE</span>: </span>&</span>str </span>= </span>r</span>#"1000
</span>2000
</span>3000
</span>
</span>4000
</span>
</span>5000
</span>6000
</span>
</span>7000
</span>8000
</span>9000
</span>
</span>10000"#</span>;
</span>
</span>    #[</span>test</span>]
</span>    </span>fn </span>test_sort_sol</span>() -> </span>Result</span><(), </span>Box</span><dyn Error>> {
</span>        </span>assert_eq!</span>(</span>sort_sol</span>(</span>EXAMPLE</span>)</span>?</span>, (</span>24000</span>, </span>45000</span>));
</span>        </span>Ok</span>(())
</span>    }
</span>}
</span></code></pre>
This somewhat resembles our Python implementation:</p>

we split the input into elf sections,</li>
we calculate the sum of each section,</li>
we sort the resulting collection and determine our results.</li>
</ul>
For the most part, Rust just makes us be a little bit more explicit about our
error conditions. This solution will still panic for empty inputs, for instance.
But our python solution would throw for lines that don't contain valid integers.</p>
Let's see if our test passes!</p>
$ cargo test
</span>   </span>Compiling day01 v0.1.0 (/Users/sgeisenh/projects/aoc/2022/rust/day01</span>)
</span>    </span>Finished test </span>[</span>unoptimized + debuginfo</span>]</span> target(s</span>) </span>in 0.45s
</span>     </span>Running unittests src/lib.rs (target/debug/deps/day01-1b30f96f7eb97059</span>)
</span>
</span>running 1 test
</span>test tests::test_sort_sol ... ok
</span>
</span>test result: ok. 1 passed</span>; </span>0 failed</span>; </span>0 ignored</span>; </span>0 measured</span>; </span>0 filtered out</span>; </span>finished in 0.00s
</span>
</span>   </span>Doc-tests day01
</span>
</span>running 0 tests
</span>
</span>test result: ok. 0 passed</span>; </span>0 failed</span>; </span>0 ignored</span>; </span>0 measured</span>; </span>0 filtered out</span>; </span>finished in 0.00s
</span></code></pre>
Excellent.</p>
Let's wire it up into a binary by creating a day01/src/bin/main.rs</code>:</p>
use </span>std::error::Error;
</span>
</span>use </span>day01::sort_sol;
</span>
</span>fn </span>main</span>() -> </span>Result</span><(), </span>Box</span><dyn Error>> {
</span>    </span>let</span> input </span>= </span>include_str!</span>(</span>"../../../../input/01/input.txt"</span>);
</span>    </span>let </span>(part1, part2) </span>= </span>sort_sol</span>(input)</span>?</span>;
</span>    </span>println!</span>(</span>"Sort sol p1: </span>{part1}</span>, p2: </span>{part2}</span>"</span>);
</span>    </span>Ok</span>(())
</span>}
</span></code></pre>
Let's run it!</p>
$ cargo run
</span>   </span>Compiling day01 v0.1.0 (/Users/sgeisenh/projects/aoc/2022/rust/day01</span>)
</span>    </span>Finished dev </span>[</span>unoptimized + debuginfo</span>]</span> target(s</span>) </span>in 0.15s
</span>     </span>Running `target/debug/main`
</span>Sort sol p1: 70509, p2: 208567
</span></code></pre>
Cool beans. Let's try to get the heap implementation in place. We can use
the unstable into_iter_sorted</code> function to convert a BinaryHeap</code>
directly to an Iterator</code> and then take the first 3 elements for part 2.</p>
Here's what that looks like:</p>
#![</span>feature</span>(binary_heap_into_iter_sorted)]
</span>use </span>std::{collections::BinaryHeap, error::Error};
</span>
</span>pub fn </span>sort_sol</span>(</span>input</span>: </span>&</span>str</span>) -> </span>Result</span><(</span>i32</span>, </span>i32</span>), </span>Box</span><dyn Error>> {
</span>    </span>let mut</span> elves </span>=</span> input
</span>        .</span>split</span>(</span>"\n\n"</span>)
</span>        .</span>map</span>(|</span>elf</span>| {
</span>            elf.</span>trim</span>()
</span>                .</span>lines</span>()
</span>                .</span>map</span>(|</span>line</span>| -> </span>Result</span><</span>i32</span>, </span>Box</span><dyn Error>> { line.</span>parse</span>().</span>map_err</span>(</span>From</span>::from) })
</span>                .</span>sum</span>()
</span>        })
</span>        .collect::<</span>Result</span><</span>Vec</span><</span>_</span>>, </span>_</span>>>()</span>?</span>;
</span>    elves.</span>sort_by</span>(|</span>a</span>: </span>&</span>i32</span>, </span>b</span>| b.</span>cmp</span>(a));
</span>    </span>Ok</span>((elves[</span>0</span>], elves.</span>iter</span>().</span>take</span>(</span>3</span>).</span>sum</span>()))
</span>}
</span>
</span>pub fn </span>heap_sol</span>(</span>input</span>: </span>&</span>str</span>) -> </span>Result</span><(</span>i32</span>, </span>i32</span>), </span>Box</span><dyn Error>> {
</span>    </span>let</span> elves </span>=</span> input
</span>        .</span>split</span>(</span>"\n\n"</span>)
</span>        .</span>map</span>(|</span>elf</span>| {
</span>            elf.</span>trim</span>()
</span>                .</span>lines</span>()
</span>                .</span>map</span>(|</span>line</span>| -> </span>Result</span><</span>i32</span>, </span>Box</span><dyn Error>> { line.</span>parse</span>().</span>map_err</span>(</span>From</span>::from) })
</span>                .</span>sum</span>()
</span>        })
</span>        .collect::<</span>Result</span><BinaryHeap<</span>_</span>>, </span>_</span>>>()</span>?</span>;
</span>    </span>Ok</span>((
</span>        </span>*</span>elves.</span>peek</span>().</span>unwrap</span>(),
</span>        elves.</span>into_iter_sorted</span>().</span>take</span>(</span>3</span>).</span>sum</span>(),
</span>    ))
</span>}
</span>
</span>#[</span>cfg</span>(test)]
</span>mod </span>tests </span>{
</span>    </span>use super</span>::</span>*</span>;
</span>
</span>    </span>const </span>EXAMPLE</span>: </span>&</span>str </span>= </span>r</span>#"1000
</span>2000
</span>3000
</span>
</span>4000
</span>
</span>5000
</span>6000
</span>
</span>7000
</span>8000
</span>9000
</span>
</span>10000"#</span>;
</span>
</span>    #[</span>test</span>]
</span>    </span>fn </span>test_sort_sol</span>() -> </span>Result</span><(), </span>Box</span><dyn Error>> {
</span>        </span>assert_eq!</span>(</span>sort_sol</span>(</span>EXAMPLE</span>)</span>?</span>, (</span>24000</span>, </span>45000</span>));
</span>        </span>Ok</span>(())
</span>    }
</span>
</span>    #[</span>test</span>]
</span>    </span>fn </span>test_heap_sol</span>() -> </span>Result</span><(), </span>Box</span><dyn Error>> {
</span>        </span>assert_eq!</span>(</span>heap_sol</span>(</span>EXAMPLE</span>)</span>?</span>, (</span>24000</span>, </span>45000</span>));
</span>        </span>Ok</span>(())
</span>    }
</span>}
</span></code></pre>
We run the tests...</p>
$ cargo test
</span>   </span>Compiling day01 v0.1.0 (/Users/sgeisenh/projects/aoc/2022/rust/day01</span>)
</span>    </span>Finished test </span>[</span>unoptimized + debuginfo</span>]</span> target(s</span>) </span>in 0.23s
</span>     </span>Running unittests src/lib.rs (target/debug/deps/day01-1b30f96f7eb97059</span>)
</span>
</span>running 2 tests
</span>test tests::test_heap_sol ... ok
</span>test tests::test_sort_sol ... ok
</span>
</span>test result: ok. 2 passed</span>; </span>0 failed</span>; </span>0 ignored</span>; </span>0 measured</span>; </span>0 filtered out</span>; </span>finished in 0.00s
</span>
</span>     </span>Running unittests src/bin/main.rs (target/debug/deps/main-abe929b1b2534503</span>)
</span>
</span>running 0 tests
</span>
</span>test result: ok. 0 passed</span>; </span>0 failed</span>; </span>0 ignored</span>; </span>0 measured</span>; </span>0 filtered out</span>; </span>finished in 0.00s
</span>
</span>   </span>Doc-tests day01
</span>
</span>running 0 tests
</span>
</span>test result: ok. 0 passed</span>; </span>0 failed</span>; </span>0 ignored</span>; </span>0 measured</span>; </span>0 filtered out</span>; </span>finished in 0.00s
</span>
</span></code></pre>
wire it up to our binary...</p>
use </span>std::error::Error;
</span>
</span>use </span>day01::{heap_sol, sort_sol};
</span>
</span>fn </span>main</span>() -> </span>Result</span><(), </span>Box</span><dyn Error>> {
</span>    </span>let</span> input </span>= </span>include_str!</span>(</span>"../../../../input/01/input.txt"</span>);
</span>    </span>let </span>(sp1, sp2) </span>= </span>sort_sol</span>(input)</span>?</span>;
</span>    </span>println!</span>(</span>"Sort sol p1: </span>{sp1}</span>, p2: </span>{sp2}</span>"</span>);
</span>    </span>let </span>(hp1, hp2) </span>= </span>heap_sol</span>(input)</span>?</span>;
</span>    </span>println!</span>(</span>"Heap sol p1: </span>{hp1}</span>, p2: </span>{hp2}</span>"</span>);
</span>    </span>Ok</span>(())
</span>}
</span></code></pre>
and run it:</p>
$ cargo run
</span>    </span>Finished dev </span>[</span>unoptimized + debuginfo</span>]</span> target(s</span>) </span>in 0.01s
</span>     </span>Running `/Users/sgeisenh/projects/aoc/2022/rust/day01/target/debug/main`
</span>Sort sol p1: 70509, p2: 208567
</span>Heap sol p1: 70509, p2: 208567
</span></code></pre>
Neat.</p>
There's a little bit of common code that I feel like we should be able to pull
into it's own function. Let's give it a shot!</p>
#![</span>feature</span>(binary_heap_into_iter_sorted)]
</span>use </span>std::{collections::BinaryHeap, error::Error};
</span>
</span>pub type </span>AocResult</span><</span>'a</span>, T</span>> = </span>Result</span><T, </span>Box</span><dyn Error </span>+ </span>'a</span>>>;
</span>
</span>fn </span>parse_input</span>(</span>input</span>: </span>&</span>str</span>) -> impl </span>Iterator</span><Item = AocResult<</span>i32</span>>> {
</span>    input.</span>split</span>(</span>"\n\n"</span>).</span>map</span>(|</span>elf</span>| {
</span>        elf.</span>trim</span>()
</span>            .</span>lines</span>()
</span>            .</span>map</span>(|</span>line</span>| -> AocResult<</span>i32</span>> { line.</span>parse</span>().</span>map_err</span>(</span>From</span>::from) })
</span>            .</span>sum</span>()
</span>    })
</span>}
</span>
</span>pub fn </span>sort_sol</span>(</span>input</span>: </span>&</span>str</span>) -> AocResult<(</span>i32</span>, </span>i32</span>)> {
</span>    </span>let mut</span> elves </span>= </span>parse_input</span>(input).collect::<</span>Result</span><</span>Vec</span><</span>_</span>>, </span>_</span>>>()</span>?</span>;
</span>    elves.</span>sort_by</span>(|</span>a</span>: </span>&</span>i32</span>, </span>b</span>| b.</span>cmp</span>(a));
</span>    </span>Ok</span>((elves[</span>0</span>], elves.</span>iter</span>().</span>take</span>(</span>3</span>).</span>sum</span>()))
</span>}
</span>
</span>pub fn </span>heap_sol</span>(</span>input</span>: </span>&</span>str</span>) -> AocResult<(</span>i32</span>, </span>i32</span>)> {
</span>    </span>let</span> elves </span>= </span>parse_input</span>(input).collect::<</span>Result</span><BinaryHeap<</span>_</span>>, </span>_</span>>>()</span>?</span>;
</span>    </span>Ok</span>((
</span>        </span>*</span>elves.</span>peek</span>().</span>unwrap</span>(),
</span>        elves.</span>into_iter_sorted</span>().</span>take</span>(</span>3</span>).</span>sum</span>(),
</span>    ))
</span>}
</span>
</span>#[</span>cfg</span>(test)]
</span>mod </span>tests </span>{
</span>    </span>use super</span>::</span>*</span>;
</span>
</span>    </span>const </span>EXAMPLE</span>: </span>&</span>str </span>= </span>r</span>#"1000
</span>2000
</span>3000
</span>
</span>4000
</span>
</span>5000
</span>6000
</span>
</span>7000
</span>8000
</span>9000
</span>
</span>10000"#</span>;
</span>
</span>    #[</span>test</span>]
</span>    </span>fn </span>test_parse_input</span>() -> AocResult<</span>'static</span>, ()> {
</span>        </span>assert_eq!</span>(
</span>            </span>parse_input</span>(</span>EXAMPLE</span>).collect::<AocResult<</span>Vec</span><</span>_</span>>>>()</span>?</span>,
</span>            </span>vec!</span>[</span>6000</span>, </span>4000</span>, </span>11000</span>, </span>24000</span>, </span>10000</span>]
</span>        );
</span>        </span>Ok</span>(())
</span>    }
</span>
</span>    #[</span>test</span>]
</span>    </span>fn </span>test_sort_sol</span>() -> AocResult<</span>'static</span>, ()> {
</span>        </span>assert_eq!</span>(</span>sort_sol</span>(</span>EXAMPLE</span>)</span>?</span>, (</span>24000</span>, </span>45000</span>));
</span>        </span>Ok</span>(())
</span>    }
</span>
</span>    #[</span>test</span>]
</span>    </span>fn </span>test_heap_sol</span>() -> AocResult<</span>'static</span>, ()> {
</span>        </span>assert_eq!</span>(</span>heap_sol</span>(</span>EXAMPLE</span>)</span>?</span>, (</span>24000</span>, </span>45000</span>));
</span>        </span>Ok</span>(())
</span>    }
</span>}
</span>
</span></code></pre>
Now it's time to see how our solutions compare.</p>
For that we'll use Criterion.rs</a>.</p>
First we install criterion</code>:</p>
$ cargo add criterion
</span></code></pre>
Then we add a benchmark to our Cargo.toml</code>:</p>
[</span>dev-dependencies</span>]
</span>criterion </span>= { </span>version </span>= </span>"0.4.0"</span>, </span>features </span>= [</span>"html_reports"</span>] }
</span>
</span>
</span>[[</span>bench</span>]]
</span>name </span>= </span>"my_benchmark"
</span>harness </span>= </span>false
</span></code></pre>
And add our benchmark file benches/my_benchmark.rs</code>:</p>
use </span>std::hint::black_box;
</span>
</span>use </span>criterion::{criterion_group, criterion_main, Criterion};
</span>
</span>static </span>INPUT</span>: </span>&</span>str </span>= </span>include_str!</span>(</span>"../../../input/01/input.txt"</span>);
</span>
</span>pub fn </span>criterion_benchmark</span>(</span>c</span>: </span>&</span>mut</span> Criterion) {
</span>    c.</span>bench_function</span>(</span>"sort_sol"</span>, |</span>b</span>| b.</span>iter</span>(|| day01::sort_sol(</span>black_box</span>(</span>INPUT</span>))));
</span>    c.</span>bench_function</span>(</span>"heap_sol"</span>, |</span>b</span>| b.</span>iter</span>(|| day01::heap_sol(</span>black_box</span>(</span>INPUT</span>))));
</span>}
</span>
</span>criterion_group!</span>(benches, criterion_benchmark);
</span>criterion_main!</span>(benches);
</span></code></pre>
And now we run our benchmarks:</p>
$ cargo bench
</span>...
</span>     </span>Running unittests src/lib.rs (target/release/deps/day01-aec6c446d7e6dd94</span>)
</span>
</span>running 3 tests
</span>test tests::test_heap_sol ... ignored
</span>test tests::test_parse_input ... ignored
</span>test tests::test_sort_sol ... ignored
</span>
</span>test result: ok. 0 passed</span>; </span>0 failed</span>; </span>3 ignored</span>; </span>0 measured</span>; </span>0 filtered out</span>; </span>finished in 0.00s
</span>
</span>     </span>Running unittests src/bin/main.rs (target/release/deps/main-cc52451e5c082913</span>)
</span>
</span>running 0 tests
</span>
</span>test result: ok. 0 passed</span>; </span>0 failed</span>; </span>0 ignored</span>; </span>0 measured</span>; </span>0 filtered out</span>; </span>finished in 0.00s
</span>
</span>     </span>Running benches/my_benchmark.rs (target/release/deps/my_benchmark-710fed971563135d</span>)
</span>Gnuplot not found, using plotters backend
</span>sort_sol                time:   </span>[</span>35.656 µs 35.678 µs 35.699 µs</span>]
</span>                        </span>change: </span>[</span>-0.1767% </span>-</span>0.0397% +0.1173%</span>]</span> (p = 0.60 </span>> </span>0</span>.05</span>)
</span>                        </span>No change in performance detected.
</span>Found 5 outliers among 100 measurements (5.00%</span>)
</span>  </span>5 (5.00%</span>) </span>high severe
</span>
</span>heap_sol                time:   </span>[</span>32.523 µs 32.553 µs 32.601 µs</span>]
</span>                        </span>change: </span>[</span>-0.3934% </span>-</span>0.2565% </span>-</span>0.1198%</span>]</span> (p = 0.00 </span>< </span>0</span>.05</span>)
</span>                        </span>Change within noise threshold.
</span>Found 6 outliers among 100 measurements (6.00%</span>)
</span>  </span>2 (2.00%</span>) </span>high mild
</span>  </span>4 (4.00%</span>) </span>high severe
</span></code></pre>
Nice, our heap solution is slightly faster than our sort solution. Also, 32 microseconds</em>
is pretty quick... so quick that it's not even really worth benchmarking our solution with hyperfine</a>.
We'll save that for another time.</p>
That's it!</h2>
That's all for now. Now that most of the boilerplate is out of the way, I'm hopeful that
the next several days will go much quicker! See you next time.</p>

Day 3 at RC

2023-01-06T00:00:00+00:00

Day 3 was really packed. Here's a quick summary before I jump in:

I started my day by writing down my reflections on day 2 (yesterday's post).</li>

Pairing session on a 2048 implementation in python.</li>

Heard from a bunch of folks about their plans/advice at RC.</li>

Paired on a LeetCode problem in JavaScript.</li>

Responded to a creative coding prompt in p5js as part of a group.</li>

Paired some more on a neat Pokémon image downloader in async rust.</li>

Had a coffee chat with an RC alum to talk about some of our experience in the programming language sphere.</li>

Had a really interesting discussion about reusing some open source C++ code to create an LSP implementation in Rust by bridging the gap with cxx</a>.</li>
Dove into some of the cxx documentation.</li>
Watched the first half of Jon Gjengset's The What and How of Futures and async/await in Rust</a> video.</li> </ul>
The rest of this post will be some of my thoughts during day 3.
Reflecting in the morning</h2>
I think I like this routine of beginning each day by reflecting on the previous day. So I'm doing it again today!
I haven't done a great job of writing down these kinds of reflections professionally. I don't really know why that is. Maybe it's because I'm not all that comfortable being vulnerable at my place of work? Regardless, I'd like to care a lot less about others' perception of me and my work and focus more on creating things that I'm proud of or excited about. So far writing down some of my thoughts has really helped clarify how I'm feeling.
2048</h2>
This morning pairing session was great. The dialogue felt pretty good and there was a nice exchange of thoughts back and forth while producing a good amount of code. We made a solid amount of progress in a little over an hour and had a mostly functional version of 2048 playable in the terminal. We planned another pairing session around wiring up a simple AI to see how well it might perform.
I'd love to work on a more in-depth writeup about this at some point in the future.
LeetCode</h2>
I really gravitate toward LeetCode-style problems. In some sense, I find it really satisfying to come up with an efficient solution to a problem. But on the other hand, I don't know that spending time on LeetCode is "working at the edge of my abilities" (one of the self directives</a> at RC). I've spent a lot of time working on competitive coding prompts like Advent of Code, Codeforces and LeetCode. I'm not the best that I could be -- I certainly don't crank out LC hards -- but it's comfortable for me.
I'll probably try to fight the temptation to spend a lot of time on this style of programming throughout the rest of my batch.
Creative Coding</h2>
I had a lot of fun navigating on a creative coding prompt. We built an interactive scene where an emoji person was trying to decide on which outfit to wear. It was a great reminder that programming doesn't have to be serious. I think these sessions will be especially good as a foil for solving really complex technical challenges. I've historically leaned on playing trumpet for this kind of release. But when I really want to focus on growing as a programmer for a few months, creative coding sessions with a group of folks seems pretty great.
I mentioned in an earlier post that I struggle with some of the improvisational skills that are sometimes called upon for activities like this. The group setting felt pretty nice and it was a lot of fun to think about ways to structure the scene we built.
Rust</h2>
Rust seems to be everywhere. There are a bunch of folks in my batch that seem really interested in learning Rust which is really exciting.
For a low-level language, Rust has a really compelling feedback loop. Top-notch developer tooling (at least compared with clangd), built-in testing functionality and a package management ecosystem that makes it easy to focus on the problem you want to solve instead of how to repackage an open source library as a Bazel library instead of using CMake (I'm not salty at all...).
And the community is fantastic. There are so many resources for learning and most of them are fairly up-to-date. I think C++ is simply too large of a language for most resources to be broadly applicable. Recently, I have enjoyed Klaus Iglberger's CppCon talks on YouTube; and I'm curious about his recently published book C++ Software Design</a>. But it gets really frustrating to see recommendations for pasting over decades-old language design mistakes by adding yet another keyword or annotation to your method declaration. Rust corrects a lot of these mistakes and incorporates better mechanisms for encoding invariants in programs. In many ways, Rust is now one of the most accessible languages with explicit memory management: you get best-practices by default instead of opting into them with a keyword.
I'm sure I'll write a lot more about Rust in the coming weeks so I'll leave it at that, for now.
Closing Thoughts</h2>
I've been spending a lot of time in group activities this week. I think I'm developing a stronger sense for how I want to spend my time during the rest of my batch. I definitely want to continue with these daily reflections. Thanks again for reading!

Day 2 at RC

2023-01-05T00:00:00+00:00

Day 2 of RC was really interesting. The biggest event of the day was a pairing workshop. But I also attended a couple of group events:

a discussion of WebGL,</li>

the start of a reading group for Nand2Tetris.</li> </ul>
And then in the evening I attended a couple of non-programming talks.
I also finally finished Protohackers day 6 in Rust which felt really good.
Pairing Workshop</h2>
I'm really interested in pair programming but I haven't done much of it. I have watched a good number of Rubber Duck Engineering</a> streams and thought "yeah, that seems like some of the most fun I can imagine". The workshop brought me back to earth a little bit: I don't think I am very good at pairing. That being said, I really want to do as much pairing as possible over the next few months.
At the beginning of the workshop, some of the RC staff discussed the benefits of pairing and provided an example of what effective pairing might look like. We then split off into groups to work on implementing Mastermind. As we jumped into groups, I found myself in a group of three. I ended up driving to start out. There are 2 roles in pair programming: the driver who types and the navigator who follows along. Driving is hard. From experiences that I've had, it feels most similar to taking a coding interview with a bit more communication. I got a little carried away and ended up taking 30 minutes instead of 20 minutes for my section of driving. When we switched off, I discovered that navigating is at least as hard as driving. Finding the right time to mention something to the driver is tough and I can get a little bit fixated on tiny issues (as my wife would surely attest). Overall, this was a really valuable session and I'm looking forward to working through some of my discomfort with pairing over the coming weeks.
As an aside, node's blocking-averse design made it surprisingly difficult to read input from the user. Especially when compared with python's input</code>.
WebGL</h2> I've always been kind of curious about graphics programming, but I consistently bounce off. I think the gap between "wow, this is a triangle with a gradient" to "holy smokes, a hand-written shader for a high-fidelity 3D flying sequence" seems insurmountable. I've also always struggled with extemporaneous creativity. Whenever I was called on to improvise in jazz band, I would clam up immediately. I'm curious how I might be able to work through some of my "improv anxiety" and come up with silly ideas to try that might help me work through some of my misgivings with shaders. We'll see.Nand2Tetris</h2> I recently read Charles Petzold's Code</a> which I thoroughly enjoyed. Slowly building up a stack of abstractions starting from sending signals between neighboring houses was really neat. I bought The Elements of Computing Systems</a> a while ago, but haven't done much aside from the first couple of chapters. Working through the book seems like it might be a fun way to bolster my mental model of hardware and the hardware/software interface. Non-programming Talks</h2> These talks were great. I learned a lot about tabletop RPGs and chairmaking. Most of all, it was further proof that the people at RC are awesome. Protohackers day 6 in Rust</h2> It was really neat to finally finish protohackers day 6 in Rust link</a>. I'm not handling connections the best and I could definitely use some logging to make debugging easier. But the automated tests passed, and a heck of a lot faster than my python implementation, at that! That's all for now</h2> Most of these will probably be quick updates with maybe a more in-depth update once every week or two. Thanks for reading!

First day at the Recurse Center

2023-01-03T00:00:00+00:00

Today was my first day at the Recurse Center</a> (RC). I had so much fun hearing about what awaits me over the next ~12 weeks and meeting a bunch of new people, all of whom seem to be just about as excited as I am!

A lot of the discussions I had with folks revolved around what everyone was planning to work on during the coming weeks. I've been curious about a handful of programming topics recently so I have some initial ideas for things to explore:

I've been working my way through Robert Nystrom's Crafting Interpreters</a> and I have a couple of ideas for different directions to take.</li>
Andy Pavlo is a great speaker and writer and I'd love to dive into some of the course material he has put together on database systems</a>.</li>
Andreas Kling</a> is my go-to for relaxing and satisfying YouTube viewing. It would be a blast to contribute to SerenityOS in some way.</li> </ul>
There are also a bunch of reading groups and other interesting gatherings to attend. I'll probably attend lots of stuff early in the batch and see what sticks.
Most of all, regardless of what I am working on, I want to make a real effort to communicate about it. I'd love to improve my writing and communication skills significantly over the next few months.
In the rest of this post, I'll dive into some of the things I've been thinking about.
Joy in Programming</h2>
I love programming. I get a deep sense of satisfaction watching a program that I've written solve a problem. I also love the craft of programming. There are so many different ways to solve problems with code. Which way is the most expressive? Which techniques help reduce or eliminate bugs? I'm constantly thinking about these sorts of things whether solving an Advent of Code problem or starting a design document at work.
Often when I've started conversations about the practice of programming at work, I've received blank stares from many of my coworkers. This has been pretty discouraging. From everything I've read and everything I've seen so far, RC is the kind of place where these kinds of discussions are met with curiosity from the people around you. Whether they are experts in a particular programming topic or looking to learn more.
The summer after I graduated, I attended the Oregon Programming Languages Summer School</a> (OPLSS). Everyone there was psyched to exchange ideas about programming languages across the full spectrum from theory to practice. That was the last time that I remember being surrounded by a group of people so enthusiastic about the subject at hand.
I really enjoy programming as a career. But necessarily, programming and the practice of programming is a means to an end in a business context. There is a lot of satisfaction to be had in generating business value, but sometimes the drive to produce impact can quell some natural curiosity. There isn't always time to peek under the hood and learn about the different systems and abstractions that we lean on to get our work done, every single day.
This is what I'm most excited about being at RC. To have the time and space to explore those nooks and crannies and muse about it with a bunch of likeminded people.
High-level Programming Language Runtimes</h2> A lot of my background in the programming language space comes from my undergraduate curriculum. I loved the PL classes in school, but there was a heavy emphasis on static type systems and theory in general. Reading and working through Crafting Interpreters has been a blast. High-level dynamic languages were generally derided (example</a>) during my school years. Writing a good amount of code in python and JavaScript over the years has me convinced that dynamic languages have their place, though I still generally prefer to have a powerful static type system.
I'm really fascinated by all of the efforts that have gone into creating absurdly fast language runtimes over the last couple of decades. I'm hoping to do some reading starting with the smalltalk and Self optimization papers of the 80s and 90s, working my way through LuaJIT in the mid 2000s and V8 and pypy in the late 2000s.
Ideally, I'd be able to incorporate some of those ideas into a Lox interpreter, but that might be overly ambitious. We'll see!
Somewhat unrelated, I've been running into an issue where a C++ implementation of Lox that I've been working on runs ~100% slower than clox on a simple for</code> loop example. Most of the time seems to be spent in std::vector</code> reads. Not totally sure what's going on, but it'd be fun to dive deeper and do a write up.
Protohackers, Rust and Python</h2> Since I wasn't sure how much time I would have amidst the orientation events, this week, I've lined up some bite-sized tasks to tackle.I started working on protohackers</a> at the end of August. I had started in Rust which has been a ton of fun. When I reached problem 6 (Speed Daemon), I stalled a little bit as there was more synchronization required than the previous problems. I made a couple of adjustments with the hope that I might be able to make more sustainable progress: Swapped from DigitalOcean</a> to fly.io</a></li> Solved the problem in Python instead of Rust</li> </ul> DO to fly</h3> DigitalOcean is great. It's super simple to spin up a tiny VM, ssh into it, grab the latest copy of my code and run my program. I've been curious to get better familiarized with fly.io and similar hosting providers. For the most part, deploying via a Dockerfile</a> is pretty straightforward. I encountered two issues: I couldn't get UDP to work; protohackers problem 4 specifies a key-value store accessed over UDP. I tried following the fly.io guide on UDP</a> but gave up before getting anything to work.</li> The default concurrency settings caused my service to reject some connections when testing my solution. The auto-generated fly.toml</code> file I was using included this snippet:</li> </ul> [services.concurrency] type = "connections" hard_limit = 25 soft_limit = 20 </code></pre> For a while, I thought I had a logic error in my program that was causing me to close connections before I found a log entry from the proxy layer that indicated the concurrent connections were being limited. Eliminating the snippet from fly.toml</code> had no effect, but replacing those lines with: [services.concurrency] type = "connections" hard_limit = 150 soft_limit = 100 </code></pre> seemed to fix the issues. Overall, fly.io is really pleasant to use. I don't have a lot of experience using Heroku which seems to be the main comparison for fly. But the ability to launch a service anywhere in the world from my command line is pretty neat. It's still quite a bit faster to iterate over ssh, but I'm hoping I can find ways to optimize my workflow. Rust to Python</h3> I'm pretty much head-over-heels for Rust. It's a really satisfying combination of ideas from ML</a> and C++. That being said, there is a lot to think about when dealing with concurrency in Rust. Sometimes you need to wrap something in Arc<Mutex<...>></code> to provide mutable access to some shared state across threads. Tokio</a> is fantastic, but you still need to think about multi-threading and synchronization. Python's asyncio</code> offers a programming model that is conceptually similar to Rust's asynchronous runtimes but without the need for synchronization (since programs that leverage asyncio</code> run the event loop in a single thread). It took me a surprisingly long time to get an implementation working in python. I don't have a ton of experience dealing with binary encodings in python, so that took some adjustment. On the asyncio</code> side of things, it took me a while to figure out that I might want to use readexactly</code></a> for the implementation of reading a single message from a client. In general, I think I have an easier time discovering python documentation but an easier time reading Rust documentation once I've found it. I also made the embarassing mistake of forgetting to explicitly set flush=True</code> for my print</code> debugging. I spent far too long trying to figure out why my service didn't seem to start. All-in-all, it was really satisfying to watch the last test case pass and I'm excited to keep working on the rest of the problems that I haven't solved yet. Maybe I'll find a chance to do a writeup on solving the problems in python and Rust at some point during my batch. Eloquent JavaScript</h2> I love reading programming language books and I had seen lots of recommendations for Eloquent JavaScript</a> by Marijn Haverbeke in the past. I am currently reading Chapter 16 which discusses the implementation of a small platformer using the DOM for rendering the game world. The book is great. I really enjoy the general style (a practical mix of functional and imperative) that Marijn uses in his examples. He also uses a lot of closures, which I think are the most powerful and expressive feature of high-level garbage collected languages. But so far, the most interesting bits from the book have been general software engineering concepts. Chapter 10 discussing modules opens with this gem: A typical real program grows organically. New pieces of functionality are added as new needs come up. Structuring--and preserving structure--is additional work. It's work that will pay off only in the future, the next time someone works on the program. So it is tempting to neglect it and allow the parts of the program to become deeply entangled. </blockquote> Marijn continues describing the issues of this sort of "entaglement". I thought it was a really empathetic way of describing how some programs accrue technical debt. On the other side of the coin, there is a section in chapter 16 labeled "Encapsulation as a Burden". Marijn discusses the tradeoffs involved when choosing where to introduce abstraction in your programs. This is a subtle topic that takes a lot of experience to master (I certainly haven't mastered it yet). But Marijn makes convincing arguments and plants the seed in the reader's mind. Overall, it's been a great read so far! I'm excited to finish it up and maybe do some JavaScript pairing. Wrap Up</h2> If you've made it this far, thanks for reading! Stay tuned to hear more about my experience at RC.

Getting Started With Advent of Code

2022-11-29T00:00:00+00:00

NOTE: This is my second post discussing Advent of Code. Check out the first post</a> to learn more about why I love AoC. </blockquote>

Advent of Code is only a day away! I can't wait to get started. If you're thinking about trying Advent of Code but you're not sure where to start, I hope I can share some tips.

First and foremost, make sure you've signed up at adventofcode.com</a>. Each day, a new puzzle will become available at midnight. Now that you've signed up, let's talk about how to start writing some code to solve each puzzle.

It's often difficult to start on any programming project. The number of options and approaches can feel overwhelming. I definitely experience this sensation of overwhelm when starting a project. Over the years, I've learned strategies for overcoming some of the anxious thoughts that can creep in when just getting started with something. The strategy I come back to time after time is to start with the simplest approach that works.

This is the strategy that I recommend for jumping into Advent of Code. In this post, I'll demonstrate two simple templates that should make it fairly easy to get started solving Advent of Code problems.

There are other resources that suggest templates that will automatically download your input and submit your output. Or that will even show performance benchmarks for each of your solutions. These are great, but I think it's even more fun to start simple and introduce your own features as you get more comfortable using a basic setup.

If you want to check out a great example of a more advanced setup, Anthony Sottile posted a great video</a> demonstrating his template for solving AoC quickly.

Python</h2>
I love solving the problems with python each year. For small puzzles like those in AoC, python gives you a lot of power with very little ceremony.
For a basic setup, I recommend using the fileinput</code></a> module from the standard library. This library takes care of all of the boilerplate of loading input files and makes it easy to swap in a sample input if you want to debug your solution on a simpler example.
Usually, I start with something like this:
import fileinput lines = [line.strip() for line in fileinput.input()] </code></pre> If I've saved my solution to a file named sol.py</code> and my input to a file named input.txt</code>, then I can run my solution by running the command:
python3 -m sol input.txt </code></pre> Let's see how it looks to solve day 1 part 1 from the 2021 edition of Advent of Code.import fileinput -lines = [line.strip() for line in fileinput.input()] +depths = [int(line.strip()) for line in fileinput.input()] + + +result = 0 +for i in range(len(depths) - 1): + if depths[i] < depths[i + 1]: + result += 1 +print("Part one:", result) </code></pre> As you can see, most of the adjustments that we've made are focused on just solving the specifics of the problem. First we parse an integer from each line, then we are able to walk our list to figure out how many times the depth increases. JavaScript</h1> JavaScript is a great language for getting started learning to code especially because the web is such a ubiquitous platform. There are several excellent JavaScript runtimes that you can use to solve AoC problems in JavaScript. I recommend node.js</a> as it is the most popular server-side JavaScript platform. Unfortunately, node does not provide quite as much convenience as python (and its enormous standard library). But once you get past the boilerplate setup, it's not so bad! Let's take a look: const fs = require("fs"); const input = fs.readFileSync("./input.txt").toString(); const lines = input.split("\n").map((s) => s.trim()); </code></pre> To run this, assuming that I've saved this to sol.js</code> (and the input to input.txt</code>) I can run the command: node sol.js </code></pre> Note that we've hardcoded the path of the input file, so you'll have to do a little bit more work to test out your solution on a sample input. It's possible to recreate our setup from the python section with a little bit of research on command line argument parsing (see the docs on process.argv</code></a> for more details). Now let's see what it looks like to solve 2021 day 1 part 1 in JavaScript: const fs = require("fs"); const input = fs.readFileSync("./input.txt").toString(); -const lines = input.split("\n").map((s) => s.trim()); +const depths = input.split("\n").map((s) => parseInt(s.trim())); + +let result = 0; +for (let i = 0; i < depths.length - 1; ++i) { + if (depths[i] < depths[i + 1]) { + ++result; + } +} +console.log("Part one:", result); </code></pre> Not too bad! Wrap Up</h2> As you can see, it's possible to get started solving Advent of Code puzzles with just a little bit of setup. Once you've solved your first few puzzles, it can be fun to experiment with different ways to speed up your solutions or to automatically handle fetching your input or submitting your output. But it's fine to start simple! Hopefully you'll join me tomorrow night (and the rest of the month) in solving some festive puzzles! Good luck!

Why I love Advent of Code

2022-11-15T00:00:00+00:00

NOTE: This will be the first post in a short series of maybe 2 or 3 posts discussing Advent of Code. </blockquote>
Advent of Code</a> is one of my favorite "events" each year. In case you haven't heard of it, Advent of Code (AoC) is an Advent calendar that consists of 25 programming puzzles of varying levels of difficulty. Each puzzle consists of two parts: the first part is released at 12 AM EST each day and the second part of each puzzle is only unlocked upon completing the first part.
Here are a few reasons why I think AoC is great:

Community</li>
Progressive difficulty</li>
Theming</li> </ul>
Community</h2>
More than almost any other programming competition, AoC fosters a wonderful supportive community.
There is a fantastic subreddit: old.reddit.com/r/adventofcode/</a> devoted to AoC with a daily thread for participants to share their solutions. And there are often fun posts with visualizations or other AoC themed content. There are great discussions of why different approaches might work better than others or helping out with tricky bits of the puzzles.
From an official perspective, there is a very competitive main leaderboard available for folks to compete for the fastest solutions each day. But you can also create and join private leaderboards to have a friendly competition among friends and/or colleagues. Many of the top scorers on the main leaderboard have YouTube channels and/or public repositories to display their solutions.
Here are a couple of my favorite YouTube channels for watching rapid solutions:

Jonathan Paulson</a>: Consistently close to the top of the leaderboards, I've learned a good bit about rapidly coding and debugging in python and Jonathan usually includes a walkthrough of the puzzle and solution at the end of each video.</li>
Neal Wu</a>: Neal produces solutions in C++ incredibly quickly with a satisfyingly clicky keyboard. Neal also has videos of himself competing in (and often winning) several other programming competitions.</li> </ul>
There are also many channels that have videos walking through solutions from a pedagogical perspective:

Anthony Sottile</a>: Great educational python videos and usually creates a playlist with AoC solutions each year.</li>
Liz Fong-Jones</a>: SRE and observability guru who solves AoC on stream each year and posts the recordings to YouTube. Solved 2021 in Go.</li> </ul>
And there are many channels with solutions in... unconventional styles:

Mitchell Hashimoto</a>: The founder of HashiCorp produced solutions to days 1-7 of AoC 2021 in PostgreSQL.</li>
Mark Gritter</a>: An engineer at Akita Software wrote solutions to all of 2021 in F*</a>!</li> </ul>
There are so many fun variations on AoC solutions to find and explore.
Progressive difficulty</h2>
AoC usually starts out fairly simple and works up to a pretty good difficulty, usually reaching close to the peak around day 15. This makes AoC an ideal candidate for learning! Whether you're trying out a new programming language or a new library or style, AoC is a fantastic resource to push yourself to get familiar with your tools, new and old.
Since the puzzle inputs are provided as plain text, the early puzzles emphasize basic string processing techniques in your chosen language. Even as a professional programmer, it's really helpful to get more comfortable with the standard library from time to time. I typically try to solve each puzzle as quickly as possible in python and then jump to a programming language that I want to know better. Usually, I spend the early puzzles getting familiarized with the standard library in the second language before the difficult really ramps up. Once the puzzles get more challenging, it's a good opportunity to see how problem solving approaches differ across different programming languages.
Since the early puzzles are usually not too difficult, AoC is great for getting folks of all different skill levels to dip their toe in and then start learning new techniques for solving puzzles using their toolset.
Theming</h2> A lot of other programming competitions have fairly dry and minimal presentations for their puzzles. Eric Wastl</a> (the creater of AoC) puts a ton of care into the preparation of the prompts for each puzzle, often relating the puzzles back to a common holiday-related theme. This is just plain fun. Eric has appeared in many videos</a> providing a behind-the-scenes look at AoC and its history.
At the end of the day, these details help AoC bring a ton of joy to my holiday season.
Wrapping up</h2>
I think Advent of Code is one of the coolest programming-related events of the year. It's a chance to learn, struggle and grow as part of a larger coding community. Give it a shot in a couple of weeks! In the meantime, you can explore the puzzles from previous years to prepare.
Stay tuned for my next post on putting together an AoC template in a few different languages.

Musings on Python Type Hints

2022-10-13T00:00:00+00:00

"Gradual Typing"</a> has become incredibly popular over the last 8 years or so. The most notable examples of this phenomena exist in the JavaScript space. Since JavaScript is the lingua franca of the web</a> there have been several efforts to leverage the benefits of static type systems to enable easier programming-in-the-large for JavaScript. Gradual type systems such as TypeScript and Flow have found the most success in this space.

Shortly after the release of TypeScript, Guido van Rossum and Ivan Levkivskyi created PEP 483</a> proposing a type hinting system for Python. Notably absent from this proposal is any "official" type checking program for verifying the correctness of type hints. As a result, there are four "major" type checkers for python:

mypy</a>: A very early project (starting in 2012) from Jukka Lehtosalo. mypy and its predecessors have heavily influenced direction of python type hints. mypy has been supported heavily by Jukka's employer Dropbox. mypy is arguably the most popular type checker for Python.</li>
pyright</a>: A type checker built by Microsoft with great VS Code and LSP integration. Pyright was first released in 2019 and is bundled in the Microsoft Python extension for VS Code.</li>
pytype</a>: A static type analyzer built by Google. Pytype was first released in 2018 and emphasizes type inference and local leniency.</li>
pyre</a>: A static type checker built by Meta. Designed for large code bases and first-class integration with Buck, Meta's build system. The first release I can find dates back to 2019.</li> </ul>
There have been countless arguments over the value of static types over the years. I really like having a type system that can help me to avoid some common mistakes and guide me when refactoring. Most of all, however, type hints should broadcast intent; they are one of the main forms of documentation that the computer can help to verify.
Since starting a new job earlier this year, I've spent a good amount of time learning how to leverage type hints to help write readable, maintainable code. Let's take a look at some patterns that arise in Python where type hinting can get a little bit tricky.
Getting started</h3>
I'll be using mypy since it seems to be the most widely used tool, at the moment.
If you don't already have python installed, follow the instructions on the official python website</a> to install python for your platform. At the time of writing, the latest stable version is 3.10.7</code> which is what I will be using for my examples.
With those out of the way, we can start our python project and install mypy:
$ mkdir python_types && cd python_types $ python3 -m venv venv $ source venv/bin/activate # may be different on Windows $ python3 -m pip install mypy </code></pre>
"Hello, world!" of type checking</h3>
Let's start with a simple example:
def add(x, y): return x + y print(add(1, 2)) </code></pre> We can run this:
$ python3 -m src.hello 3 </code></pre> Now, let's see what mypy has to say:
$ mypy --strict src/hello.py src/hello.py:1: error: Function is missing a type annotation src/hello.py:5: error: Call to untyped function "add" in typed context Found 2 errors in 1 file (checked 1 source file) </code></pre> In strict mode, mypy expects type annotations in certain locations. In particular, function parameters and return types must be annotated. Let's modify our program and try again:
def add(x: int, y: int) -> int: return x + y print(add(1, 2)) </code></pre> When we run mypy now:
$ mypy Success: no issues found in 1 source file </code></pre> Cool. Now you may have noticed that there are some seemingly valid uses of the add</code> function that mypy now rejects; for example:def add(x: int, y: int) -> int: return x + y print(add("foo", "bar")) </code></pre> This program runs just fine: $ python3 -m src.hello foobar </code></pre> But when we run mypy: $ mypy src/hello.py:5: error: Argument 1 to "add" has incompatible type "str"; expected "int" src/hello.py:5: error: Argument 2 to "add" has incompatible type "str"; expected "int" Found 2 errors in 1 file (checked 1 source file) </code></pre> Of course, if we annotate the function parameters as being integers, any reasonable type checker should complain here. But a lot of the power of using python comes from being able to use functions in a flexible manner. Fortunately, there are features in the standard library that enable us to create better annotations fairly easily. Let's start with TypeVar</code> (introduced in pep 483</a>). If you are familiar with generics in another language such as TypeScript or Java, TypeVar</code> enables something similar. Let's take a look at what our function signature might look like with the use of TypeVar</code>: from typing import TypeVar T = TypeVar("T") def add(x: T, y: T) -> T: return x + y print(add("foo", "bar")) </code></pre> This looks close to reasonable, let's make sure it runs: $ python3 -m src.hello foobar </code></pre> Nice. Let's see if mypy likes it: $ mypy src/hello.py:8: error: Returning Any from function declared to return "T" src/hello.py:8: error: Unsupported left operand type for + ("T") Found 2 errors in 1 file (checked 1 source file) </code></pre> Huh. Well, when you think about it, just because x</code> and y</code> have the same type doesn't necessarily mean that x</code> and y</code> can be added together. Maybe x</code> and y</code> are dictionaries; the add operation is not defined for dictionaries. Perhaps we can do a little bit better. The critical idea here is called a Protocol</code> (introduced in pep 544</a>). A Protocol</code> lets us specify the shape of objects that we expect to see. In the literature, you might see this described as "structural subtyping" but most of the time you'll hear it as "duck typing": if it looks like a duck and quacks like a duck, it must be a duck. Let's see if we can get mypy to approve our program: from typing import Protocol, TypeVar Self = TypeVar("Self", bound="Addable") class Addable(Protocol): def __add__(self: Self, other: Self) -> Self: ... T = TypeVar("T", bound=Addable) def add(x: T, y: T) -> T: return x + y print(add("foo", "bar")) </code></pre> Whew, that's quite a bit of typing for such a simple example. Let's at least see if it works. First we run it: $ python3 -m src.hello foobar </code></pre> Now let's run mypy: $ mypy src/hello.py Success: no issues found in 1 source file </code></pre> Hooray! This is a lot of machinery for such a simple example. Let's unpack what's going on here: Self = TypeVar("Self", bound="Addable") class Addable(Protocol): def __add__(self: Self, other: Self) -> Self: ... </code></pre> This allows us to refer to the type of objects that can be added to other objects of the same type to produce yet another object of that same type. The Self</code> nonsense going on here allows us to constrain the type of the value that can be added to only those of the same type (and constrain the return type to the same type). This gets a lot easier in python 3.11 with the introduction of the Self</code> type to the standard library (pep 673</a>). Anthony Sotille has a great video covering the Self</code> type, as well: link</a>. As an aside, many of Anthony's videos are great bite-sized examples of various python concepts; he also has a great live stream! Now, this protocol isn't perfect: addition may be defined for combinations of types. As far as I know, there isn't a great way to express that sort of thing, just yet. Furthermore, while mypy accepts this program, both pyre and pyright seem to reject it. This is one of the downsides of having several different type checker implementations. Hopefully these discrepancies get ironed out as the ecosystem matures. With that introduction out of the way, let's take a look at a pattern that seem to come up pretty often in python and how we might be able to add type annotations. Decorator factories</h3> Decorators are a curiously functional feature of python. Decorator factories (functions that produce decorators) are fairly common but surprisingly difficult to write type annotations for. We're going to take a look at a much more complex example than the add</code> case that we explored above. In particular, we are going to leverage asynchronous code! If you aren't familiar with asyncio</code> in python, I encourage you to explore the official documentation: asyncio docs</a> To start out, we have a complete implementation of a decorator that continuously prints out the running time of the decorated async function: import asyncio import functools import inspect import time from typing import Final async def print_with_timer(format: str, interval: float = 0.1) -> None: start = time.time() while True: elapsed = time.time() - start print(format.format(elapsed=elapsed), end="") await asyncio.sleep(interval) OVERWRITE_LINE: Final[str] = "\033[F" TIMER_FORMAT: Final[str] = f"\n{OVERWRITE_LINE}Invocation {{module}}.{{name}}({{arguments}}) has been running for {{{{elapsed:0.2f}}}}s" def with_timer(interval_or_callback): def inner(func): @functools.wraps(func) async def wrapper(*args, **kwargs): arguments = inspect.signature(func).bind(*args, **kwargs).arguments argument_string = ", ".join(f"{name} = {value!r}" for name, value in arguments.items()) timer_format = TIMER_FORMAT.format(module=func.__module__, name=func.__qualname__, arguments=argument_string) if isinstance(interval_or_callback, float): task = asyncio.create_task(print_with_timer(timer_format, interval_or_callback)) else: task = asyncio.create_task(print_with_timer(timer_format)) try: result = await func(*args, **kwargs) finally: task.cancel() try: await task except asyncio.CancelledError: pass print() return result return wrapper if isinstance(interval_or_callback, float): return inner return inner(interval_or_callback) @with_timer async def do_stuff() -> int: await asyncio.sleep(5.0) return 42 @with_timer(0.5) async def do_more_stuff(duration: float) -> str: await asyncio.sleep(duration) return "Do a good turn daily." async def main() -> int: important_number = await do_stuff() print(f"The meaning of life is {important_number}") scout_slogan = await do_more_stuff(3.0) print(f"The scout slogan is \"{scout_slogan}\"") return 0 if __name__ == "__main__": raise SystemExit(asyncio.run(main())) </code></pre> First, let's run it to see what this program does: $ python3 -m src.timer Invocation __main__.do_stuff() has been running for 4.97s The meaning of life is 42 Invocation __main__.do_more_stuff(duration = 3.0) has been running for 2.51s The scout slogan is "Do a good turn daily." </code></pre> That's neat! We can see which coroutine we are waiting for and how long it has been running. Let's see if mypy likes this: $ mypy --strict src/untyped_timer.py src/untyped_timer.py:20: error: Function is missing a type annotation src/untyped_timer.py:21: error: Function is missing a type annotation src/untyped_timer.py:23: error: Function is missing a type annotation src/untyped_timer.py:46: error: Call to untyped function "inner" in typed context src/untyped_timer.py:49: error: Untyped decorator makes function "do_stuff" untyped src/untyped_timer.py:55: error: Call to untyped function "with_timer" in typed context src/untyped_timer.py:55: error: Untyped decorator makes function "do_more_stuff" untyped Found 7 errors in 1 file (checked 1 source file) </code></pre> Aha, mypy loses track of the types of do_stuff</code> and do_more_stuff</code> since we haven't added type hints to the decorator. This isn't a huge deal, here, but if we are relying on mypy to be able to determine the type of an intermediate expression as we are coding, this can be a pretty big pain. For example, let's try revealing the type of do_stuff</code>: ... reveal_type(do_stuff) reveal_type(do_more_stuff) ... </code></pre> When we run mypy: $ mypy --strict src/untyped_timer.py src/untyped_timer.py:20: error: Function is missing a type annotation src/untyped_timer.py:21: error: Function is missing a type annotation src/untyped_timer.py:23: error: Function is missing a type annotation src/untyped_timer.py:46: error: Call to untyped function "inner" in typed context src/untyped_timer.py:49: error: Untyped decorator makes function "do_stuff" untyped src/untyped_timer.py:55: error: Call to untyped function "with_timer" in typed context src/untyped_timer.py:55: error: Untyped decorator makes function "do_more_stuff" untyped src/untyped_timer.py:61: note: Revealed type is "Any" src/untyped_timer.py:62: note: Revealed type is "Any" Found 7 errors in 1 file (checked 1 source file) </code></pre> We see that the revealed type is Any</code> which isn't very helpful. If we remove the decorator, let's see if we can get a little bit more help: $ mypy --strict src/untyped_timer.py src/untyped_timer.py:20: error: Function is missing a type annotation src/untyped_timer.py:21: error: Function is missing a type annotation src/untyped_timer.py:23: error: Function is missing a type annotation src/untyped_timer.py:46: error: Call to untyped function "inner" in typed context src/untyped_timer.py:59: note: Revealed type is "def () -> typing.Coroutine[Any, Any, builtins.int]" src/untyped_timer.py:60: note: Revealed type is "def (duration: builtins.float) -> typing.Coroutine[Any, Any, builtins.str]" Found 4 errors in 1 file (checked 1 source file) </code></pre> So we see that mypy can handle this sort of thing, in general. How can we leverage type hints to give mypy enough information to tell us the type with the decorator in use? Let's dive in! import asyncio import functools import inspect import time from typing import Awaitable, Callable, Final, overload, ParamSpec, Protocol, TypeVar async def print_with_timer(format: str, interval: float = 0.1) -> None: start = time.time() while True: elapsed = time.time() - start print(format.format(elapsed=elapsed), end="") await asyncio.sleep(interval) OVERWRITE_LINE: Final[str] = "\033[F" TIMER_FORMAT: Final[str] = f"\n{OVERWRITE_LINE}Invocation {{module}}.{{name}}({{arguments}}) has been running for {{{{elapsed:0.2f}}}}s" P = ParamSpec("P") R = TypeVar("R") class Decorator(Protocol): def __call__(self, func: Callable[P, Awaitable[R]]) -> Callable[P, Awaitable[R]]: ... @overload def with_timer(interval_or_callback: float) -> Decorator: ... @overload def with_timer(interval_or_callback: Callable[P, Awaitable[R]]) -> Callable[P, Awaitable[R]]: ... def with_timer(interval_or_callback: float | Callable[P, Awaitable[R]]) -> Decorator | Callable[P, Awaitable[R]]: def inner(func: Callable[P, Awaitable[R]]) -> Callable[P, Awaitable[R]]: @functools.wraps(func) async def wrapper(*args: P.args, **kwargs: P.kwargs) -> R: arguments = inspect.signature(func).bind(*args, **kwargs).arguments argument_string = ", ".join(f"{name} = {value!r}" for name, value in arguments.items()) timer_format = TIMER_FORMAT.format(module=func.__module__, name=func.__qualname__, arguments=argument_string) if isinstance(interval_or_callback, float): task = asyncio.create_task(print_with_timer(timer_format, interval_or_callback)) else: task = asyncio.create_task(print_with_timer(timer_format)) try: result = await func(*args, **kwargs) finally: task.cancel() try: await task except asyncio.CancelledError: pass print() return result return wrapper if isinstance(interval_or_callback, float): return inner return inner(interval_or_callback) @with_timer async def do_stuff() -> int: await asyncio.sleep(5.0) return 42 @with_timer(0.5) async def do_more_stuff(duration: float) -> str: await asyncio.sleep(duration) return "Do a good turn daily." async def main() -> int: important_number = await do_stuff() print(f"The meaning of life is {important_number}") scout_slogan = await do_more_stuff(3.0) print(f"The scout slogan is \"{scout_slogan}\"") return 0 if __name__ == "__main__": raise SystemExit(asyncio.run(main())) </code></pre> There's a lot going on here, so let's walk through some of the new additions: ParamSpec</code>: this enables us to capture the signature of a Callable</code> object. You can do some pretty powerful things including adding additional arguments to functions while enabling your type checker to still understand the types being used. See pep 612</a> to read more about how ParamSpec</code> works.</li> Decorator protocol: in many cases, mypy doesn't actually force us to use a protocol for this sort of thing, but it is actually pretty nice. This basically says that our decorator can take any callable object that takes in arguments that adhere to parameter specification P</code> and returns objects of type R</code> and our decorator will return a callable object with that same type.</li> Overload: we want users to be able to use with_timer</code> with and without an interval. Always having to write with_timer()</code> is kind of awkward, so we can branch on the type that we receive as input. overload</code> informs the type checker that the return type of our decorator factory is always consistent depending on the input type to our decorator. Details on overload</code> can be found in pep 484</a>.</li> </ul> By leveraging these ideas, we are able to get mypy to fully understand what is going on with the types of our decorated functions. Let's first run mypy and see if it works: $ mypy --strict src/timer.py Success: no issues found in 1 source file </code></pre> Great! Let's see if mypy can correctly identify the types of our decorated functions by adding reveal_type</code> ... reveal_type(do_stuff) reveal_type(do_more_stuff) ... </code></pre> Now we run mypy again: $ mypy --strict src/timer.py src/timer.py:75: note: Revealed type is "def () -> typing.Awaitable[builtins.int]" src/timer.py:76: note: Revealed type is "def (duration: builtins.float) -> typing.Awaitable[builtins.str]" Success: no issues found in 1 source file </code></pre> Pretty neat! Wrapping up</h3> You can see that as library authors, there are lots of tools at our disposal for ensuring that we can produce well-typed APIs for our users while still leveraging many of the dynamic features of python. There are plenty more things to learn about in the world of python type hints. If these ideas are exciting to you, dive into the peps and consider contributing to one of the type checking projects! The static typing story for python is still very young. We don't have many of the type computation facilities of TypeScript. But the story is getting better! And from my perspective, having another way to help communicate intent and enable users to better understand interfaces is fantastic.

That's it!</h2>
That's all for now. Now that most of the boilerplate is out of the way, I'm hopeful that the next several days will go much quicker! See you next time.</p>

`That's it!</h2> I'm a little bit sad that day 1 had some more interesting algorithmic ideas to explore than day 2. Oh well. I'll probably jump to something like day 20 for my next post so we can get into some juicy ideas.</p>`

Closing Thoughts</h2>
I've been spending a lot of time in group activities this week. I think I'm developing a stronger sense for how I want to spend my time during the rest of my batch. I definitely want to continue with these daily reflections. Thanks again for reading!</p>

That's all for now</h2>
Most of these will probably be quick updates with maybe a more in-depth update once every week or two. Thanks for reading!</p>

Wrap Up</h2>
If you've made it this far, thanks for reading! Stay tuned to hear more about my experience at RC.</p>

samgeo.codes

Resource management and generators in Python

Return Statement

Understanding the costs of our abstractions

Day 1

Day 2

Weeks 2 and 3 at RC

Day 4 at RC

Databases</h3>
I had a great coffee chat with an RC alum mostly discussing databases. I think I definitely want to pursue something in this space. I think I'm a bit anxious about building a toy database but maybe it's worth just trying.</p>

Presentations</h3>
The presentations were awesome. It's a lot of fun to see the sheer variety of things that people are working on.</p>

That's all for now</h2>
Thanks for reading. These posts will probably continue to lose structure over time. It's a lot of fun to write about random thoughts that I've been having even if they're not quite fully baked.</p>

Day 3 at RC

Day 2 at RC

First day at the Recurse Center

Getting Started With Advent of Code

Why I love Advent of Code

Musings on Python Type Hints

samgeo.codes

Resource management and generators in Python

Return Statement

Understanding the costs of our abstractions

Understanding abstractions</h2> Most software projects are built atop a perilously tall stack of abstractions.</p>

Day 1

Problem statement</h2> The example input looks like this:</p>

Part Two</h2> Now we want the total of the top 3 elves!</p>

Improvements</h2> Maybe we can do a little bit better, while also learning a bit more about the (unnecessarily enormous) Python standard library!</p>

Day 2

Problem statement</h2> Rock Paper Scissors!</p> We are given a "strategy guide" that looks like:</p>

That's it!</h2> I'm a little bit sad that day 1 had some more interesting algorithmic ideas to explore than day 2. Oh well. I'll probably jump to something like day 20 for my next post so we can get into some juicy ideas.</p>

Weeks 2 and 3 at RC

Day 4 at RC

Databases</h3> I had a great coffee chat with an RC alum mostly discussing databases. I think I definitely want to pursue something in this space. I think I'm a bit anxious about building a toy database but maybe it's worth just trying.</p>

Presentations</h3> The presentations were awesome. It's a lot of fun to see the sheer variety of things that people are working on.</p>

That's all for now</h2> Thanks for reading. These posts will probably continue to lose structure over time. It's a lot of fun to write about random thoughts that I've been having even if they're not quite fully baked.</p>

Day 3 at RC

Day 2 at RC

First day at the Recurse Center

DO to fly</h3> DigitalOcean is great. It's super simple to spin up a tiny VM, ssh into it, grab the latest copy of my code and run my program. I've been curious to get better familiarized with fly.io and similar hosting providers.</p>

Wrap Up</h2> If you've made it this far, thanks for reading! Stay tuned to hear more about my experience at RC.</p>

Getting Started With Advent of Code

Why I love Advent of Code

Community</h2> More than almost any other programming competition, AoC fosters a wonderful supportive community.</p>

Musings on Python Type Hints

`That's it!</h2> I'm a little bit sad that day 1 had some more interesting algorithmic ideas to explore than day 2. Oh well. I'll probably jump to something like day 20 for my next post so we can get into some juicy ideas.</p>`

Databases</h3>
I had a great coffee chat with an RC alum mostly discussing databases. I think I definitely want to pursue something in this space. I think I'm a bit anxious about building a toy database but maybe it's worth just trying.</p>

Presentations</h3>
The presentations were awesome. It's a lot of fun to see the sheer variety of things that people are working on.</p>

That's all for now</h2>
Thanks for reading. These posts will probably continue to lose structure over time. It's a lot of fun to write about random thoughts that I've been having even if they're not quite fully baked.</p>

Wrap Up</h2>
If you've made it this far, thanks for reading! Stay tuned to hear more about my experience at RC.</p>