mliezun.com

Vercel-like development setup for Django using Caddy

2026-02-11T00:00:00Z

Vercel-like development setup for Django using Caddy

Vercel provides sites like {branch}.vercel.app that show a preview of your changes and it usually builds really fast, which makes it a seamless experience.

This post gets you the same for Django (or any ASGI/WSGI app) on a single VM with Caddy, Python and rsync.

On each push, package the commit as a .tar.gz and rsync it to /workspace/branches/{branchName}.

The VM runs Caddy with caddy-snake. Which means you get automatic HTTPS, support for Websockets and fast static files serving.

Your production app runs on yourdomain.com.

Branches get a subdomain like feature-x.dev.yourdomain.com.

Overview

graph LR subgraph repo["Code Repository"] main[main] featureA[featureA] featureB[featureB] end subgraph cicd["CI/CD Pipeline"] pipeline[Build & Deploy] end subgraph vm["Single VM Server"] direction TB maindir["📁 /workspace/branches/main/"] fadir["📁 /workspace/branches/featureA/"] fbdir["📁 /workspace/branches/featureB/"] caddy["Caddy Server"] maindir --> caddy fadir --> caddy fbdir --> caddy end subgraph dns["DNS Records"] direction TB dns1["yourdomain.com
A → serverIP"] dns2["*.dev.yourdomain.com
A → serverIP"] end subgraph domains["Public Access"] d1["yourdomain.com"] d2["featureA.dev.yourdomain.com"] d3["featureB.dev.yourdomain.com"] end main --> pipeline featureA --> pipeline featureB --> pipeline pipeline --> maindir pipeline --> fadir pipeline --> fbdir dns1 -.-> vm dns2 -.-> vm caddy --> d1 caddy --> d2 caddy --> d3 style repo fill:#e1f5ff,stroke:#0066cc style cicd fill:#fff4e1,stroke:#ff9800 style vm fill:#f3e5f5,stroke:#9c27b0 style dns fill:#ffe0e0,stroke:#d32f2f style domains fill:#e8f5e9,stroke:#4caf50 style caddy fill:#c5cae9,stroke:#3f51b5

CI/CD: On each push, build a .tar.gz and rsync to /workspace/branches/$BRANCH/.
VM: Caddy + caddy-snake. Production at yourdomain.com, previews at *.dev.yourdomain.com.
DNS: A records for yourdomain.com and *.dev.yourdomain.com → VM IP.
Caddy: Single configuration file and automatic HTTPS.

1. Deploy pattern: tar.gz + rsync

On every push (e.g. in GitHub Actions or GitLab CI), do something like:

# Example: in CI, after checkout
BRANCH_NAME="${GITHUB_REF#refs/heads/}"
tar -czf app.tar.gz \
  --exclude='.git' \
  --exclude='__pycache__' \
  --exclude='.venv' \
  .
rsync -avz --delete app.tar.gz \
  deploy@YOUR_VM_IP:/workspace/branches/"$BRANCH_NAME"/
ssh deploy@YOUR_VM_IP "cd /workspace/branches/$BRANCH_NAME \
  && tar -xzf app.tar.gz \
  && rm app.tar.gz \
  && uv sync \
  && uv run manage.py migrate \
  && uv run manage.py collectstatic --noinput"

Sanitize branch names in CI if needed (e.g. replace / with -) so the path and subdomain match.

The server can create the venv per branch with uv sync (as in the example) or you can ship a pre-built .venv.

2. Caddy + caddy-snake

Install Caddy with the caddy-snake plugin (see repo for build steps; releases often include a binary).

Example Caddyfile

Production: yourdomain.com → fixed working_dir for main.
Previews: *.dev.yourdomain.com → working_dir uses the placeholder; autoreload so the next request after a push serves the new code.

yourdomain.com {
  handle_path /static/* {
    root * /workspace/branches/main/staticfiles
    file_server
  }
  route {
    python {
      module_wsgi app:wsgi.application
      working_dir /workspace/branches/main/
      venv .venv
    }
  }
}

*.dev.yourdomain.com {
  handle_path /static/* {
    root * /workspace/branches/{http.request.host.labels.3}/staticfiles
    file_server
  }
  route {
    python {
      module_wsgi app:wsgi.application
      working_dir /workspace/branches/{http.request.host.labels.3}/
      venv .venv
      workers_runtime thread
      autoreload
    }
  }
}

Notes:

workers_runtime thread is required with placeholders and autoreload.
Use your Django WSGI path (e.g. mysite.wsgi:application) instead of app:wsgi.application.
The branch directory must exist before the first request (CI creates it). Caddy-snake loads and caches the app on first request; autoreload watches for .py changes after each deploy.
Set Django’s STATIC_ROOT to staticfiles so collectstatic matches the staticfiles/ paths in the Caddyfile.

Subdomain → branch name

Caddy splits the host into labels, right to left: for featureA.dev.yourdomain.com, labels.0 = com, labels.1 = yourdomain, labels.2 = dev, labels.3 = featureA. So the branch is {http.request.host.labels.3} for *.dev.yourdomain.com.

Adjust the index if your dev domain has a different structure.

3. DNS configuration

Add an "A record" for @.dev.yourdomain.com (use wildcard depending on your DNS provider) pointing to the VM’s IP.

After propagation, Caddy can obtain a cert for *.dev.yourdomain.com via ACME and route each subdomain to the matching branch directory.

4. What happens at runtime

First request to feature-x.dev.yourdomain.com: Caddy resolves the placeholder to /workspace/branches/feature-x/; caddy-snake loads and caches the app. If that directory doesn’t exist yet (branch not deployed), the request fails until CI has run.
After a push: CI overwrites the branch directory; the next request triggers autoreload and serves the new code.
New branch: Push → CI creates the directory → first request to new-branch.dev.yourdomain.com loads it.

Push a branch, wait for CI, then open {branch}.dev.yourdomain.com.

Bonus: clean up on merge

When a branch is merged and deleted, trigger a pipeline step that removes its directory on the VM (e.g. ssh deploy@VM "rm -rf /workspace/branches/$BRANCH_NAME"). Caddy-snake sees the directory is gone and stops serving that app; the next request to {branch}.dev.yourdomain.com fails until a new branch with that name is deployed. That keeps preview URLs only for active branches and frees disk space.

Work Hard

2026-01-26T00:00:00Z

Work Hard

If working hard is not working long hours nor performing extenuating tasks, then what is it?

Working hard is working with purpose and care.

We often relate the following conditions to hard work but they're not part of its essence:

Working long hours.
Not taking breaks/days off.
Performing physically extenuating activities.
Toxic work environment.
Away from friends and family.
Repetitive and boring tasks.

Those properties are part of any job, but none of them are actually necessary to doing hard work.

An example from my own work as a Software Engineer is that when I try to solve a problem I never stop working, my mind is always connected to the topic and trying to draw inspiration from the environment.

I had plenty of epiphany moments on how to solve a problem or improve a system while doing something completely different: jogging, watching a movie, walking around the city, talking with friends.

This happens when you're tasked with something challenging that sparks your creativity, and you care about the result.

The more you care, the more you engage that creative side that is working on the background all the time. But then it has to be followed with the discipline to deliver what you have envisioned.

Being creative is what makes us human and in the age of AI is more important than ever.

Try to put yourself in the line of work where you care about what you do, that let's you work towards a purpose that you find valuable and express your creativity. That will make you work hard, and then you'll not mind working longer hours or during weekends.

What is caring?

In a workplace it can mean different things:

Solve a problem.
Improve some part of the system.
Being brave to speak up and challenge the current status quo.
Respect others opinions even if you think they're wrong.
Engage in dialogue.

To summarize

For me working hard can be summarized into 3 words:

Care
Purpose
Responsibility

If those things are part of your work then you're working hard.

What's in the future?

2025-12-22T00:00:00Z

What's in the future?

Capabilities of AI tools are still advancing rapidly, doubling the length of tasks AI can do every 7 months.

At the end of 2025, we're sitting at around 5 hours. This means that AI tools can now complete tasks that would take a human around 5h and they succeed 50% of the time.

By the end of 2026 this will be close to 20 hours roughly, that's almost equivalent to a human work-week.

But what's more striking is that for Software Engineering this is accelerating at a higher pace.

For coding, we're seeing capabilities double in just 70 days!

By extrapolating this trend, discussing this with friends and seeing what people are saying online we can see that the burden of software is less and less coding and increasingly about proper design and verification.

Even Karpathy is saying: Software 2.0 easily automates what you can verify.

He's talking about writing software to solve problems that can be verified to be correct or not with a certain grade of accuracy.

For coding in particular this means we need tools to verify the quality of the code written. The better our tools are to formally verify the code the more we can trust our automation to work.

Seeing the landscape of what's out there I think we're missing a tool that has all of the following:

Ownership of the software process: Design + Implement + Test (like Cursor)
Hosts the code itself (like Github/Gitlab)
Runs the code and makes it available (like Vercel)
Checks that the code is correct and provides guardrails to users (static analysis)
Is opinionated in what technology should be used (e.g. Python only hosting)
Provides integration with other services (e.g. easy Stripe integration)
Open source

The closest we have to this is Lovable, Replit and v0 from Vercel.

But those platforms still rely on external services and are not open source. I think LLMs will work best when we're able to let them handle the full environment were our code lives and making them open source will enable people to trust them and run them on their own.

AI generated content in this blog

2025-11-19T00:00:00Z

AI generated content in this blog

In the last year I increased the usage of LLM based tools for code generation.

They have been of great help to improving the CSS styles and adding support for Dark and Light mode in this blog.

The previous post: Benchmarking Wagtail was almost completely generated by Cursor. I did review it and decided to publish it but something felt off.

Which led me to think about the purpose of this blog itself.

I started this blog with the intention of being a place for tinkering, for expressing myself and to be creative.

I don't want to use LLM tools to generate the content of my posts anymore, since I wouldn't be really expressing myself.

I will still use them to modify some parts of the blog's engine and probably still use them for CSS styles.

For the sake of the readers I introduced some new tags that let you know if the content of a post contains AI-generated text or is completely human written.

Benchmarking Wagtail CMS Across Python Versions

2025-10-16T00:00:00Z

Benchmarking Wagtail CMS Across Python Versions

Testing real-world performance across 8 different Python versions

Published: October 16, 2025

Introduction

I've been curious about how different Python versions perform in real-world web applications. Python keeps getting faster with each release, but I wanted to see how much of a difference it actually makes when running a real application under load.

So I decided to benchmark Wagtail CMS a Django-based content management system, tested across 8 different Python versions. This is the first in what I hope will be a series of benchmarks testing different Python web applications.

Here's what I tested:

Python 3.10 (our baseline)
Python 3.11
Python 3.12
Python 3.13
Python 3.13t (free-threaded, GIL disabled)
Python 3.14
Python 3.14-tailcall (experimental tailcall optimization)
Python 3.14t (free-threaded, GIL disabled)

The results were pretty interesting! You can check out the full benchmark setup and results at https://github.com/mliezun/python-web-benchmarks.

How I Set Up the Tests

The Setup

I wanted to keep things realistic, so I used:

Wagtail CMS 6.3 with Django 5.1.4 (a real production setup)
Gunicorn 21.2.0 as the WSGI server
SQLite with actual content (15+ blog posts, multiple pages)
Apache Bench (ab) for load testing
Docker to keep everything isolated and reproducible

For the experimental Python versions, I used custom Docker images:

ghcr.io/mliezun/python-web-benchmarks/python-freethreaded:3.14t-slim
ghcr.io/mliezun/python-web-benchmarks/python-freethreaded:3.13t-slim
ghcr.io/mliezun/python-web-benchmarks/python-freethreaded:3.14-tailcall-slim

Test Scenarios

I ran three scenarios that you'd actually encounter in real usage:

1. Homepage Load: Full page with navigation, sections, and dynamic content

2. Admin Login: Authentication and admin interface access

3. Blog Browsing: Listing and viewing blog posts

Each scenario was tested with 1, 5, 10, and 20 concurrent users to see how things scale.

What I Measured

Requests per Second (RPS): How many requests we can handle
Response Time: Average, p50, p95, and p99 percentiles
CPU Usage: How hard the server was working
Memory Usage: RAM consumption during tests

The Results

Overall Throughput Performance

Here's how many requests per second each Python version can handle. Higher is better, obviously.

Response Time Analysis

This shows how response times change as we add more concurrent users. Lower is better here.

Relative Performance Improvements

This is probably the most interesting chart - it shows how much faster each version is compared to Python 3.10 (our baseline). Positive numbers mean it's faster.

Resource Utilization

CPU and memory usage is important for capacity planning. You want good performance without burning through resources.

What I Found

🏆 Python 3.11 is the Winner

Python 3.11 came out on top with about 3.2% better performance than Python 3.10 on average. That might not sound like much, but when you're handling thousands of requests per day, it adds up.

The Big Picture

1. Experimental versions are interesting: Python 3.14t (free-threaded) shows some promising results for concurrent workloads, but I'd be careful about using it in production just yet.

2. All versions scale well: Every version handled 20 concurrent users without breaking a sweat. The newer versions just do it slightly better.

3. Memory usage is consistent: All versions used roughly the same amount of RAM (~300-360 MB), so upgrading won't cost you more in memory.

4. Everything was stable: Zero errors across all tests. All versions are production-ready.

My Recommendations

For Production

Go with Python 3.11 if you want the best performance. It's stable, fast, and has been around long enough that most packages support it well.

Python 3.12 is also a solid choice if you want something newer but still conservative.

I'd hold off on the experimental versions (3.14t, 3.14-tailcall) for production use until they're more mature.

If You're Upgrading

If you're currently on Python 3.10 or earlier, here's what I'd suggest:

Important note: Python 3.10 reaches end-of-life on October 31, 2026** according to the Python Developer's Guide. After that date, it will only receive security fixes, so you'll want to plan your upgrade well before then.

1. Test everything: Run your full test suite on the target version first

2. Check your dependencies: Make sure all your packages support the new version

3. Benchmark your specific app: My results are for Wagtail - your mileage may vary

4. Deploy carefully: Use canary deployments or blue-green strategies

The good news is that memory usage is pretty consistent across versions, so you won't need to resize your servers.

Wrapping Up

So what did I learn from all this testing?

Python version upgrades are worth it. You get measurable performance improvements without needing more resources. Python 3.11 is the sweet spot right now - fast, stable, and well-supported.

The experimental versions are definitely interesting to watch. The free-threaded versions (3.13t, 3.14t) could be a game-changer for certain workloads, but I'd wait until they're more mature before using them in production.

I'm planning to benchmark more Python web applications in the future - maybe Django REST Framework, FastAPI, or even some data processing workloads. If you have suggestions for what to test next, let me know!

You can find all the benchmark code, results, and setup instructions at https://github.com/mliezun/python-web-benchmarks. Feel free to run your own tests or contribute to the project.

Serving Python apps using the Caddy web server

2025-10-03T00:00:00Z

Serving Python apps using Caddy web server

Caddy is an enterprise grade web server and reverse proxy. I recently released a new version of caddysnake a plugin to serve python apps more easily using Caddy without the need of installing another server like gunicorn, uvicorn, hypercorn or others.

In this latest release there are some new features to explore:

Distribution of a single caddy binary with bundled python.
Added python-server subcommand.
Multiple worker process.
Package in PyPI available for a quick way to install: caddysnake.

Distribution of single binary

The latest release is available on github.

Now you can see for example caddy-standalone-3.10-x86_64_v2-unknown-linux-gnu.zip as an available asset to download.

After downloading and unzipping you get a single binary named caddy. This packages an entire Python distribution inside so you now don't have the need to install it on the target system.

This would be useful for docker images since now you don't have to install the correct python version, just include the caddy binary and that carries along all of its dependencies.

Python server subcommand

With the new release you get a shorthand caddy python-server.

This will configure a virtual Caddyfile and run the server for you.

For example if you have main.py file with a Flask app inside you'll simply run:

caddy python-server -t wsgi -a main:app

This will start serving request on http://localhost:9080/ by default.

See more settings with caddy python-server --help.

Multiple worker processes

Up until now caddysnake has been serving request from a single python processes, given that the GIL hasn't been removed yet that means we were not fully utilizing all the cores available.

Now by default caddysnake starts as many processes as cores you have in your cpu, configurable with --workers .

This increases throughput but also overhead of communication between the Go and Python processes.

Package available in PyPI

Now it's possible to install caddysnake as a package using pip:

pip install caddysnake
caddysnake python-server --help

This is a quick way to get started without the need to compile caddy yourself.

The Enduring Simplicity of C

2025-09-22T00:00:00Z

The Enduring Simplicity of C

I recently came across a video by Salvatore Sanfilippo, the creator of Redis, titled "Why I use C and not Rust". It made me stop and think about the constant push for newer, "safer" programming languages.

Watch the video here.

The gist of his argument is that while C has well-known memory safety issues, the modern alternatives, particularly Rust, come with their own significant costs. This isn't just about nostalgia for an older language, it's a pragmatic choice based on a different set of values.

His points really resonated with me:

Safety has a complexity cost. Rust's main selling point is memory safety, but this safety net makes certain things, like a doubly linked list, much more complex to implement. Sanfilippo argues that he'd rather have the directness and simplicity of C and manage the risks with good practices and tools like Valgrind.
The curse of hyper-dependencies. He makes a point that C's lack of a built-in package manager is a feature, not a bug. In ecosystems like Rust or Node.js, it's incredibly easy to accumulate massive dependency trees for simple projects. C forces you to be more deliberate.
The need for speed. The feedback loop in programming is critical. Sanfilippo notes that C compiles in seconds (or fractions of a second), while Rust is known for "biblical" compilation times. He argues that waiting for the compiler kills productivity and flow.
AI understands C better. This was the most surprising insight. Because of C's simple semantics and the sheer volume of C code in the world, LLMs are currently much better at reasoning about and generating C code than Rust. In an age where AI is becoming a co-pilot, this is a huge practical advantage.

It's not about being against progress. It's a realization that the "new and safe" way isn't always the "better" way. In engineering we always face compromise and deciding which language to use is just one of those cases.

Personally, I like both languages, I'm more fond of C because of its simplicity, but the Rust toolchain and community pushing it forward makes it really easy to start a project and get everything you need to start building.

A systems level programmer like Salvatore might pick C because he has the need to build an object system perfectly tailored for his application. But I mostly spend time writing APIs and glue code in Python, I would pick up Rust when I'm in the need for a little extra speed because is easy to integrate with Python code thanks to PyO3.

Removing read time

2025-07-18T00:00:00Z

Removing read time

For a long time I've had "Reading time: x minutes" at the top of all articles in my blog. I cannot recall why I choose to do this, I suppose I saw it in another place and decided to copy it without thinking the reason. Maybe there was a reason and I just forgot.

Now I'm making a statement and removing it.

Other sites put this "time advertisement" so users can know wether it's gonna be a long or short read and they can decide to stay and read the full thing or to bookmark the article for later.

I don't like that, I want to start reading and keep on going if I feel captivated without thinking about the time.

Sometimes I also feel bad if the estimation says 5 minutes and it takes my 20 to read it and understand it.

For me, when writing, there's also this insecurity of an article not being good enough if it doesn't reach a certain number of minutes.

I'm getting rid of all that and doing one of my favorite activities in programming: deleting code.

AWS Spot instance advisor

2025-07-16T00:00:00Z

AWS Spot instance advisor

Spot instances are very useful for background jobs where you can afford them to be interrupted and retried, in return you usually pay a fraction of the cost, up to a 90% discount on the on-demand price. AWS provides a tool called Instance Advisor where you can see which instances are more requested and try to avoid them.

Let's see a comparison between Graviton 3 and Graviton 4 ARM instances.

m7gd instances are Graviton 3 with local SSD and they get interrupted a lot >20% of the time.

m8gd are the latest Graviton 4 with local SSD and they get interrupted between 5-20% of the time based on which instance type you choose. This could be due to slower adoption of newly released instances.

In general it's a good idea to check the frequency of interruption here to optimize your spot instance usage:

https://aws.amazon.com/ec2/spot/instance-advisor/

Rick and Morty and the end of history

2025-07-04T00:00:00Z

Rick and Morty and the end of history

In Season 2 Episode 6 named "The Ricks Must Be Crazy" we see how Rick and Morty go inside Rick's microverse car battery.

The gist of it is that there's a universe inside a battery and the people have to pull cranks to make power. Some of that power is used in their own planet, the rest is used for Rick's car.

But then inside each universe there's a new scientist that develops the same technology. In total we get 3 levels of nestedness inside batteries throughout the episode.

There's a particular joke in this episode that is repeated each time a new character learns about the people inside the battery being used to generate the power.

The joke goes something like this:

> Morty: "You have a whole planet sitting around making power for you? That's slavery!"

> Rick: "It's society, they work for each other, Morty. They pay each other. They buy houses. They get married and make children that replace them when they get too old to make power."

> Morty: "That just sounds like slavery with extra steps."

> Rick: "Oh-la-la! Someone's gonna get laid in college."

Until recently I just took this joke at face-value, I interpreted it as Morty being progressive and caring about people, and that's why he would get laid in college.

But I just recently got recommended this video on youtube:

Watch video

Francis Fukuyama and the end of history

The video explains how all others political ideologies have been defeated by liberalism. In modern times called neoliberalism.

It basically goes on to explain how starting from the 80s the western world has shifted from thinking of people as workers to calling them consumers.

And this is because we know see everything through an economic lense, we think how much a person is worth in terms of money, we want to go to school to get better jobs to make more money instead of to just learn.

Consumerism is the perfection of slavery, because people are enslaved by money, but they have a sense of choice (car, school, house, etc.), they perceive themselves as free and have no reason to rebel.

There's an article published by Francis Fukuyama where this is explained in more detail.

It basically says that we have reached the end of history in terms of political ideology because we cannot escape liberalism. Is a system that within itself allows for contradiction and even then remains stable. Where individuals feel free and in control and have no reason to rebel.

What is more surprising is that he cites Hegel and Kojève as people that had already realized this was in motion in 1806 and 1930 respectively.

For me this was eye-opening, I had never been exposed to this ideas, at least in such clear and distilled form. Something inside me tells me this is true, we're at the end of history. I don't think this is good or bad, but a powerful realization that allows me to see the world through a different lens.

EDIT

After posting this originally I found another video from Noam Chomsky which reinforces the idea of liberal capitalism as a form of slavery.

Watch video

He basically says that having a job is like surrendering yourself to a master that has control over you life even bigger than the state. As a "free" person in a liberal society you're free to have a job or starve to death. That job is basically a tyranny.

AI is a dishwasher

2025-06-26T00:00:00Z

AI is a dishwasher

AI agents are like dishwashers. Yes, you can probably be more precise and clean the dishes better by hand, some of them might be even not dishwasher-friendly and require that you clean them by hand. But for most cases you just load the dishwasher and turn it on and come back in a few hours.

By "AI agents" I mean particularly LLMs used for code generation like: Cursor, Claude Code, Codex, Windsurf and others.

I've seen many people claim that AI is not good at software development and that you shouldn't use it. I think that's like claiming you should not use a dishwasher, a washing machine, an oven or any other appliances that you have in your house.

"But I can do it faster without AI"

I have said this many times, I can do it faster and I can do it better. Many of the times this is the case in software development. Because we're all highly skilled individuals, we know what we're doing and we can do it well.

I'm not an AI fanatic, but refusing to use a convenient tool just doesn't make sense.

Where objections make sense

I've seen Open Source projects reject AI-generated code: cURL and Qemu come to mind.

And to be honest, I think they're pretty good reasons.

cURL is rejecting it because there's too much poor quality reports.
Qemu is rejecting it for licensing reasons.

Learn to use your tools

You can generate a lot of code with AI, but that doesn't mean you should do it. You shouldn't mindlessly give LLMs control of your projects.

When you do laundry you need to correctly use the machine and know how to take care of your clothes based on their fabrics, colors and composition. When you fail to separate your whites from your blacks you screw up your clothes. The same happens to your codebase if you do not keep control of AI Agent outputs.

AI is not really a dishwasher

This post is a simplification, there are many more nuances about the topic. Moral, ethic, legal and political implications about current AI models.

You should use your best judgement to when is appropriate to use it.

DjangoCon Europe 2025

2025-05-14T00:00:00Z

DjangoCon Europe 2025

I attended DjangoCon Europe this year, it was my first software conference, I always wanted to go to one, but never took the time until this year. A great experience overall, the most rewarding thing was meeting so much wonderful people.

I had sparking conversations with many individuals, and I have taken notes from those before the memories fade.

But I worry that I may have forgotten already about some of those, because there were so many.

You can see more about the event here: https://2025.djangocon.eu/

I gave a lightning talk about caddy-snake. A lot of people were confused about the name, so I might change it.

Here's the talk: caddy-snake-djangocon-2025.pdf.

If anyone is thinking about joining the Django community, please don't hesitate and do it. The people are great.

There's a program called Djangonaut Space which welcomes newcomers. You can find your way there.

There are also many Django Social events that you can join.

Cheers!

Advent of Code 2024

2025-03-20T00:00:00Z

Advent of Code 2024

I discovered AOC quite late in my career. I feel it's more like something one does when it's in college, my first participation was in 2023, and 2024 was my second. The first time I was very into it and I had a lot of fun, the second time I just found challenges to be repetitive, very similar to the year before but with a little bit different of a story.

What happened in 2024?

I did a lot of things in 2024, like moving to Paris, France. And I have also moved apartments within Paris. During December 2024 I was still adapting to life and responsibilities in Paris, so I felt like I didn't have much time to dedicate to AOC.

The first time I did it I was in a much more comfortable and I could dedicate as many hours as I wanted to solve a problem. Even do brute forcing and let it run for hours.

In December 2024 I didn't have much time, and every minute that I was spending in AOC I felt like it was being wasted. That I could be out hanging out, doing something fun, visiting some place new or just meeting with people.

I feel like is a game for kids

It's ok, I discovered really late and I wanted to play it at least once.

I have a feeling that if I was in college and found out about it I would have played it every year.

But now, juggling work and life, trying to participate in AOC is just not worth it.

There's a lot more things that I'd rather be doing/building than spending time in front of the computer solving christmas themed puzzles.

December is also a month in which you have many festivities, and have to plan for them, like visiting family and friends on Christmas, going on a trip for New Year's or just decorating your house. Those things are a lot more work, and in my opinion way more fun.

Also, if you're fresh out of college and you mention participating in AOC I feel like it would sound cool and could get you hired. If you're an experienced software engineer it just doesn't matter.

Bye bye

Advent of Code is really cool. But I think it's just not for me anymore.

It's ok to let things go when you recognize that they don't bring much value to your life anymore.

Caddy Snake improvements

2024-11-07T00:00:00Z

Caddy Snake improvements

I've been planning on improving Caddy Snake, making it more stable and easier to use. Right now, I want to add automatic test for Django, and build binaries and Docker images for arm64 and riscv64. I'd also like to turn it into a Python package so you can plug it straight into your code.

Fix/add/test support for Django

I've tried to use it a couple times just to test that it works and seems to have been broken. I'd like to test it a little bit more to make sure it works and solve all the bugs.

Also add an automatic test so that I'm sure it doesn't break after I make some changes.

Make sure we don't get segfaults on tests anymore

Sometimes I get random segfaults when tests run on CI. I'd like to take time to investigate this and fix the issues.

Add support for arm64 and riscv64

For now I'm only building caddy-snake for Linux x86_64. It would be cool to distribute more binaries for different OSes and CPU architectures.

For example:

linux / x86_64 / arm64 / riscv64

macOS / x86_64 / arm64

windows / x86_64 / arm64

Also deliver docker images for all of those.

Expose as python package that can be used directly

import caddysnake

from flask import Flask

app = Flask(__name__)

@app.route("/hello")
def hello():
 return "Hello world"
 
if __name__ == "__main__":
    caddysnake.run(app=app, host="localhost:8082")

Better performance and isolation

Complete the work of integrating subinterpreters in caddy-snake.

https://github.com/mliezun/caddy-snake/pull/9

Track performance compared to other web servers and improve it.

Finding and fixing a bug in Python subinterpreters

2024-08-19T00:00:00Z

Finding and fixing a bug in Python subinterpreters

tldr

Filed an issue in CPython.
Sent a PR.
My code was garbage and was not merged, but helped to get the issue fixed.

Backstory

Lately, I've been working with Python C-API. I wanted to use subinterpreters with their own GIL to unlock the performance gains promised by being able to execute many threads in parallel which was not possible before Python 3.12.

The reason is that I've been building a Caddy web server plugin called: Caddy Snake.

The plugin let's users embed a Python interpreter and serve requests directly from Caddy.

Caddy is written in Go, and to interact with Python I had to use CGO, a compatibility layer that makes it easy to call C functions from Go.

To improve performance I wanted to use the new feature of having separate GIL per subinterpreter so requests could be served by many threads at the same time.

Today I saw a great blogpost about finding a bug in subinterpreters and that inspired me to write about my experience.

The issue

I started coding up a basic implementation to see if I could serve simple requests from subinterpreters.

From the get go I found that requests were failing in Python version 3.12 but working for previous versions thanks to CI tests.

The failure was constrained to a C function where the HTTP status code was being converted from string to int: strtol(statusCode).

By printing the content of the statusCode variable I found the following difference.

In main interpreter:

In a subinterpreter:

I managed to track down the enum that was causing this. Then I crafted some code to try to reproduce the error in a standard Python interpreter.

I came up with this:

import _xxsubinterpreters as interpreters


script = """from enum import _simple_enum, IntEnum

@_simple_enum(IntEnum)
class MyEnum:
    DATA = 1
    
print(str(MyEnum.DATA))
"""

exec(script)
# Output: 1

interp_id = interpreters.create(isolated=False)
interpreters.run_string(interp_id, script)
# Output: , Expected: 1

That piece of code executes the same python code in both the main interpreter and a freshly created subinterpreter. The output should be the same, but it's not. The problem was independent of running with a separate GIL or with the same GIL.

Checking the __str__ methods I could see that they were clearly different.

In main interpreter:

...
print(MyEnum.DATA.__str__)
# Output:

In subinterpreter:

...
print(MyEnum.DATA.__str__)
# Output:

In the main interpreter the method wraps __repr__. In the subinterpreter it wraps __str__.

At this point I couldn't believe what I was looking at. When your program has a bug you always assume is your fault, it's rare to see the case where the problem is in the INTERPRETER.

But to my own disbelief I had found a problem with the CPython implementation.

Doing my homework

After that I decided that I was gonna do the right thing: file an issue in Github telling what I just witnessed.

First thing first: read the guide on how to contribute to python.

This is what you should always do when contributing to a project because you might find that your problem is actually not worth reporting or that it was already solved in the main development branch.

I made sure to test with all supported Python versions 3.8, 3.9, 3.10, 3.11, 3.12, 3.13 and in both Linux and macOS.

The problem was only present in 3.12 and 3.13, after subinterpreters with separate GIL was introduced.

The community is awesome

In less than 1 hour and a half of posting the issue I got a response from a member of the Python Triage Team: "Bisected to de64e75".

I thought to myself: "that is awesome, this is happening so fast!".

Then others pointed out that you could reproduce it with a shorter script:

import _xxsubinterpreters as interpreters

script = """print(int.__str__)"""


exec(script)
# Output: 

interp_id = interpreters.create()
interpreters.run_string(interp_id, script)
# Output:

Having the exact commit where this problem was introduced and a concise way of seeing what the underlying problem is, I decided to give try to fix the issue myself. Diving into the CPython codebase.

Diving into CPython

The commit where the problem was introduced de64e75 has 185 additions and 86 deletions, and changes only 5 files.

If you take a look at it the most suspicious one is typeobject.c, you can see that it's changing some behavior in the MRO: Method Resolution Order. Which is the way methods are "inherited" from one class to another in Python.

I thought that was related because in the subinterpreter the int class inherits the method __str__ instead of __repr__. That was not the real issue, but I was correct about the file where the problem was.

Debugging was done in the good old fashioned way, adding print statements all over the place.

With my printing mechanism I managed to find out that the function that creates those objects is called type_ready().

That slot wrapper is a function that calls code from a type-slot. For example the int type defines a slot called tp_str. That slot is a C function that knows how to convert the object into a string. That C function is wrapped into a Python function which is executed when you do str(5).

I could see that type_ready() was called the first time from the main interpreter, and the second time from a subinterpreter, but this time it added more stuff.

One would expect that type_ready() gives the same result wether executed from a subinterpreter or from the main interpreter. The builtin types ( int, str, float, bool, ...) should be the same in all cases.

Another group of changes in the typeobject.c file was a lot of functions that obtain attributes from a builtin type, for example lookup_tp_dict():

static inline PyObject *
lookup_tp_dict(PyTypeObject *self)
{
+   if (self->tp_flags & _Py_TPFLAGS_STATIC_BUILTIN) {
+       PyInterpreterState *interp = _PyInterpreterState_GET();
+       static_builtin_state *state = _PyStaticType_GetState(interp, self);
+       assert(state != NULL);
+       return state->tp_dict;
+   }
    return self->tp_dict;
}

This function is obtaining the tp_dict property from a type. Which is a dictionary that stores attributes/methods of the class.

In this case the entire if-statement was added. If you take a closer look you can infer what's happening. Before we just obtained the value by doing self->tp_dict. Now, if it's a builtin type, we read the value from the interpreter state state->tp_dict.

With this information I started to dig deeper to see if I could find the exact place where the extra attributes were being added.

Taking int.__str__ as an example to see what was going on:

The type_ready() function gets executed as part of the main interpreter initialization process.
The function starts "readying" the int type: tp_dict gets filled by type_ready_fill_dict(), one of the thing this function does is lookup which slots are not empty and add them to the dict, for int the tp_str slot is empty.
After that, type_ready_inherit() gets called, and copies the tp_str slot from object to int.
The init process for the main interpreter finishes and we're ready to start with the second interpreter.
In this case, the tp_str slot is not empty for int, it was filled by step 3. So it gets added to the dict by type_ready_fill_dict().
The program continues and we see different behavior depending on which interpreter we run.

The problem here is that inside type_ready() we expect type_ready_fill_dict() to be called before type_ready_inherit(). Which is true for the main interpreter but not for the subinterpreter. Also, this is caused because all the slots in int are shared except for tp_dict which is stored in each interpreter.

Summary

To summarize the problem: type_ready() didn't receive a "clean" type. It was receiving a type with slots partially filled.

Some slots were filled because they were shared among interpreters, others like tp_dict were not shared. That caused inconsistencies between executions of type_ready().

The solution?

Given my lack of experience working in the CPython codebase I decided that the simplest solution was to make a quick and dirty solution.

My solution was to check if type_ready() was being called from a subinterpreter and in that case cleanup the base type so it looked the same as in the main interpreter.

It boils down to adding this function:

static int
fix_builtin_slot_wrappers(PyTypeObject *self, PyInterpreterState *interp)
{
    assert(self->tp_flags & _Py_TPFLAGS_STATIC_BUILTIN);
    assert(!_Py_IsMainInterpreter(interp));

    // Getting subinterpreter state
    managed_static_type_state *state = _PyStaticType_GetState(interp, self);
    assert(state != NULL);

    // Getting main interpreter state
    PyInterpreterState *main_interp = _PyInterpreterState_Main();
    managed_static_type_state *main_state = _PyStaticType_GetState(main_interp, self);
    assert(main_state != NULL);

    // Check wich attributes the type has in subinterpreter and it doesn't have
    // in the main interpreter. Store them in keys_to_remove.
    int res = -1;
    PyObject* keys_to_remove = PyList_New(0);
    if (keys_to_remove == NULL) {
        goto finally;
    }
    Py_ssize_t i = 0;
    PyObject *key, *value;
    while (PyDict_Next(state->tp_dict, &i, &key, &value)) {
        if (!PyDict_Contains(main_state->tp_dict, key)) {
            if (PyList_Append(keys_to_remove, key) < 0) {
                goto finally;
            }
        }
    }

    // Go through keys_to_remove and remove those attributes from
    // the base type in the subinterpreter.
    Py_ssize_t list_size = PyList_Size(keys_to_remove);
    for (Py_ssize_t i = 0; i < list_size; i++) {
        PyObject* key = PyList_GetItem(keys_to_remove, i);
        if (PyDict_DelItem(state->tp_dict, key) < 0) {
            goto finally;
        }
    }

    res = 0;

finally:
    Py_XDECREF(keys_to_remove);
    return res;
}

Then at the end of type_ready(), call that function for builtin types and only if we're in a subinterpreter:

    if (type->tp_flags & _Py_TPFLAGS_STATIC_BUILTIN) {
        PyInterpreterState *interp = _PyInterpreterState_GET();
        if (!_Py_IsMainInterpreter(interp)) {
            if (fix_builtin_slot_wrappers(type, interp) < 0) {
                return -1;
            }
        }
    }

And that's it. That was my solution. And it worked like a charm.

The wait

After sending a PR with my proposed changes I waited for a couple months until I saw some real activity.

Eric Snow is in charge of the subinterpreters implementations. He found a better more focused solution that was less fragile and took into account what would happen if the interpreter was reinitialized.

In the end I decided to close my PR because Eric's pushed the real solution into a separate PR.

I think it would have been cool to ship code to be included in all python interpreters around the world. But I'm happy my analysis helped solve the issue.

In Python 3.12.5 the fix was released: https://docs.python.org/release/3.12.5/whatsnew/changelog.html#core-and-builtins

Thanks!

I wanted to say thanks again to all Python contributors that helped get this solved. From my experience I felt welcome by the community and that they cared about the time and effort I put in.

I would say it was a great experience that I hope to repeat. And my cheers go to the Python community for setting up such an easy to follow process.

From Script to Binary, Creating single executables with Grotsky

2024-04-11T00:00:00Z

From Script to Binary, Creating single executables with Grotsky

Recently I added the possibility to embed compiled scripts to Grotsky, this makes it super easy to generate single executables that can be easily distributed.

Using the release v0.0.13 of Grotsky, a toy programming language that I've been developing for a while, you can compile scripts into bytecode and embed them into a single binary that can be easily distributed.

For now, it's only possible to embed a single script, so if your script needs to import something it won't work.

How embedding works

Grotsky by default generates a magic pattern at compile time. It's 512 bytes and is stored as a static variable.

To generate that pattern we use the const-random crate.

We use that to define a marker and identify if the Grotsky binary is running in embedded mode or not.

#[repr(C)]
struct Marker {
    magic_pattern: [u8; 512],
    is_embedded: u8,
}

const fn new_marker() -> Marker {
    Marker{
        magic_pattern: const_random!([u8; 512]),
        is_embedded: 0,
    }
}

static EMBEDDED_MARKER: Marker = new_marker();

Then we can use a very hacky trick to take a compiled script and generate an single executable with the embedded bytecode.

pub fn embed_file(compiled_script: String, output_binary: String) {
    // Get the path of the current executable (Grotsky interpreter)
    let exe_path = env::current_exe().unwrap();
    let mut exe_contents = read(exe_path).unwrap();
    let pattern = &EMBEDDED_MARKER.magic_pattern;

    // Find the magic pattern inside the executable. Given that is a static
    // variable with a value defined at compile time, it has to be stored in
    // the binary, we can find it and switch the `is_embedded` flag.
    if let Some(pos) = find_position(&exe_contents, pattern) {
        // We defined the Marker struct with a C representation
        // which means that right after the magic PATH we have a byte
        // that indicates if the interpreter is running in embedded mode or not.
        exe_contents[pos+512] = 1;

        // We add the magic pattern at the end of the executable again.
        // As a stop mark that right after that the bytecode will come.
        for i in 0..512 {
            exe_contents.push(pattern[i]);
        }

        // Now we read the compiled code and add it to the end of the new executable.
        let mut compiled_content = read(compiled_script).unwrap();
        exe_contents.append(&mut compiled_content);

        // We write a single file with the bytecode concatenated at the end.
        write(output_binary, exe_contents).unwrap();
    }
}

// Function to find the position of magic pattern in a stream of bytes
fn find_position(haystack: &Vec, needle: &[u8; 512]) -> Option {
    if haystack.len() < needle.len() {
        return None;
    }
    for i in 0..=haystack.len() - needle.len() {
        if &haystack[i..i + needle.len()] == needle.as_ref() {
            return Some(i);
        }
    }
    None
}

We're using the magic pattern as a stop mark. Our resulting binary will have the same magic pattern twice. First is the original that gets loaded as a global static variable. The second one is almost at the end of the file and indicates the beginning of the embedded bytecode.

We also need a function to detect if we're running under "embedded" mode. In that case the interpreter should only read the embedded bytecode and execute it.

pub fn is_embedded() -> bool {
    let embedded_indicator = &EMBEDDED_MARKER.is_embedded as *const u8;
    unsafe {
        // Need to perform this trick to read the actual memory location.
        // Otherwise during compilation Rust does static analysis and assumes
        // this function always returns the same value.
        return ptr::read_volatile(embedded_indicator) != 0;
    }
}

We change the value without the Rust compiler ever knowing, so we do a volatile read of the pointer to make sure we actually load the value from memory.

Otherwise the Rust compiler assumes that this always returns 0, because it is hardcoded in the new_marker function and is never changed in the codebase.

Now we can proceed to run in "embedded" mode.

pub fn execute_embedded() {
    // Get path of current executable
    let exe_path = env::current_exe().unwrap();
    interpreter::set_absolute_path(exe_path.clone().to_str().unwrap().to_string());

    let exe_contents = read(exe_path).unwrap();
    let pattern = &EMBEDDED_MARKER.magic_pattern;

    // The offset is 512 because that's the size of the magic pattern
    let offset: usize = 512;

    // Find first match (original)
    let first_match = find_position(&exe_contents, pattern).unwrap();

    // We try to find the second mark by reading what comes after the first one
    let remaining = &exe_contents[first_match+offset..].to_vec();
    let pos = find_position(remaining, pattern).unwrap();

    // The bytecode is located right after the second mark
    let compiled_content = &remaining[pos+offset..];
    
    // Run interpreter from bytecode
    if !interpreter::run_interpreter_from_bytecode(&compiled_content) {
        println!("Could not read embedded script");
        exit(1);
    }
}

With all those function only thing I need to do is add an if-statement to the main function in the Rust project to check if we're on embedded mode and proceed accordingly.

fn main() {
    if embed::is_embedded() {
        embed::execute_embedded();
        return;
    }
    // Continue executing normally
    // ...
}

That's it. That's all that takes to implement single binaries with Grotsky. Continue reading to see an example of how to actually use this feature.

Embedding example: Make your own Grep

Let's try to reproduce a simple version of the well-known Unix tool grep.

Store the following script in a file called grep.gr:

# Join a list of strings separated by space " "
fn join(list) {
	let out = ""
	for let i = 0; i < list.length; i = i + 1 {
		out = out + list[i]
		if i < list.length - 1 {
			out = out + " "
		}
	}
	return out
}

# Check that a pattern was provided
if process.argv.length == 1 {
	io.println("Usage:\n\tgrep [pattern ...]")
	return 1
}

# Join argv[1:] into a pattern
let pattern = join(process.argv[1:])

# Read first line
let line = io.readln()

# While we are not in EOF
#   Check that line matches pattern and print it
#   Consume next line
while line != nil {
	if re.match(pattern, line) {
		io.println(line)
	}
	line = io.readln()
}

Then it can be used like this:

$ cat file.txt | ./grotsky grep.gr pattern

And it will print all lines that match the "pattern".

We can also package it as a single binary by doing the following commands.

$ ./grotsky compile grep.gr
$ ./grotksy embed grep.grc

Now we should have a grep.exe in our directory. And we can use it:

$ chmod +x grep.exe
$ cat file.txt | ./grep.exe pattern

Should work the same as the previous example.

Using Go Generics to create a Single file JSON DB

2024-01-26T00:00:00Z

Using Go Generics to create a Single file JSON DB

Let's start with an example use case: you have a multitenant web server that serves pages for various customers, you can know how to which page to serve based on Host header in the request.

You can track each site in a struct like this:

type Site struct {
    Name    string  `json:"name"`
    Host    string  `json:"host"`
}

Your sites.json file could be something like this:

{
    "www.example.com": {
        "name": "Example",
        "host": "www.example.com"
    }
}

Which can be opened as a db:

sitesdb, err := sfjdb.Open[map[string]Site]("./sites.json")

Then you can simply get the content of the json file and validate requests:

func handle(w http.ResponseWriter, req *http.Request) {
    sites := sitesdb.View()
    host := req.Header["Host"][0]
    if site, ok := sites[host]; ok {
        // Render sites that are stored in DB
        renderSite(w, req, site)
    } else {
        w.WriteHeader(404)
    }
}

Implementation

Thanks to Go generics and built-in json Marshaling/Unmarshaling this is quite easy to build:

type DB[T any] struct {
	rw       sync.RWMutex
	data     T
	filepath string
}

We have a DB struct to keep track of the file, data and a RW mutex.

Then we can easily read data from a local file:

// Load loads data from file.
func (db *DB[T]) Load() error {
	db.rw.Lock()
	defer db.rw.Unlock()
	content, err := os.ReadFile(db.filepath)
	if err != nil {
		return err
	}
	return json.Unmarshal(content, &db.data)
}

Each time we use the data in the DB we make a copy, to avoid modifying some attributes via pointers by mistake.

func objcopy[T any](obj T) *T {
	data, err := json.Marshal(obj)
	if err != nil {
		panic(err)
	}
	newobj := new(T)
	if err := json.Unmarshal(data, newobj); err != nil {
		panic(err)
	}
	return newobj
}


// View returns a copy of the data.
func (db *DB[T]) View() T {
	db.rw.RLock()
	defer db.rw.RUnlock()
	return *objcopy(db.data)
}

We can also save a new state in case we modified something (e.g. added a new site to our list).

// Save saves a copy of the data as plain json in the file.
func (db *DB[T]) Save(data T) error {
	db.rw.Lock()
	defer db.rw.Unlock()
	db.data = *objcopy[T](data)
	content, err := json.Marshal(db.data)
	if err != nil {
		return err
	}
	return WriteFile(db.filepath, content, 0644)
}

You can get the WriteFile function here.

This function writes the file atomically, which means is sucessfully written or not written at all.

Finally we have to implement Open to instantiate our DB:

// Open opens a json file as a database.
func Open[T any](filepath string) (db *DB[T], err error) {
	db = &DB[T]{filepath: filepath}
	if err := db.Load(); err != nil {
		return nil, err
	}
	return db, nil
}

In praise of Go generics

At the beginning I was a little sad about generics, go the simple language was at end becoming a huge complex beast. But overall if you don't over-abuse generics it feels natural and easy to use/understand.

This still feels like the good old Go, but just a bit more power in the right way.

If you wish to use this in your project you can just import it from https://github.com/mliezun/sfj-db.

Generating posts using markdown

2024-01-04T00:00:00Z

Generating posts using markdown

This is pretty standard for Github pages. But in this case, the parser has been written by me. It takes some subset of markdown and compiles it to HTML.

Only what you see in this post is what's supported.

Code blocks

Standard code block:

Hello, this is a code block!

Syntax highlighting:

from functools import reduce, partial
import operator

mul = partial(reduce, operator.mul)
print("Factorial of 5:", mul(range(1, 6)))

Unordered list

This
is an
unordered list

Bolds, Italics and Inline code

Some text can be bolded, while some other can be in Italics.

But the best is to have print("inline code").

Links and images

Link to the blog source code where you can see how the parser works (tldr: is awful).

Picture of NYC:

HEADINGS

There's a thing you haven't noticed so far. There's support for different kinds of headings. You can see them in increasing order here:

Microscopic

Small

Good

Pretty good

Big

Good bye!

Thanks for checking out the blog. I've done this to reduce the complexity of creating new blogposts. It was a headache before.

Hopefully now I'll write more. Stay tuned.

Day 20. My favourite problem from Advent of Code 2023

2023-12-25T00:00:00Z

Day 20. My favourite problem from Advent of Code 2023

Im gonna briefly describe the problem here, but if you want to see the real thing go check it out https://adventofcode.com/2023/day/20.

I like it because it involves some simple electronic devices that are wired together and send pulses/signals to each other. In this problem you have to make sure to correctly propagate the signals and simulate the behaviour of the devices.

There are two devices that have a very distinct behaviour:

Flip flops: similar to a T flip-flop electronic device.

Conjunctions: similar to a NAND gate with memory on its inputs.

In this problem, Flip flops are initially off and whenever they reiceve a low pulse they toggle between on/off. Each time it toggles state it sends a pulse as an output. When turned off sends a low pulse, when turned on sends a high pulse.

Conjunction modules remember the most recent pulse on each input. By default it remembers a low pulse for all inputs. When a pulse is received it updates the memory for that input. Then, if it remembers high pulses for all inputs, it sends a low pulse; otherwise, it sends a high pulse.

There is also some "dummy" modules:

Broadcaster: has 1 input and N outputs. It replicates the input in all its outputs.

Button: when pressed sends a low pulse. The button is always connected as the broadcaster input. This is similar to a normally closed switch.

Test module: module that receive and process inputs but has no output.

One important thing to have in mind is that modules only send output pulses when they receive a pulse as input.

Problem input

The example input looks something like this:

broadcaster -> a, b, c%a -> b%b -> c%c -> inv&inv -> a

There will always be just one Broadcaster module called "broadcaster" that has the Button connected as input. In this case it has module's "a", "b" and "c" connected to its output.

The arrow -> indicates what modules are connected to the output of the module to the left.

Lines that start with % means the module is a Flip flop, for example: %a -> b indicates that there's a flip flop called "a" whose output is connected to module's "b" input.

Lines that start with & means the module is a Conjunction, for example: &inv -> a indicates that there's a conjunction called "inv" whose output is connected to module's "a" input.

Let's analyze how this circuit behaves once the button is pushed:

button -0-> broadcasterbroadcaster -0-> abroadcaster -0-> bbroadcaster -0-> ca -1-> bb -1-> cc -1-> invinv -0-> aa -0-> bb -0-> cc -0-> invinv -1-> a

In this example 8 low (0) pulses and 4 high (1) pulses are sent.

Part 1

To solve the first part we need to calculate the multiplication between high and low pulses sent between devices.

In the previous example that would be 8*4=32.

But this time we don't push the button only once, but we push it a 1000 times. Each time we push the button we wait until all signals propagate and the circuit settles into a state before pushing the button again.

Solution

First I started by modelling the devices as objects. Starting with a single base class that has most of the common behaviour.

from abc import ABCmeasure_pulses = {0: 0, 1: 0}class Module(ABC):    def __init__(self, name: str):        self.name = name        self.outputs = []    def receive_pulse(self, mod: "Module", pulse: int) -> list[tuple["Module", int]]:        measure_pulses[pulse] += 1        print(f"{mod and mod.name} -{pulse}-> {self.name}")        return self.process_pulse(mod, pulse)    def connect_output(self, mod: "Module"):        self.outputs.append(mod)    def propagate_pulse(self, pulse: int):        mods = []        for m in self.outputs:            mods.append((m, pulse))        return mods    def process_pulse(self, mod: "Module", pulse: int):        raise NotImplementedError()    def __str__(self) -> str:        return f"{self.__class__.__name__}(name={self.name})"    def __repr__(self) -> str:        return str(self)

What we see here is that we expect all modules to have a name and outputs. See __init__(), __str__(), __repr__() and connect_output().

Each module can receive a pulse 0 or 1 from another module. See receive_pulse(). Each time we process a pulse we record it in a global dict called measure_pulses.

Also we leave process_pulse() to be defined by each particular module type.

We have a method that returns a list of all modules to which signals should be propagated. See propagate_pulse().

Let's start by the easiest module type:

class TestModule(Module):    def process_pulse(self, mod: "Module", pulse: int):        return []

Give that it's a dummy module, it doesn't do anything when it receives an input.

class Broadcaster(Module):    def process_pulse(self, mod: "Module", pulse: int):        return super().propagate_pulse(pulse)

As expected the Broadcaster always propagates the received input to all its outputs.

class FlipFlop(Module):    def __init__(self, name: str):        super().__init__(name)        self.state = 0    def process_pulse(self, mod: "Module", pulse: int):        if pulse == 0:            self.state = (self.state + 1) % 2            return super().propagate_pulse(self.state)        return []

The flip flop start initially turned off. See self.state = 0 in __init__().

In process_pulse() we implement the behaviour:

If receives a low pulse, toggles the state and sends a pulse equals to the state to all its outputs.

Otherwise it doesn't do anything.

class Conjunction(Module):    def __init__(self, name: str):        super().__init__(name)        self.memory = {}    def remember_input(self, mod: Module):        self.memory[mod.name] = 0    def process_pulse(self, mod: Module, pulse: int):        self.memory[mod.name] = pulse        if all(self.memory.values()):            return self.propagate_pulse(0)        return self.propagate_pulse(1)

The conjunction initializes its memory as empty. See __init__().

Each time a module is plugged in as an input it remembers it as OFF (0). See remember_input().

The way it processes pulses is by first recording the pulse for the input in its memory. Then if all inputs are 1s it sends a 0 pulse to all its outputs.

Otherwise it sends a 1 pulse to all its outputs.

At this point we have all our building blocks for solving this problem. We only need to parse the input and something that pushes the button and makes sure signals are propagated to the end.

Parsing modules is straightforward:

def parse_modules(modules: list) -> dict[str, Module]:    modules_by_name = {}    outputs_by_name = {}    # Parse all modules into their correspondig class and store    # them in a dict.    for m in modules:        module_type = m[0]        module_outputs = [o.strip() for o in m[1].split(",") if o.strip()]        if module_type.startswith("broadcaster"):            modules_by_name[module_type] = Broadcaster(module_type)            outputs_by_name[module_type] = module_outputs        elif module_type.startswith("%"):            modules_by_name[module_type[1:]] = FlipFlop(module_type[1:])            outputs_by_name[module_type[1:]] = module_outputs        elif module_type.startswith("&"):            modules_by_name[module_type[1:]] = Conjunction(module_type[1:])            outputs_by_name[module_type[1:]] = module_outputs    # Once all the modules are parsed use connect their outputs.    # If the module doesn't exist at this point is a TestModule.    # If the module is a Conjunction, call remember_input().    for name, outputs in outputs_by_name.items():        for mod_name in outputs:            mod = modules_by_name.get(mod_name, TestModule(mod_name))            modules_by_name[name].connect_output(mod)            if isinstance(mod, Conjunction):                mod.remember_input(modules_by_name[name])    return modules_by_name

If we parse our example using that function we will receive a dictionary as its output. Keys are module names and values are the objects representing the module.

If we parse the example we get something like this:

example = """broadcaster -> a, b, c%a -> b%b -> c%c -> inv&inv -> a"""example_modules = [m.split(" -> ") for m in example.splitlines() if m.strip()]print(parse_modules(example_modules))# Output{    'broadcaster': Broadcaster(name=broadcaster),    'a': FlipFlop(name=a),    'b': FlipFlop(name=b),    'c': FlipFlop(name=c),    'inv': Conjunction(name=inv)}

Then we need a function that pushes the button and makes sure all signals are propagated:

def push_button(modules_by_name: dict[str, Module]):    broad = modules_by_name["broadcaster"]    queue = [(broad, broad.receive_pulse(None, 0))]    while queue:        current, signals = queue.pop(0)        for mod, pulse in signals:            queue.append((mod, mod.receive_pulse(current, pulse)))

Here, we lookup the broadcaster module by name. And send a pulse (note that we pass None as the module because we didn't implement a button class) to the broadcaster.

We store the current module (broadcaster) along with all the propagated signals (return value from receive_pulse()) in a queue to be processed.

While the signal queue to be processed is not empty we do the following:

Extract the first element of the queue.

Go trough all the signals that this element is sending.

Send the pulses to each corresponding module and store them in the queue to be processed.

This process will stop when all responses from receive_pulse() are empty and there are no more signals added to the queue.

If we run this for our example:

example_modules = parse_modules(example_modules)push_button(example_modules)# OutputNone -0-> broadcasterbroadcaster -0-> abroadcaster -0-> bbroadcaster -0-> ca -1-> bb -1-> cc -1-> invinv -0-> aa -0-> bb -0-> cc -0-> invinv -1-> a

It looks the same as when we analyzed the example above!!

We're ready for processing our problems input. (Remeber to comment out the print statement inside receive_pulse()).

modules = open("input.txt", "r").read().strip()modules = [m.split(" -> ") for m in modules.splitlines() if m.strip()]modules = parse_modules(modules)for _ in range(1000):    push_button(modules)print("result:", measure_pulses[0] * measure_pulses[1])# Outputresult: x

Based on your problem input x will be the solution.

Part 2

This part as always is much trickier than the first part. It doesn't involve much code changes, just figuring out a way of avoiding large computations.

For this part, the problem tells us that there's a module called rx. And we need to find out the lowest amount of button pulses that will make the rx module receive a low pulse.

As I have learned troughout this entire challenge, just nahively letting it run and see when the rx module gets a low signal will get me nowhere. It will run forever.

So, taking a look at the input and see what the rx module is actually connected to might provide some guidance.

Following is for my case (I don't know if all problem inputs are the same). Looking up "rx" in the input I find a single line:

...&lv -> rx...

That means rx is a TestModule (a dummy module that has nothing connected to its output). And that has only one input: a Conjunction called lv.

Ok, that feels like progress. Let's see what lv is connected to:

...&st -> lv&tn -> lv&hh -> lv&dt -> lv...

Other 4 Conjunctions are connected as inputs of lv. That's interesting. Because lv is a Conjuction it means that to send the low pulse required by rx it should receive all high pulses from its inputs.

The solution from here is kind of intuitive at this point. If we figure out how many button pulses does it take for each of the input devices to send a 1 signal we can multiply them together and get the result.

I'll explain better. Let's say st sends a 1 on every button push, tn sends a 1 every second button push (this means you have to press the button twice to get tn to send a 1 as an output), hh sends a 1 every fourth button push and dt sends a 1 every eighth button push.

So it looks like this:

module | pushes---------------  st   |   1  tn   |   2  hh   |   4  dt   |   8

In this example, if we push the button 8 times. They are all gonna send a high pulse. Because 8 is divisible by 1, 2, 4 and 8.

If the table were different:

module | pushes---------------  st   |   1  tn   |   3  hh   |   5  dt   |   7

In this case there's no obvious number of times we should push the button. But if we multiply the numbers together we get a number that is divisible by every number in the table. Pushing the button 1 * 3 * 5 * 7 = 105 times will make all the outputs send a 1, and consequently rx will receive a 0.

Solution

What we need to do then is to figure after out how many button presses we get a 1 on each of those modules.

from collections import defaultdict# Store number of button presses in a global variableITERATIONS = 0# Store high pulses for target modulesOUT_PULSES = defaultdict(list)class Conjunction(Module):    def __init__(self, name: str):        super().__init__(name)        self.memory = {}    def remember_input(self, mod: Module):        self.memory[mod.name] = 0    def start_recording(self):        self.recording = True    def record(self):        if hasattr(self, "recording"):            OUT_PULSES[self.name].append(ITERATIONS)    def process_pulse(self, mod: Module, pulse: int):        self.memory[mod.name] = pulse        if all(self.memory.values()):            return self.propagate_pulse(0)        self.record()        return self.propagate_pulse(1)

We introduced 2 new methods to the conjunction module: start_recording() and record(). The first just initializes a bool attribute. And the second makes sure to only record high pulses for objects that have been initialized (method start_recording() called).

Also introduced 2 global variables: ITERATIONS to keep track of button pushes and OUT_SIGNALS to track each time one of the modules outputs a high pulse.

Now we need to make those specific modules record their outputs:

# Get the "lv" module by namelv = modules["lv"]lv_inputs = [modules[k] for k in lv.memory.keys()]for m in lv_inputs:    m.start_recording()

I wasn't sure if the cycle was always going to be the same, so just to be sure I did 100_000 button pushes and recorded all the "1" outputs for those modules.

for i in range(100_000):    ITERATIONS += 1    push_button(modules)print(OUT_PULSES)# Output{'hh': [3769, 7538, 11307, 15076, 18845, 22614, 26383, 30152, 33921, 37690, 41459, 45228, 48997, 52766, 56535, 60304, 64073, 67842, 71611, 75380, 79149, 82918, 86687, 90456, 94225, 97994], 'tn': [3863, 7726, 11589, 15452, 19315, 23178, 27041, 30904, 34767, 38630, 42493, 46356, 50219, 54082, 57945, 61808, 65671, 69534, 73397, 77260, 81123, 84986, 88849, 92712, 96575], 'st': [3929, 7858, 11787, 15716, 19645, 23574, 27503, 31432, 35361, 39290, 43219, 47148, 51077, 55006, 58935, 62864, 66793, 70722, 74651, 78580, 82509, 86438, 90367, 94296, 98225], 'dt': [4079, 8158, 12237, 16316, 20395, 24474, 28553, 32632, 36711, 40790, 44869, 48948, 53027, 57106, 61185, 65264, 69343, 73422, 77501, 81580, 85659, 89738, 93817, 97896]}

We can observe that for each module we have a periodicity given by:

hh = n*3769tn = n*3863st = n*3929dt = n*4079

This means we can just multiply the first element of each list for each module and we'll get our result.

In my case it was:

accum = 1for name, pulses in OUT_PULSES.items():    accum *= pulses[0]print("result:", accum)# Outputresult: 233338595643977

Edit: As some people have pointed out in the HN discussion, just multiplying the numbers together only works because the numbers are coprimes and the correct solution is to use LCM.

Closing words

This problem is my favourite because it has a few characteristics that I personally enjoy:

It's based on real world stuff. In this case electronic devices (which is also a plus because they're fun).

It can be easily translated to an OOP approach which makes it easy to implement and understand.

To solve the second part you need to look at the data and make a solution for your particular input.

It doesn't involve any Graph traversal or specific Math, Calculus or Algebra knowledge. Or any obscure CS algorithm.

In the end this is one of my favourites because to solve it you just have to understand the problem and understand the data.

Link to my github project with the solutions https://github.com/mliezun/aoc.

I rewrote my toy language interpreter in Rust

2023-11-23T00:00:00Z

I rewrote my toy language interpreter in Rust

Im rewriting Grotsky (my toy programming language) in Rust, the previous implementation was done in Go. The goal of the rewrite is to improve my Rust skills, and to improve the performance of Grotsky, by at least 10x. This has been a serious of posts, this one is the latest one. Hopefully the best and most insightful of them all.

In previous posts:

Rewrite my toy language interpreter in Rust

Rewrite my toy language interpreter in Rust, an update

I've outlined a plan to migrate Grotsky to a Rust based platform.

Originally, my plan was very ambitious and I thought I would be able to finish the transition in like two months. In reality it took five months :-)

Performance improvement

I was aiming at a 10x improvement. In reality is not that much. I ran various benchmarks and get at most a 4x improvement. Which is not great, but also not that bad given that I know very little of how to do high performance Rust. The interpreter is written in the most dumbest and easiest way I managed to do it.

Let's look at some numbers for different programs.

Loop program

let start = io.clock()
fn test() {
    let a = 1
    while a < 10000000 {
        a = a + 1
    }
}
test()
io.println(io.clock() - start)

The result is:

trial #1
build/grotsky   best 5.135s  3.877x time of best
build/grotsky-rs   best 1.325s  287.6800% faster
trial #2
build/grotsky   best 5.052s  3.814x time of best
build/grotsky-rs   best 1.325s  281.4182% faster
trial #3
build/grotsky   best 5.035s  3.802x time of best
build/grotsky-rs   best 1.325s  280.1663% faster
trial #4
build/grotsky   best 5.003s  3.777x time of best
build/grotsky-rs   best 1.325s  277.6831% faster
trial #5
build/grotsky   best 5.003s  3.777x time of best
build/grotsky-rs   best 1.325s  277.6831% faster

Recursive fibonacci

let start = io.clock()
fn fib(n) {
	if n <= 2 {
		return n
	}
	return fib(n-2) + fib(n-1)
}
fib(28)
io.println(io.clock() - start)

The result is:

trial #1
build/grotsky   best 0.8409s  294.5155% faster
build/grotsky-rs   best 3.317s  3.945x time of best
trial #2
build/grotsky   best 0.8168s  271.3829% faster
build/grotsky-rs   best 3.033s  3.714x time of best
trial #3
build/grotsky   best 0.797s  245.6835% faster
build/grotsky-rs   best 2.755s  3.457x time of best
trial #4
build/grotsky   best 0.7784s  249.9964% faster
build/grotsky-rs   best 2.724s  3.5x time of best
trial #5
build/grotsky   best 0.7784s  249.9964% faster
build/grotsky-rs   best 2.724s  3.5x time of best

In this case is like 3.5x slower. This is due to function calls. Im not very well versed in Rust, so on each call im copying a lot of data over and over. In the go implementation everything is just pointers so there's less copying.

Compile to bytecode

With the Rust implementation, generating and compiling to bytecode was added. Now it's possible to generate a bytecode file to later read it. This is a way of distributing files without giving away source code and also a little bit more performant because you skip parsing and compilation phases.

How it works:

grotsky compile example.gr  # Compile file
grotsky example.grc  # Run compiled file

Memory model

Grotsky is a reference-counted language. We're using Rust's Rc and RefCell to keep track of values.

pub struct MutValue(pub Rc>);

impl MutValue {
    pub fn new(obj: T) -> Self {
        MutValue::(Rc::new(RefCell::new(obj)))
    }
}

pub enum Value {
    Class(MutValue),
    Object(MutValue),
    Dict(MutValue),
    List(MutValue),
    Fn(MutValue),
    Native(NativeValue),
    Number(NumberValue),
    String(StringValue),
    Bytes(BytesValue),
    Bool(BoolValue),
    Slice(SliceValue),
    Nil,
}

pub enum Record {
    Val(Value),
    Ref(MutValue),
}

pub struct VM {
    pub activation_records: Vec,
}

Most of the simple values are just stored as-is: Native (builtin functions), Number, String, Bytes, Bool, Slice and Nil.

For the other complex values we need to use 'pointers' which in this case are MutValue.

Then the Grotsky VM uses Records which can be a plain Value or a reference to a Value. The records are registers, each function has up to 255 registers. The reference to values are used to store upvalues. A register is turned into an upvalue when a variable is closed by another function.

This implementation ends up being very slow, but easy to manage. Because Rust stdlib does all the work.

Using Rust in this blogpost

As you may know, this blog is powered by grotsky. Im happy to say that I successfully migrated from grotsky to grostky-rs as the backend for the blog. And what you're reading now is generated by the latest implementation of the language using Rust.

Even for local development the Rust version is used. Which means Im using a TCP server and an HTTP implementation written in Grotsky.

Closing remarks

This has been a great learning, Im happy to have finished because it required a lot of effort. Im not gonna announce any new work on this interpreter but I would like to keep adding stuff. Improving it further to make it more performant and more usable.

In the end I encourage everyone to try it and also start their own project. Is always cool to see what everyone else is doing.

Thanks for reading.

Rewrite my toy language interpreter in Rust, an update

2023-09-23T00:00:00Z

Rewrite my toy language interpreter in Rust, an update

In a previous post I've outlined a plan to migrate Grotsky to a Rust based platform.

I was suposed to finish it more than a month ago :').

Of course, I wasn't able to do it.

I think my original estimation was actually very flawed, but I also didn't put enough work.

I failed my estimation, because I decided to skip the step of first writing a Tree-based interpreter in Rust. To go directly to a Register-based virtual machine.

The reason that is taking me so long is that I travel a lot. I've been a digital nomad for more than a year now. And when I finish working I prefer to go outside and explore the place where Im staying. I go to a lot of interesting places: at this time Im in Turkey and heading to Hong Kong, after that Im going to South Korea. So, it makes sense to actually experience the places where Im staying than to stay inside writing code.

Im here to propose a new roadmap that is more achivable with the time I have to work on this. And the idea is to finish it before the end of the year.

Plus, I recently heard some advice that I think it's worth to try: Work on someone for 15 minutes every day. I do have 15 minutes everyday where Im probably scrolling through social media or just consuming content. I can make better use of that time by putting it into this project.

Updated Roadmap

Sep 30: Publish a blogpost about the memory model of the Rust implementation.
Oct 15: Migrate automated tests to the new backend and make sure they pass.
Oct 30: Implement stdlib in bytecode interpreter. Share results.
Nov 15: Add ability to compile to bytecode and run from bytecode.
Dec 30: Finish up everything and publish a blogpost update before end of year.

This is gonna be rough, because of the traveling and being jetlaged all the time. But I think the roadmap is easy enough so I can do it.

Thanks for reading.

The end of a side project

2023-07-15T00:00:00Z

The end of a side project

Cloud Outdated was a personalized digest of updates for cloud services. It's sad to see it go, but it was a fun project to work on, learn some new stuff and collab with a friend. There are some takeaways from this that I'd like to share.

Building something simple is hard

From the beginning we were trying to build a simple product to get notified when new versions of Python were supported in AWS Lambda. But we figured we could support other things too, like all Lambda runtimes, then also GCP and Azure, then more services of each one.Features started piling up pretty quickly.

When building stuff I try to think of every edge case, even the most improbable ones, and make the software infalible. Of course, it's impossible to succeed at that, software will always be fallible.And this premature optimization ends up making the project more complex than it should be.

We planned to work on this for a 1 or 2 months, and it ended up taking 6+ months :-).

My takeaway here is: start building the dumbest thing possible that gets the job done, then adjust as needed.

Getting users is hard

We're killing this project because nobody uses it.And nobody except us has used it since it was launched more than a year ago.Some people subscribed but never even opened an email.

We tried to advertise in our social media and post it in different builders communities.But that will get you so far if you're not an influencer that has the right audience.

We thought that we'll get more traffic from organic search or people telling their friends about this.But in the end I think nobody really needed something like this that much.

My takeaway here is: building something that people really want and getting the product to the hands of the people that want it is very complicated. You should think very deeply about what problem your product is solving and how your users will find you.

The money

This has costed like $200 (US dollars) since inception. For two people that's like $100 each.It's not a lot, but for something that has no users that's quite expensive.

We used Lambdas to serve this project.I feel like we were promised that the cloud and serverless are cheap and easy solution.But in my opinion it doesn't seem to be the case.It's definitely not easier nor cheaper than having a PHP website in this case.Im sure there are other cases in which it makes more sense.

This is also a reason of why we're killing the project.Our service started failing because after a deploy dependencies were updated and the code was to big to fit on a Lambda.It would have been a lot of work to fix it, so we decided to kill it and save some bucks every month.

For personal projects which you're not sure how they're going to scale, I think serverless is probably not the right choice.Serverless make sense if you're going to have huge burst of traffics and don't want to handle a lot of infra.

My takeaway here is: beware of the cloud, if you're just building a small side project or don't have huge infra needs stick with the cheapest providers (Hostinger, PythonAnywhere, Hetzner) and avoid cloud providers (GCP, Azure, AWS).

Final thougths

If I haven't made this clear enough, building a successful product is *hard*.There are many things to think about when starting, and the technical stuff hopefully is the easy part.I think this are the 3 most important lessons that I've learned working on this:

Build the dumbest thing that does the job, improve as needed.
Think deeply about what problem are you solving and how you're going to deliver the solution to the people that need it.
Beware of the cloud, if possible use a cheaper provider. It will save you money and headaches.

Rewrite my toy language interpreter in Rust

2023-06-02T00:00:00Z

Rewrite my toy language interpreter in Rust

As of this writting the Rust implementation is 4x faster than the Golang implementation.

I've rewritten the Lexer, Parser and part of the Runtime. Enough for run this code and measure how long it takes for each implementation to finish:

let a = 1
while a < 100000000 {
    a = a + 1
}

I was inspired by Mitchel Hashimoto's post: My Approach to Building Large Technical Projects. And I want to create a roadmap of little projects to reach my goal of having a full-fledged interpreter in Rust.

Goal

Until now Grotsky has been running on a Tree-based interpreter. My goal is that at the end of the rewrite I will have a Bytecode interpreter.

First I want to rewrite the Tree-based interpreter in Rust and achieve at least 10x performance improvement.

Then figure out if I want to use a Register based or Stack based bytecode interpreter.

Also, I would like to have a stable bytecode representation to be able to compile programs to a binary format that can be shipped as is or packaged into a binary.

Finally, it's time Grotsky gets a REPL.

Roadmap

Believe it or not, Grotsky it's pretty big. It has support for reading and writting files, sockets and more on the stdlib. Which means I have a huge task ahead of me.

First thing I want to do is to have some sort of automated testing setup that can compare Rust vs Go implementation. Right now all test are written as Go unit test, I need to make them agnostic of language backend.

Jun 9: Have a complete setup of automated tests for correctness and performance.
Jun 16: Language runtime (without stdlib) rewritten in Rust.
Jun 23: Finish migrating stdlib and publish results (new blog post). Use new implementation for blog engine.
Jun 30: Decide which kind of bytecode interpreter to use, then make a design and plan for the implementation.
Jul 7: Have a working bytecode interpreter that is able to run the program shown in this post ^. Just a simple while loop. Compare performance and share progress.
Jul 14: Add support for functions and closures.
Jul 21: Finish runtime without stdlib in bytecode interpreter.
Jul 28: Implement stdlib in bytecode interpreter. Share results.
Aug 5: Add ability to compile to bytecode and run from bytecode.
Aug 12: Add REPL and finish up project.

Im gonna have lots of fun with this project and Im sure next time a post here I'll be a changed man. Surely gonna learn a lot, Im excited about what lies ahead.

Writing a Redis clone in Go from scratch

2023-04-08T00:00:00Z

Writing a Redis clone in Go from scratch

In this post we're going to write a basic Redis clone in Go that implements the most simple commands: GET, SET, DEL and QUIT. At the end you'll know how to parse a byte stream from a live TCP connection, and hopefully have a working implementation of Redis.

What's intersting about this project is that it's production ready (not really). It's being used in production in an old Web app that I made for a client in 2017. It has been running for a few months now without issues.

I mantain that app to this day and I charge like 50 bucks a month for it. I do it because Im friends with the person that uses the app.

Long story short, the app's backend is written in PHP and uses Redis for caching, only GET, SET and DEL commands. I asked my friend if I could replace it with my custom version and said yes, so I decided to give it a go.

If you're looking for C/C++ implementation, go check out this book.

What we'll be building

If you go to the command list on redis webpage you'll see that there are 463 commands to this day (maybe more if you're in the future).

That's a crazy number. Here, we're only implementing 4 commands: GET, SET, DEL, QUIT, the other 459 commands are left as an exercise to the reader.

GET

GET key

Returns the value referenced by key. If the key does not exist then nil is returned.

SET

SET command gains more features on newer versions of Redis. We're going to implement one that has all features that were realeased up until version 6.0.0.

SET key value [NX | XX] [EX seconds | PX milliseconds]

Stores value as a string that is referenced by key. Overwrites any data that was previously referenced by the key.

Options

EX seconds -- Set the specified expire time, in seconds.
PX milliseconds -- Set the specified expire time, in seconds.
NX -- Only set the key if it does not already exist.
XX -- Only set the key if it already exist.

DEL

DEL key [key ...]

Takes 'any' amount of keys as input and removes all of them from storage. If a key doesn't exist it is ignored. Returns the amount of keys that were deleted.

QUIT

QUIT

When receiving this command the server closes the connection. It's useful for interactive sessions. For production environments the client should close the connection without sending any commands.

Examples

Let's start an interactive session of redis to test some commands. We can install redis-server with docker and run it locally. Then we can use telnet to connect directly via TCP. Open a terminal and execute the following instructions:

$ docker run -d --name redis-server -p 6379:6379 redis:alpine

$ telnet 127.0.0.1 6379
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

At this point the prompt should be waiting for you to write something. We're gonna test a couple of commands. In the code boxes below the first line is the command, following lines are the response.

GET a
$-1

^ That weird $-1 is the special nil value. Which means there's nothing stored here.

set a 1
+OK

^ First thing to notice here is that we can use lowercase version of SET. Also, when the command is successful returns +OK.

set b 2
+OK

SET c 3
+OK

^ Just storing a couple more values.

GET a
$1
1

^ Here the response is returned in two lines. First line is the length of the string. Second line is the actual string.

get b
$1
2

^ We can also use lowercase version of GET, I bet commands are case-insensitive.

get C
$-1

^ Testing with uppercase C gives a nil. Keys seem to be case-sensitive, probably values too. That makes sense.

del a b c
:3

^ Deleting everything returns the amount of keys deleted. Integers are indicated by ':'.

quit
+OK
Connection closed by foreign host.

^ When we send QUIT, the server closes the connection and we're back to our terminal session.

With those tests we have enough information to start building. We learned a little bit about the redis protocol and what the responses look like.

Sending commands

Until now we've been using the inline version of redis command. There's another kind that follows the RESP (Redis serialization protocol).

The RESP protocol is quite similar to what we've seen in the examples above.The most important addition is arrays. Let's see a Client<>Server interaction using arrays.

Client

*2
$3
GET
$1
a

Server

$-1

The server response looks the same as in the inline version. But what the client sends looks very different:

In this case, the first thing the client sends is '*' followed by the number of elements in the array, so '*2' indicates that there are 2 elements in the array and they would be found in the following lines.
After that we have '$3' which means we're expecting the first element to be a string of length 3. Next line is the actual string, in our case is the command 'GET'.
The next value is also a string and is the key passed to the command.

That's almost everything we need to start building a client. There's one last thing: error responses.

-Example error message
-ERR unknown command 'foo'
-WRONGTYPE Operation against a key holding the wrong kind of value

A response that starts with a '-' is considered an error. The first word is the error type. We'll only gonna be using 'ERR' as a generic response.

RESP protocol is what client libraries use to communicate with Redis. With all that in our toolbox we're ready to start building.

Receiving connections

A crucial part of our serve is the ability to receive client's information. The way that this is done is that the server listens on a TCP port and waits for client connections. Let's start building the basic structure.

Create a new go module, open main.go and create a main function as follows.

package main

import (
	"bufio"
	"fmt"
	"log"
	"net"
	"strconv"
	"strings"
	"sync"
	"time"
)

var cache sync.Map

func main() {
	listener, err := net.Listen("tcp", ":6380")
	if err != nil {
		log.Fatal(err)
	}
	log.Println("Listening on tcp://0.0.0.0:6380")

	for {
		conn, err := listener.Accept()
		log.Println("New connection", conn)
		if err != nil {
			log.Fatal(err)
		}

		go startSession(conn)
	}
}

After declaring the package and imports, we create a global sync.Map that would be our cache. That's where keys are gonna be stored and retrieved.

On the main function we start listening on port 6380. After that we have an infinite loop that accepts new connections and spawns a goroutine to handle the session.

Session handling

// startSession handles the client's session. Parses and executes commands and writes
// responses back to the client.
func startSession(conn net.Conn) {
	defer func() {
		log.Println("Closing connection", conn)
		conn.Close()
	}()
	defer func() {
		if err := recover(); err != nil {
			log.Println("Recovering from error", err)
		}
	}()
	p := NewParser(conn)
	for {
		cmd, err := p.command()
		if err != nil {
			log.Println("Error", err)
			conn.Write([]uint8("-ERR " + err.Error() + "\r\n"))
			break
		}
		if !cmd.handle() {
			break
		}
	}
}

It's super important that we close the connection when things are done. That's why we set a deferred function, to close the connection when the session finishes.

After that we handle any panics using recover. We do this mainly because at some point we might be reading from a connection that was closed by the client. And we don't want the entire server to die in case of an error.

Then we create a new parser and start trying to parse commands from the live connection. If we encounter an error we write the error message back to the client and we finish the session.

When cmd.handle() returns false (signaling end of session) we break the loop and the session finishes.

Parsing commands

Basic parser structure:

// Parser contains the logic to read from a raw tcp connection and parse commands.
type Parser struct {
	conn net.Conn
	r    *bufio.Reader
	// Used for inline parsing
	line []byte
	pos  int
}

// NewParser returns a new Parser that reads from the given connection.
func NewParser(conn net.Conn) *Parser {
	return &Parser{
		conn: conn,
		r:    bufio.NewReader(conn),
		line: make([]byte, 0),
		pos:  0,
	}
}

This is pretty straight-forward. We store a reference to the connection, a reader and then some attributes that will help us with parsing.

The NewParser() function should be used as a contructor for Parser objects.

We need some helper functions that will make parsing easier:

func (p *Parser) current() byte {
	if p.atEnd() {
		return '\r'
	}
	return p.line[p.pos]
}

func (p *Parser) advance() {
	p.pos++
}

func (p *Parser) atEnd() bool {
	return p.pos >= len(p.line)
}

func (p *Parser) readLine() ([]byte, error) {
	line, err := p.r.ReadBytes('\r')
	if err != nil {
		return nil, err
	}
	if _, err := p.r.ReadByte(); err != nil {
		return nil, err
	}
	return line[:len(line)-1], nil
}

Also quite simple.

current(): Returns the character being pointed at by pos inside the line.
advance(): Point to the next character in the line.
atEnd(): Indicates if we're at the end of the line.
readLine(): Reads the input from the connection up to the carriage return char. Skips the '\n' char.

Parsing strings

In Redis we can send commands like so:

SET text "quoted \"text\" here"

This means we need a way to handle \, " chars inside a string.

For that we need a special parsing function that will handle strings:

// consumeString reads a string argument from the current line.
func (p *Parser) consumeString() (s []byte, err error) {
	for p.current() != '"' && !p.atEnd() {
		cur := p.current()
		p.advance()
		next := p.current()
		if cur == '\\' && next == '"' {
			s = append(s, '"')
			p.advance()
		} else {
			s = append(s, cur)
		}
	}
	if p.current() != '"' {
		return nil, errors.New("unbalanced quotes in request")
	}
	p.advance()
	return
}

From the functions that we've declared up to this point it's pretty clear that our parser will be reading the input line by line. And the consuming the line one char at a time.

The way consumeString() works is quite tricky. It assumes that the initial " has been consumed before entering the function. And it consumes all characters in the current line up until it reaches the closing quotes character or the end of the line.

Inside the loop we can see that we're reading the current character and advancing the pointer, then the next character. When the user is sending an escaped quote inside the string we detect that by checking the current and the next characters. In this special case we end up advancing the pointer twice. Because we consumed two: chars the backslash and the quote. But we added only one char to the output: ".

We append all other characters to the output buffer.

When the loop finishes, if we're not pointing to the end quote char, that means that the user sent an invalid command and we return an error.

Otherwise we advance the pointer and return normally.

Parsing commands

// command parses and returns a Command.
func (p *Parser) command() (Command, error) {
	b, err := p.r.ReadByte()
	if err != nil {
		return Command{}, err
	}
	if b == '*' {
		log.Println("resp array")
		return p.respArray()
	} else {
		line, err := p.readLine()
		if err != nil {
			return Command{}, err
		}
		p.pos = 0
		p.line = append([]byte{}, b)
		p.line = append(p.line, line...)
		return p.inline()
	}
}

We read the first character sent by the client. If it's an asterisk we handle it using the RESP protocol. Otherwise we assume that it's an inline command.

Let's start by parsing the inline commands first.

// Command implements the behavior of the commands.
type Command struct {
	args []string
	conn net.Conn
}

// inline parses an inline message and returns a Command. Returns an error when there's
// a problem reading from the connection or parsing the command.
func (p *Parser) inline() (Command, error) {
	// skip initial whitespace if any
	for p.current() == ' ' {
		p.advance()
	}
	cmd := Command{conn: p.conn}
	for !p.atEnd() {
		arg, err := p.consumeArg()
		if err != nil {
			return cmd, err
		}
		if arg != "" {
			cmd.args = append(cmd.args, arg)
		}
	}
	return cmd, nil
}

This is also quite easy to skim through. We skip any leading whitespace in case the user sent something like ' GET a'.

We create a new Command object with a reference to the session connection.

While we're not at the end of the line we consume args and append them to the arg list of the command object if they are not empty.

Consuming arguments

// consumeArg reads an argument from the current line.
func (p *Parser) consumeArg() (s string, err error) {
	for p.current() == ' ' {
		p.advance()
	}
	if p.current() == '"' {
		p.advance()
		buf, err := p.consumeString()
		return string(buf), err
	}
	for !p.atEnd() && p.current() != ' ' && p.current() != '\r' {
		s += string(p.current())
		p.advance()
	}
	return
}

Same as before we consume any leading whitespace.

If we find a quoted string we call our function from before: consumeString().

We append all characters to the output until we reach a carriage return \r, a whitespace or the end of the line.

Parsing RESP protocol

// respArray parses a RESP array and returns a Command. Returns an error when there's
// a problem reading from the connection.
func (p *Parser) respArray() (Command, error) {
	cmd := Command{}
	elementsStr, err := p.readLine()
	if err != nil {
		return cmd, err
	}
	elements, _ := strconv.Atoi(string(elementsStr))
	log.Println("Elements", elements)
	for i := 0; i < elements; i++ {
		tp, err := p.r.ReadByte()
		if err != nil {
			return cmd, err
		}
		switch tp {
		case ':':
			arg, err := p.readLine()
			if err != nil {
				return cmd, err
			}
			cmd.args = append(cmd.args, string(arg))
		case '$':
			arg, err := p.readLine()
			if err != nil {
				return cmd, err
			}
			length, _ := strconv.Atoi(string(arg))
			text := make([]byte, 0)
			for i := 0; len(text) <= length; i++ {
				line, err := p.readLine()
				if err != nil {
					return cmd, err
				}
				text = append(text, line...)
			}
			cmd.args = append(cmd.args, string(text[:length]))
		case '*':
			next, err := p.respArray()
			if err != nil {
				return cmd, err
			}
			cmd.args = append(cmd.args, next.args...)
		}
	}
	return cmd, nil
}

As we know, the leading asterisk has already been consumed from the connection input. So, at this point, the first line contains the number of elements to be consumed. We read that into an integer.

We create a for loop with that will parse all the elements in the array. We consume the first character to detect which kind of element we need to consume: int, string or array.

The int case is quite simple, we just read until the rest of the line.

The array case is also quite simple, we call respArray() and append the args of the result, to the current command object.

For strings we read the first line and get the size of the string. We keep reading lines until we have read the indicated amount of characters.

Handling commands

This is the 'fun' part of the implementation. Were our server becomes alive. In this section we'll implement the actual functionality of the commands.

Let's start with the cmd.handle() function that we saw in handleSession().

// handle Executes the command and writes the response. Returns false when the connection should be closed.
func (cmd Command) handle() bool {
	switch strings.ToUpper(cmd.args[0]) {
	case "GET":
		return cmd.get()
	case "SET":
		return cmd.set()
	case "DEL":
		return cmd.del()
	case "QUIT":
		return cmd.quit()
	default:
		log.Println("Command not supported", cmd.args[0])
		cmd.conn.Write([]uint8("-ERR unknown command '" + cmd.args[0] + "'\r\n"))
	}
	return true
}

Needs no further explanation. Let's implement the easiest command: QUIT.

// quit Used in interactive/inline mode, instructs the server to terminate the connection.
func (cmd *Command) quit() bool {
	if len(cmd.args) != 1 {
		cmd.conn.Write([]uint8("-ERR wrong number of arguments for '" + cmd.args[0] + "' command\r\n"))
		return true
	}
	log.Println("Handle QUIT")
	cmd.conn.Write([]uint8("+OK\r\n"))
	return false
}

If any extra arguments were passed to QUIT, it returns an error.

Otherwise write +OK to the client and return false. Which if you remember handleSession() is the value to indicate that the session has finished. After that the connection will be automatically closed.

The next easieast command is DEL

// del Deletes a key from the cache.
func (cmd *Command) del() bool {
	count := 0
	for _, k := range cmd.args[1:] {
		if _, ok := cache.LoadAndDelete(k); ok {
			count++
		}
	}
	cmd.conn.Write([]uint8(fmt.Sprintf(":%d\r\n", count)))
	return true
}

Iterates through all the keys passed, deletes the ones that exists and writes back to the client the amount of keys deleted.

Returns true, which means the connection is kept alive.

Handling GET

// get Fetches a key from the cache if exists.
func (cmd Command) get() bool {
	if len(cmd.args) != 2 {
		cmd.conn.Write([]uint8("-ERR wrong number of arguments for '" + cmd.args[0] + "' command\r\n"))
		return true
	}
	log.Println("Handle GET")
	val, _ := cache.Load(cmd.args[1])
	if val != nil {
		res, _ := val.(string)
		if strings.HasPrefix(res, "\"") {
			res, _ = strconv.Unquote(res)
		}
		log.Println("Response length", len(res))
		cmd.conn.Write([]uint8(fmt.Sprintf("$%d\r\n", len(res))))
		cmd.conn.Write(append([]uint8(res), []uint8("\r\n")...))
	} else {
		cmd.conn.Write([]uint8("$-1\r\n"))
	}
	return true
}

As before, we validate that the correct number of arguments were passed to the command.

We load the value from the global variable cache.

If the value is nil we write back to the client the special $-1.

When we have a value we cast it as string and unquote it in case it's quoted. Then we write the length as the first line of the response and the string as the second line of the response.

Handling SET

This is the most complicated command that we'll implement.

// set Stores a key and value on the cache. Optionally sets expiration on the key.
func (cmd Command) set() bool {
	if len(cmd.args) < 3 || len(cmd.args) > 6 {
		cmd.conn.Write([]uint8("-ERR wrong number of arguments for '" + cmd.args[0] + "' command\r\n"))
		return true
	}
	log.Println("Handle SET")
	log.Println("Value length", len(cmd.args[2]))
	if len(cmd.args) > 3 {
		pos := 3
		option := strings.ToUpper(cmd.args[pos])
		switch option {
		case "NX":
			log.Println("Handle NX")
			if _, ok := cache.Load(cmd.args[1]); ok {
				cmd.conn.Write([]uint8("$-1\r\n"))
				return true
			}
			pos++
		case "XX":
			log.Println("Handle XX")
			if _, ok := cache.Load(cmd.args[1]); !ok {
				cmd.conn.Write([]uint8("$-1\r\n"))
				return true
			}
			pos++
		}
		if len(cmd.args) > pos {
			if err := cmd.setExpiration(pos); err != nil {
				cmd.conn.Write([]uint8("-ERR " + err.Error() + "\r\n"))
				return true
			}
		}
	}
	cache.Store(cmd.args[1], cmd.args[2])
	cmd.conn.Write([]uint8("+OK\r\n"))
	return true
}

As always, first thing we do is validate the number of arguments. But in this case, SET is more tricky than the others.

When more than 3 arguments are passed we check for the NX or XX flags and handle them accordingly.

NX -- Only set the key if it does not already exist.
XX -- Only set the key if it already exist.

Then we parse the expiration flags if any. We'll see how that's done in a second.

After handling all those special cases we finally store the key and value in the cache, write the +OK response and return true to keep the connection alive.

Expiration

// setExpiration Handles expiration when passed as part of the 'set' command.
func (cmd Command) setExpiration(pos int) error {
	option := strings.ToUpper(cmd.args[pos])
	value, _ := strconv.Atoi(cmd.args[pos+1])
	var duration time.Duration
	switch option {
	case "EX":
		duration = time.Second * time.Duration(value)
	case "PX":
		duration = time.Millisecond * time.Duration(value)
	default:
		return fmt.Errorf("expiration option is not valid")
	}
	go func() {
		log.Printf("Handling '%s', sleeping for %v\n", option, duration)
		time.Sleep(duration)
		cache.Delete(cmd.args[1])
	}()
	return nil
}

We read the option and the expiration value, then we compute the duration for each case and we spawn a new goroutine that sleeps for that amount of time and the deletes the key from the cache.

This is not the most efficient way to do it, but it's simple and it works for us.

Working server

At this point we have an usable implementation of Redis.

Let's start the server the server and test it.

$ go run main.go2023/04/08 21:09:40 Listening on tcp://0.0.0.0:6380

On a different terminal connect to the server.

$ telnet 127.0.0.1 6380GET a$-1set a "test \"quotes\" are working"+OKget a$25test "quotes" are working

It's alive!! Go have fun.

If you'd like to access the source code of this project there's a public gist containing all of the code displayed here.

Link to source code

How to write a program that can replicate itself

2022-11-26T00:00:00Z

How to write a program that can replicate itself

Grotsky is a toy programming language that I made for fun. Today we're visinting the concept of Quines, a.k.a. self replicating programs. It's said that any turing-complete language should be able to write a program that replicates itself. And grotsky is no exception.

Read more about grotsky in previous blogposts:

Quines are very easy to write. The language that you're using needs to be able to do a couple things:

Write to a file or stdout (print)
Support for string arrays
Translate numbers/integers to character ascii representation
Concatenate strings
Loop through arrays from arbitrary indexes

Super simple quine: less than 30 lines of code

let tabChar = 9
let quoteChar = 34
let commaChar = 44
let code = [
	"let tabChar = 9",
	"let quoteChar = 34",
	"let commaChar = 44",
	"let code = [",
	"]",
	"for let i = 0; i < 4; i = i+1 {",
	"	io.println(code[i])",
	"}",
	"for let i = 0; i < code.length; i = i+1 {",
	"	io.println(strings.chr(tabChar) + strings.chr(quoteChar) + code[i] + strings.chr(quoteChar) + strings.chr(commaChar))",
	"}",
	"for let i = 4; i < code.length; i = i+1 {",
	"	io.println(code[i])",
	"}",
]
for let i = 0; i < 4; i = i+1 {
	io.println(code[i])
}
for let i = 0; i < code.length; i = i+1 {
	io.println(strings.chr(tabChar) + strings.chr(quoteChar) + code[i] + strings.chr(quoteChar) + strings.chr(commaChar))
}
for let i = 4; i < code.length; i = i+1 {
	io.println(code[i])
}

Now we can use grotksy cli to run the program and compare the output to the original source.

Save the original source to a file called quine.gr then run the following commands:

$ grotsky quine.gr > quine_copy.gr
$ cmp quine.gr quine_copy.gr
$ echo $?
0

If you see a 0 as the final output that means the files are the same. Otherwise if you saw an error message or a different output, that means something has gone wrong.

How exciting is this?!!We've just written a program that gives itself as an output. That sounds impossible when you hear it for the first time. But it was actually pretty easy!

Source code available here: https://gist.github.com/mliezun/c750ba701608850bd86d646a3ebf700f.

Grotsky cli binary available here:https://github.com/mliezun/grotsky/releases/tag/v0.0.6

Migrate from Heroku to Fly.io

2022-09-22T00:00:00Z

How to migrate from Heroku to Fly.io

A couple weeks ago Heroku announced the removal. I have plenty of projects running on free dynos. I have taken some time to move my code to Fly.io. And also I've written a little tutorial of how to perform the migration.

I'll use one of my public repos as an example https://github.com/mliezun/getmyip. It's a simple service that returns the IP from which you're making the request. It's useful when you want to know your public IP.

That project ^ was covered in a previous blogpost.

Migration instructions

The first thing we need to do is to remove heroku from the remotes. Inside your project run:

git remote remove heroku

If you have a heroku.yml file, delete it.

rm -rf heroku.yml

Then, we're ready to start using fly. There are tutorials on the official fly.io docs for many frameworks and languages. We're going to be following the one for a Docker app, since it's the most general case.

First thing you need to do is create an account in Fly.io if you don't have one yet.

Once you created your account, install the flyctl command line tool. After that, login by running the following command:

flyctl auth login

After you've logged in to your account, you're ready to launch your application. Execute the next command and follow the interactive setup.

$ flyctl launch
Scanning source code
Detected a Dockerfile app
? App Name (leave blank to use an auto-generated name): 
Automatically selected personal organization: Miguel Liezun
? Select region: mia (Miami, Florida (US))
Created app morning-breeze-4255 in organization personal
Wrote config file fly.toml
? Would you like to set up a Postgresql database now? No
? Would you like to deploy now? Yes
Deploying morning-breeze-4255
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 -> 8080
Remote builder fly-builder-green-pond-8004 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
...

Make sure your app listens to port 8080, that's the default for fly apps. You can change the port inside the file fly.toml if you want, just search for the internal port and change it. Remember to run launch again if you change the port.

# fly.toml file generated for morning-breeze-4255 on 2022-09-21T21:50:20-03:00

app = "morning-breeze-4255"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080 # <- Put your desired port here
# ...

Finally, you only need to open the app and enjoy!You migrated your first app from heroku to fly :-)

$ flyctl open
opening http://morning-breeze-4255.fly.dev ...

Access the newly deployed 'getmyip' service using the link http://morning-breeze-4255.fly.dev.

Branchable MySQL: Managing multiple dev environments

2022-09-20T00:00:00Z

Branchable MySQL: Managing multiple dev environments

When teams start to grow, having a single dev environment becomes an issue. People start stepping on each others toes. A common problem is that two people want to apply incompatible migrations on the database. That problem is impossible to fix if folks are working on parallel branches. If we can have a database for each branch of a project, that will remove much of the pain of having multiple devs applying changes to the db.

There are already projects that solve this problem: PlanetScale and Neon.

A common case where this problem arises is when two devs want to add a column to the same table.

We have a people table in the database. One of the devs wants to add the last_name column and the other one wants to add the address.

Dev1's code thinks the table will have 3 columns after he applies his operation: id, name, last_name.

Dev2's code also thinks the table will have 3 columns: id, name, address.

In reality the table will have 4 columns.So neither of them will be able to run their code unless they talk to each other and figure out how to make this work.

This is far from ideal.

What we want instead, is that each one of them can develop their features independently.

They both apply to the same table, but each table lives on an instance that was 'replicated' from the original.

How can we implement the ideal case?

MySQL writes data (by default) to the directory /var/lib/mysql/data.

We can use an Union filesystem. And configure MySQL to use a different directory to read and write data.

That way we can have a feature/user-last-name 'branch' read and write data from a directory like /app/user-last-name/mysql/data.

And a feature/user-address 'branch' read and write data from a directory like /app/user-address/mysql/data.

Those branches can be mounted using fuse-overlayfs by executing the following commands:

# Directory /app/base contains data from the original branch

fuse-overlayfs -o lowerdir=/app/base,upperdir=/app/user-last-name,workdir=/tmp/user-last-name overlayfs /app/user-last-name

fuse-overlayfs -o lowerdir=/app/base,upperdir=/app/user-address,workdir=/tmp/user-address overlayfs /app/user-address

This means both 'branches' of the database are able to coexist and have different schemas during their lifetime.

Experimenting with a use case

I had this idea in my head for months. I finally convinced myself that it was worth a shot.

I decided to do a little implementation using Docker and python FastAPI.Exposing a simple interface so that it's easy to create and delete branches.

The project is live on github branchable-mysql.

The container image is published on Docker Hub branchable-mysql.

To start using the image let's create a docker-compose.yml file.

version: "3"

services:
  mysql:
    image: mliezun/branchable-mysql
    platform: linux/amd64
    privileged: true
    restart: always
    volumes:
      - appdata:/app/

volumes:
  appdata:

Then you can execute docker-compose up and the MySQL server should start running.

After that, connect easily to the db docker compose exec mysql mysql -uroot -h127.0.0.1 --skip-password -P33061. You should enter to an interactive mysql console.

Let's create an initial schema, a table and insert some data so that we can see how branching works.On the console that we just opened execute:

mysql> create schema s1;
Query OK, 1 row affected (0.01 sec)

mysql> use s1;
Database changed
mysql> create table people (id int primary key auto_increment, name varchar(255) not null);
Query OK, 0 rows affected (0.07 sec)

mysql> insert into people select 0, 'Miguel';
Query OK, 1 row affected (0.02 sec)
Records: 1  Duplicates: 0  Warnings: 0

mysql> select * from people;
+----+--------+
| id | name   |
+----+--------+
|  1 | Miguel |
+----+--------+
1 row in set (0.00 sec)

That's enough for now, we're ready to start creating branches.

On a separate terminal, without closing the previous mysql interactive console, execute:

docker compose exec mysql /app/scripts/create_branch.sh base feature/user-last-name

{"branch_name":"feature/user-last-name","base_branch":"base","port":33062}

Now you can login to the new database branch using port 33062 docker compose exec mysql mysql -uroot -h127.0.0.1 --skip-password -P33062

mysql> use s1;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> alter table people add column last_name varchar(255) not null;
Query OK, 0 rows affected (0.03 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> select * from people;
+----+--------+-----------+
| id | name   | last_name |
+----+--------+-----------+
|  1 | Miguel |           |
+----+--------+-----------+
1 row in set (0.00 sec)

In a new terminal we can create a another branch:

docker compose exec mysql /app/scripts/create_branch.sh base feature/user-address

{"branch_name":"feature/user-address","base_branch":"base","port":33063}

Then connect using port 33063 docker compose exec mysql mysql -uroot -h127.0.0.1 --skip-password -P33063

mysql> use s1;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> alter table people add column last_name varchar(255) not null;
Query OK, 0 rows affected (0.03 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> select * from people;
+----+--------+
| id | name   |
+----+--------+
|  1 | Miguel |
+----+--------+
1 row in set (0.00 sec)

mysql> alter table people add column address varchar(255) not null;
Query OK, 0 rows affected (0.02 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> select * from people;
+----+--------+---------+
| id | name   | address |
+----+--------+---------+
|  1 | Miguel |         |
+----+--------+---------+
1 row in set (0.00 sec)

As you can see, we have 3 servers running at the same time, each one with different schemas.

This is great for local development and for having branch-aware dev environments.

Final thoughts

I hope you find this blogpost useful.If you want to start using branchable-mysql go ahead.If you encounter any issues please report them in the github repo or create a pull request.

Webscraping as a side project

2022-07-20T00:00:00Z

Webscraping as a side project

A friend and I were looking for a side project to work together. We realized we both faced a similar problem.

Let's use AWS Lambda Python runtime as an example.AWS will send out emails when a version is at the end of life making it difficult to stay on the latest if desired.Plus, reacting to them usually means you are many many versions behind already.

Our journey started.We made a list of providers for the MVP: AWS, GCP and Azure.Then a list of the services that have versions (for example S3 doesn't have versions).After that we realized that we could get some versions using APIs.Other services exclusively require webscraping.

We support 46 services and counting.Take a look atCloud Outdated and subscribe to get notified.If you are looking for a service that's not therecontact us.

Picking a language, framework and platform

We're both Python programmers. The choice was obvious."Let's use Python and Django framework for the job" we said.We didn't want to spend our innovation tokens on new language/framework.So we chose Boring Technology.

For the db we spent our first innovation token.We decided to go with the flashy new serverless postgres-compatibleCockroachDB.

On the hosting side we're using AWS Lambda. Taking advantage of the free compute time.Helps mantaining the costs down.

Make webscraping reliable

A webpage that's being scraped can change at any time. First thing we did was account for those edge cases.We created a custom exception that is triggered when something changed. So that we can react to that downstream.

class ScrapingError(Exception):
    pass

We wanted to keep the implementation simple. Each service is scraped by a single function.The signature of the function is something likeaws_lambda_python() -> List[Version].All the implementations follow a similar pattern:

def aws_lambda_python():
    # Read versions from aws docs website:
    # https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

    if not found_versions:
        raise ScrappingError

    # Process and return versions

That's ^ what we call a poll function.

We pass poll functions through a polling class that handles all the errors and results.When we detect an scraping error we have a special case. We send an email with the detailsof what happened. Because the problem is something that requires manual action. We receivethat email in our personal inboxes and fix the problem ASAP.

The poll class that handles all the magic behind cloud outdated is actually very simple:

class PollService:
    def __init__(self, service: Service, poll_fn: Callable):
        self.poll_fn = poll_fn
        # Some other attributes...

    def poll(self):
        try:
            results = self.poll_fn()
            self.process_results(results)
        except ScrapingError as e:
            notify_operator(
                f"{type(e).__name__} at line {e.__traceback__.tb_lineno} of {__file__}: {e.__str__()}"
            )

    def process_results(self, results):
        # if results contains new versions:
        #     save new versions to db
        # if results contains deprecated versions:
        #     set versions in db as depreacted

That's the hearth of Cloud Outdated.After that we have to send notifications to subscribed users.That part is trivial.We send an email that contains the difference between what was last sent to a user and what wehave stored in the db at the moment.

Last toughts

Having a side project is usually a good idea.For us has been a journey were we got to know some new stuff (CockroachDB).We also learned about how to build a product and keep a MVP mentality.The most difficult challenge we face is to bring more users to the platform.

We'd love to see more people subscribed.If this blogpost sparked your interest go toCloud Outdated and subscribe to start getting emails.

See you next time!

Blake3 hash plugin for MySQL written in Rust

2022-05-05T00:00:00Z

Implementing a blake3 hash plugin for MySQL in Rust

It's been long since I've written something, I wanted to bring you some new stuff, so here you have a short blog post.I encourage you to try this new plugin that uses thisblake3 hash implementation.Blake3 is secure, unlike MD5 and SHA-1. And secure against length extension, unlike SHA-2.Start using it and create an issue in the github repo if you would like a feature implemented!

Checkoutblake-udf source code.

How to use

Download and install MySQL plugin

$ wget 'https://github.com/mliezun/blake-udf/releases/download/v0.1.0/libblake_udf.so'$ mv libblake_udf.so /usr/lib/mysql/plugin/

Load UDF in MySQL

$ mysql -uroot -p -e 'create function blake3_hash returns string soname "libblake_udf.so";'

Execute function

$ mysql --binary-as-hex=0 -uroot -p -e 'select blake3_hash("a");'

Output: 17762fddd969a453925d65717ac3eea21320b66b54342fde15128d6caf21215f

Playing with Javascript Proxies (getters/setters)

2021-12-31T00:00:00Z

Playing with Javascript Proxies (getters/setters)

Overview

Happy New Year!

This is my final post for the 2021. This year I didn't post that much, but a lot of work was put into the blog to rewrite it using Grotksy.I hope everyone has a great 2022 and that next year is much better than the last one.

The inspiration for this blog post comes from the idea of building a tiny db that feels more natural to Javscript.All the databases that I've seen make a heavy use of methods like: db.get(), db.put(), db.scan(), db.query(). And many others that Im sure you have seen.I think it would be great to see something like:

const db = getDb("...")// Create new userconst u = {username: "jdoe", email: "jdoe@example.com", id: 100}// Store new user in the databasedb.objects.users[u.username] = u// Commit the changes to the databasedb.actions.save()

In this blog post we will be building a much simpler version that stores everything in memory. Each change made to the objects will be stored in a log (called layers) and the final object will be composed of all the small layers present in the log.

Defining a proxy

We need to implement some generic getters/setters.

const objects = new Proxy({}, {
    get: function(obj, prop) {
        validate(prop, null)
        // Implementation
    },
    set: function(obj, prop, value) {
        validate(prop, value)
        // Implementation
    }
})

Let's define the validation function. In this case we want the objects to be able to be serialized to JSON.

const validate = (prop, value) => {
    // Make sure that the property and value are serializable
    // JSON.stringify throws an error if not serializable
    const l = {}
    l[prop] = value
    JSON.stringify(l)
    return l
}

This empty proxy will validate that the values and prop are serializable and do nothing else. Now we can start building on top of it.

Building a tree to hold everything together

We need a root object where we will store all the changes that are applied to an object.We will have a sort of tree structure to hold everything together.It will look something like this:

              rootObject({})  -> layers([{users: {jdoe: ...}}, {tokens: {tk1: ...}}])
                    |
        --------------------------
        |                        |
 child(.users{})          child(.tokens{})
        |                        |
       ...                      ...

The root object contains the layers with all the changes made from the beginning of the existence of the object.Each time a property of the root object is accessed a child is returned that internally holds a reference to the root.This way we can go through the entire chain of access and be able to reach the stored layers.By chain of access I mean the following: objects.users.jdoe.metadata.login.ip.As you can see, we need to traverse through many objects to be able to reach the ip field. But the layer that contains the information is only stored in the root, so each child needs to mantain a reference to the parent to be able to reach the root node.

Let's define a simple function to be able to create a new rootObject.

const wrapObject = (parent, key, current) => {
    const rootObj = {
        parent: Object.freeze(parent),
        layers: [Object.freeze({'value': current, 'previous': null})],
        pushLayer (l) {}, // Push new layer
        getLayer (ks) {}, // Get layer where information is stored based on given keys
        getValue (k) {} // Get value that matches given key
    }

    const rootProxy = {
        get: function(obj, prop) {
            validate(prop, null)
            const val = rootObj.getValue(prop)
            if (typeof val == 'object') {
                // If the value is an object we need to have a child instance
                // with a reference to the parent
                return wrapObject(rootObj, prop, val).objects
            }
            // If the value is other kind like a number or string we can safely return that
            return val
        },
        set: function(obj, prop, value) {
            const l = validate(prop, value)
            // Add new layer to the rootObj
            rootObj.pushLayer({'value': l})
        }
    }

    return {
        actions: {
            revert () {
                // Deleting the last layer will revert the changes
                const pop = rootObj.layers[rootObj.layers.length-1]
                rootObj.layers.splice(rootObj.layers.length-1, rootObj.layers.length)
                return pop
            }
        },
        objects: new Proxy({}, rootProxy)
    }
}

Handling layers

The layer format:

const layer = {
    value: {status: 'active'},
    previous: null // Reference to a previous layer that has the key 'status' in it
}

The layers are stored in an array, each layer holds the value and a reference to the previous layer that set a value for the same key (in this case the key was 'status'). Also the layers form a simple linked list through the 'previous' reference. That way we have the entire history of a given key.

We would need a function to be able to tell if an object has a list of nested keys. Trust me for now, you'll see.

const nestedExists = (obj, ks) => {
    for (let j = 0; j < ks.length; j++) {
        let k = ks[j];
        if (!(k in obj)) {
            return false
        }
        obj = obj[k]
    }
    return true
}

In this function we receive an object and a list of keys, we start accessing the first internal object with the first key and we keep doing the same till we make sure that all the keys are present.

Now we're almost done. Let's define the functions for handling the store and retrieval of layers.

const rootObj = {
    parent: Object.freeze(parent),
    layers: [Object.freeze({'value': current, 'previous': null})],
    pushLayer (l) {
        // If this is a child object we need to build the entire chain of access
        // from the bottom up
        if (parent) {
            const ll = {}
            ll[key] = l['value']
            // Search for a previous layer modifying the same key
            const previous = parent.getLayer([key])
            // Tell the parent object to push the new layer
            parent.pushLayer(Object.freeze({'value': ll, previous}))
        } else {
            // We are in the root object, add the layer to the array
            this.layers.push(Object.freeze(l))
        }
    },
    getLayer (ks) {
        // Search through the entire list of layers to see if one contains all the keys
        // that we are looking for. Start from the end of the array (top of the stack)
        for (let i = this.layers.length - 1; i >= 0; i--) {
            let v = nestedExists(this.layers[i]['value'], ks)
            if (v) {
                return this.layers[i]
            }
        }
        if (parent) {
            // If we are in a child object, look through all the previous layers
            // and see if the key we're looking for is contained in one of them.
            let ll = parent.getLayer([key].concat(ks))
            while (ll) {
                let a = nestedExists(ll['value'][key], ks)
                if (a) {
                    return Object.freeze({'value': ll['value'][key]})
                }
                ll = ll.previous
            }
        }
    },
    getValue (k) {
        // Straightforward, get layer and return value
        const l = this.getLayer([k])
        if (l) {
            return Object.freeze(l['value'][k])
        }
    }
}

That's all we need. We can create a new object and start adding and modifying properties. Each change will be added to the end of the log and worked out when a property is accessed.

Wrapping Up

Let's try the final result. The source code is loaded in this page, so you can open a dev console in the browser and try for yourself.

const store = wrapObject(null, null, {})

// Create new user
const user = {username: 'jdoe', email: 'jdoe@example.com', name: 'John Doe', id: 100}

// Add new user
store.objects.users = {}
store.objects.users[user.username] = user

// Print user email
console.log(store.objects.users.jdoe.email)

// Change user email and print
store.objects.users.jdoe.email = 'jdoe2@example.com'
console.log(store.objects.users.jdoe.email)

// Revert last change and print email again
store.actions.revert()
console.log(store.objects.users.jdoe.email)

That's it for now. We defined a Javascript object that contains the entire history of changes that were made to itself.And at any point we can revert the changes and go back to a previous state.Everything is stored in an array and is easily serializable.If we wanted to take this to the next level, each change could be written to a persistence storage (s3, sqlite, mysql, ...)

The full source code is available in apublic gist.

I created a programming language and this blog is powered by it

2021-10-04T00:00:00Z

I created a programming language and this blog is powered by it

Why did I do it?

Mostly for fun.

If you follow my blog or take a look at some of the posts that I made, you will see that I wasbuilding a programming language called Grotksy. Just a toy programming language that I made based on the book Crafting Interpreters, which I totally recommend buying and reading if you haven't yet.

I wanted to build something interesting but simple enough that could be made with Grotsky.I tought that replacing Jekyll with my own engine was a task worth a try.There is nothing groundbreaking or innovative being made here, just a little experimentation.

I have to give credit to the project lith, because the 'templating' engine for the blog is inspired by it.

How did I do it?

That's a good question.

Originally, this blog was powered by Jekyll, that translated markdown to html and hosted by Github Pages. I decided that I was going to build a templating engine and generate html to keep things simple.

But also, as a challenge I made a simple HTTP server to use as a dev server when trying the blog locally.

HTTP Server

For the purpose of having a custom HTTP Server I had to add support for TCP sockets to the language.I wrapped the go standard library in some functions and exposed that to the Grotsky language.In grotsky looks something like this

let socket = net.listenTcp(host + ":" + port)
let conn
while true {
	try {
		conn = socket.accept()
	} catch err {
		io.println("Error accepting new connection", err)
		continue
	}
	try {
		# Call function that handles connection
		handleConn(conn)
	} catch err {
		io.println("Error handling connection", err)
	}
	try {
		conn.close()
	} catch err {}
}

This means that the server listens on a socket, accepts connections, writes some text/bytes to the connection and then closes the connection.

Template Engine

The templating engine is built using the native support Grotsky provide for lists.A regular page for the blog looks like this:

let base = import("../base.gr")
# Create new Post Object
let post = base.Post(
	"Title",
	"Brief description of blog post.",
	"Author Name",
	"descriptive,tags",
	[
		[
			"h2",
			[],
			[
				"Title"
			]
		],
		[
			"div",
			["class", "content"],
			[
				"Content line 1",
				"Content line 2",
			]
		]
	]
)

It's pretty straightforward: the first element of the list is the html tag, the second isan array of properties for the tag and the last one is a list that contains what will bethe *content* of enclosed by the tags.

Resources

If you want to take a peek, the source code for both projects is available on github:

Mlisp: My own lisp implementation compiled to WASM

2021-04-01T00:00:00Z

Mlisp, My own lisp implementation

Mlisp a tiny lispy language based on the book Build Your Own Lisp.The interpreter is written in C and compiled directly to WASM. You can try it in this page by openning the developer console of your browser and typing Mlisp.interpret("+ 2 2") or using the repl shown below.

Interface

To be able to access C functions from your browser you have to export them. Let's see how we can define a function that is exported.

#if __EMSCRIPTEN__EMSCRIPTEN_KEEPALIVE#endifint mlisp_init();

When compilen with emcc the emscripten compiler to wasm, you have to add EMSCRIPTEN_KEEPALIVE macro before your function so it doesn't get optimized away.The exported functions in this project are:

int mlisp_init();char *mlisp_interpret(char *input);void mlisp_cleanup();

The project is then compiled with:

emcc -std=c99  -Wall -O3 -s WASM=1 -s EXTRA_EXPORTED_RUNTIME_METHODS='["cwrap"]'

That means that you would be able to access the exported functions using a cwrap that let's you wrap a C function call from a Javascript function call.This compilation generates two files mlisp.js and mlisp.wasm.The javascript file defines a Module that provides useful tool to access exported functions.

Let's start using it

const Mlisp = {    init: Module.cwrap('mlisp_init', 'number', []),    interpret: Module.cwrap('mlisp_interpret', 'string', ['string']),    cleanup: Module.cwrap('mlisp_cleanup', 'void', []),};// Init interpreterMlisp.init();// Run some commandsconsole.log(Mlisp.interpret("+ 2 2"));// Cleanup interpreterMlisp.cleanup();

Automated Build & Release from github

I made a github workflow for this project to automatically build and release so you can retrieve them from Github.

REPL

Interesting commands to try out

foldl: Fold left (same as reduce left) - (foldl + 0 {1 2 3 4 5}): Sum of elements

filter - (filter (\ {e} {> e 3}) {1 2 3 4 5 6}): Elements bigger than 3

map - (foldl * 1 (map (\ {e} {* e 2}) {1 1 1 1 1})): Multiply elements by 2 and then multiply all elements

Grotsky Part 4: Writing a service to get your public IP

2020-12-17T00:00:00Z

Writing a service to get your public IP

Grotsky (my toy programming language) finally can be used to make something useful.In this post I want to show you how I made a service that let's your retrieve your public IP as a response to a HTTP Request.

Show me the code

Let's start by building the http request handler.The service will be deployed to heroku. Heroku passes the port that the http server has to listen as an environment variable named PORT.

Let's get the server up and running

let listen = ":8092"let port = env.get("PORT")if port != "" {    listen = ":" + port}io.println("Listen " + listen)http.listen(listen)

We listen by default at the port 8092 and if the environment variable is given we change it.Then we print what is the port and start the server with http.listen. That blocks the execution and starts the server.Grotsky interpreter is written in Go, and uses Go's standard http server. Each requests is handled by a goroutine, but because Grotsky is single threaded only one goroutine executes at any given point in time. When a request is received the goroutine has to hold the GIL (Global Interrupt Lock) to be able to give control to the interpreter.

Now lets add some code to handle requests

fn getIP(rq, rs) {    io.println("Request from --> " + rq.address)    rs.write(200, rq.address)}http.handler("/", getIP)let listen = ":8092"let port = env.get("PORT")if port != "" {    listen = ":" + port}io.println("Listen " + listen)http.listen(listen)

Now we have something interesting to try out!What we've done is to log and write back as response the address of the device that is doing the request.To try it out you need to download grotsky.

$ go get github.com/mliezun/grotsky/cmd/grotsky

Save the Grotsky code under a filed called getip.g and the execute it using the grotsky interpreter:

$ go run $(go env GOPATH)/src/github.com/mliezun/grotsky/cmd/grotsky getip.g

Output:

Listen :8092

Now you can make a request to see if it is working

$ curl localhost:8092

Output:

[::1]:43464

We see that the address contains the port we want to split it and show just the IP.

Let's write a couple functions to do that

fn findReversed(string, char) {    for let i = string.length-1; i > -1; i = i - 1 {        if string[i] == char {            return i        }    }    return -1}fn parseIP(address) {    let ix = findReversed(address, ":")    return address[:ix]}

The function findReversed finds the first index where char appears in string starting from the end.The function parseIP uses findReversed to obtain the index where ":" splits the IP and the PORT and uses that index to return just the IP address.

Now we can send just the IP address

fn getIP(rq, rs) {    let address = parseIP(rq.address)    io.println("Request from --> " + address)    rs.write(200, address)}

Add the two functions at the beginning of the file and modify the getIP function.Restart the server and now if you make a request you should get just the IP.

$ curl localhost:8092[::1]

Voila!

We have just one last issue: Proxies!

Our service will probably sit behind a proxy, so we need to read the address from a special header X-Forwarded-For.Let's implement that!

fn getIP(rq, rs) {    let address = parseIP(rq.address)    let forwarded = rq.headers["X-Forwarded-For"]    if forwarded != nil {        address = forwarded[0]    }    io.println("Request from --> " + address)    rs.write(200, address)}

We read the header from the request and if X-Forwarded-For is present we sent that as a response to the user.

Our work is complete. Let's try it!

$ curl localhost:8092 -H 'X-Forwarded-For: 8.8.8.8'8.8.8.8$ curl localhost:8092[::1]

Well done. Now you can deploy it to Heroku (that's up to you) or any other cloud platform.I have my own version running under: https://peaceful-lowlands-45821.herokuapp.com/

Deployed to Fly.io after Heroku killed the free plan: http://morning-breeze-4255.fly.dev

Try it from your command line

$ curl http://morning-breeze-4255.fly.dev

Executing python code from MySQL Server

2020-04-19T00:00:00Z

Executing python code from MySQL Server

Trying py_eval

Generate a list of integers from 0 to 10

> select py_eval('[i for i in range(10)]') list;+--------------------------------+| list                           |+--------------------------------+| [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] |+--------------------------------+

Generate a dictionary (json object) from a list of dicts

> select replace(py_eval('{ str(user["id"]) : user for user in [{"id": 33, "name": "John"}, {"id": 44, "name": "George"}] }'), "'", '"') dict;+------------------------------------------------------------------------+| dict                                                                   |+------------------------------------------------------------------------+| {"33": {"id": 33, "name": "John"}, "44": {"id": 44, "name": "George"}} |+------------------------------------------------------------------------+

Replace is needed here, because python uses single quotes for dictionaries.

Make a function that receives a json array and a key and sorts the array by key

DROP function IF EXISTS sort_by_key;DELIMITER $$CREATE FUNCTION sort_by_key (arr json, k text)RETURNS jsonBEGIN    RETURN replace(py_eval(CONCAT("sorted(", arr, ", key=lambda e: e['", k, "'])")), "'", '"');END$$DELIMITER ;

Test

> select sort_by_key('[{"a":2}, {"a":1}, {"a": 722}, {"a": 0}]', 'a') sorted;+--------------------------------------------+| sorted                                     |+--------------------------------------------+| [{"a": 0}, {"a": 1}, {"a": 2}, {"a": 722}] |+--------------------------------------------+

How to write a MySQL UDF

There is a pretty good guide at the MySQL 8.0 Reference Manual. I'll give you a brief explanation so you can start quickly, but reading the full guide is highly recomended.MySQL's UDFs are written in C++ and need to follow certain conventions so they can be recognized as such.In our case, we want our MySQL function to be called py_eval, so we have to define the following C++ functions:

py_eval_init or py_eval_deinit

py_eval

**py_eval_init**: (Optional) Initializes memory and data structures for the function execution.**py_eval**: Executes the actual function, in our case evaluates a python expression.**py_eval_deinit**: (Optional) If any memory was allocated in the init function, this is the place where we free it.For py_eval we only need **py_eval_init** and **py_eval**.

Functions signatures

bool py_eval_init(UDF_INIT *initid, UDF_ARGS *args,                             char *message);char *py_eval(UDF_INIT *, UDF_ARGS *args, char *result,                         unsigned long *res_length, unsigned char *null_value,                         unsigned char *);

These are the standard definitions for MySQL functions that return string, as is the case of py_eval. To be able to declare this functions, you need to have the definition of UDF_INIT and UDF_ARGS, you can find that at the source code of mysql server -> right here.

Evaluating python expression

For evaluating python expression, we'll be using pybind11. That gives us the ability to directly access the python interpreter and execute code.

Example

Make sure you have g++ installed. Try executing: g++ --help. And some version of python running of your system, for this tutorial I'll be using version _3.8_.

$ mkdir py_eval && cd py_eval$ git clone https://github.com/pybind/pybind11

Create a new file called main.cpp with the following content:

#include "pybind11/include/pybind11/embed.h"#include "pybind11/include/pybind11/eval.h"#include namespace py = pybind11;py::scoped_interpreter guard{}; // We need this to keep the interpreter aliveint main(void) {    auto obj = py::eval("[i for i in range(10)]");    std::cout << std::string(py::str(obj)) << std::endl;}

To run the example we have to compile the file.First, we need the compilation flags.

$ pkg-config python-3.8 --libs --cflags-I/usr/include/python3.8

Then, we can compile and run our code with the following.

$ g++ main.cpp -I/usr/include/python3.8 -lpython3.8$ ./a.out[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Putting all together

Download udf types to the project folder.

$ wget https://raw.githubusercontent.com/mysql/mysql-server/8.0/include/mysql/udf_registration_types.h

Create a new file called py_eval.cpp, with the following content:

c++#include "pybind11/include/pybind11/embed.h"#include "pybind11/include/pybind11/eval.h"#include "udf_registration_types.h"#include namespace py = pybind11;py::scoped_interpreter guard{}; // We need this to keep the interpreter aliveextern "C" bool py_eval_init(UDF_INIT *initid, UDF_ARGS *args,                             char *message){    // Here we can check if we received one argument    if (args->arg_count != 1)    {        // The function returns true if there is an error,        // the error message is copied to the message arg.        strcpy(message, "py_eval must have one argument");        return true;    }    // Cast the passed argument to string    args->arg_type[0] = STRING_RESULT;    initid->maybe_null = true; /* The result may be null */    return false;}extern "C" char *py_eval(UDF_INIT *, UDF_ARGS *args, char *result,                         unsigned long *res_length, unsigned char *null_value,                         unsigned char *){    // Evaluate the argument as a python expression    auto obj = py::eval(args->args[0]);    // Cast the result to std::string    std::string res_str = std::string(py::str(obj));    // Copy the output string from py::eval to the result argument    strcpy(result, res_str.c_str());    // Set the length of the result string    *res_length = res_str.length();    return result;}

Then, we have to compile the project as a shared library, and move it to the plugin folder of mysql (in your case, it could be located in some other directory).

$ g++ -I/usr/include/python3.8 -lpython3.8 -shared -fPIC -o py_eval.so py_eval.cpp$ sudo cp py_eval.so /usr/lib/mysql/plugin/

Now, it's time to try it from mysql.First, connect to your server as root.

$ sudo mysql -uroot

Create and test the function.

> create function py_eval returns string soname 'py_eval.so';Query OK, 0 rows affected (0.029 sec)> select py_eval('[i for i in range(10)]') list;+--------------------------------+| list                           |+--------------------------------+| [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] |+--------------------------------+1 row in set (0.001 sec)

Future

There is a lot to do, for example there is no error control on the function execution. The python expression that we are trying to evaluate could fail causing a server reboot. Also, there is some extra work to do to be able to use import. And there are many concerns regarding concurrency issues.If you want to contribute to improve execution of python code on mysql server, please go to my github project and make a PR.I hope you enjoyed this tutorial and come back soon for more.

Writing your own C malloc and free

2020-04-11T00:00:00Z

Writing your own C malloc and free

Challenge

This challenge comes from the book Crafting Interpreters by Bob Nystrom. And can be found in Chapter 14 - Challenge 3.The challenge goes:> You are allowed to call malloc() once, at the beginning of the interpreters execution, to allocate a single big block of memory which your reallocate() function has access to. It parcels out blobs of memory from that single region, your own personal heap. Its your job to define how it does that.

Check out this comparison of malloc() to S3.

Solution

As stated in the challenge I'll be using a big chunk of _contiguous_ memory. The main idea of my solution is to store the blocks of memory in the array prepending a header with metadata.

 _______________________________________________|head_0|block_0 ... |head_1|block_1    ...      |

The structure of the header is pretty similar to that of a linked list.

struct block_meta{    size_t size;    struct block_meta *next;    int free;};#define META_SIZE sizeof(struct block_meta)

It stores the size of the block, a pointer to the next block and a flag to mark wether it's free or not.Then, a function to traverse the list of blocks and find if there is any freed block is needed:

void *first_block = NULL;struct block_meta *find_free_block(struct block_meta **last, size_t size){    struct block_meta *current = first_block;    while (current && !(current->free && current->size >= size))    {        *last = current;        current = current->next;    }    return current;}

This function receives a double pointer to a block_meta struct called last that at the end of the execution should be pointing to the last node of the list and a size_t variable that indicates the minimum size that the block needs to be.

Memory initialization

Two functions are needed to handle the big chunk of memory, one to initialize and the other to free it.

void initMemory();void freeMemory();

To implement initMemory I've decided that I would ask for the maximum amount of memory that I could get from the OS.

#define MINREQ 0x20000// Big block of memoryvoid *memory = NULL;// Position where the last block endssize_t endpos = 0;void initMemory(){    size_t required = PTRDIFF_MAX;    while (memory == NULL)    {        memory = malloc(required);        if (required < MINREQ)        {            if (memory)            {                free(memory);            }            printf("Cannot allocate enough memory\n");            exit(ENOMEM);        }        required >>= 1;    }}void freeMemory(){    free(memory);}

As you can see, initMemory starts trying to allocate the maximum amount a memory allowed, and starts to divide that amount by 2 every time the allocation fails. If there isn't at least 128KB of memory available the program crashes with ENOMEM.Now that we have our chunk of memory ready to go, we can start to start giving blocks away.

struct block_meta *request_block(size_t size){    struct block_meta *last = NULL;    struct block_meta *block = find_free_block(&last, size);    if (block)    {        block->free = 0;        return block;    }    // Append new block to list    block = memory + endpos;    endpos += META_SIZE + size;    if (last)    {        last->next = block;    }    else    {        first_block = block;    }    block->free = 0;    block->next = NULL;    block->size = size;    return block;}

How request_block works:1. Tries to find a free block with enough space. If there is one, it is set as occupied and returns that block.2. If there isn't a free block available. It adds a new block with enough space at the end of memory (the big chunk).3. If this is the first call, points the head of the list to the recently created block, else point the last node to the block.4. Set the new block as occupied, set the size and next to null. Then return the new block.With this function, implementing malloc and free is pretty easy:

void *my_malloc(size_t size){    struct block_meta *block = request_block(size);    return block + 1;}void my_free(void *ptr){    struct block_meta *block = ptr - META_SIZE;    block->free = 1;}

To finish the challenge, I have to implement realloc, that is a little bit more tricky.

void *my_realloc(void *ptr, size_t size){    if (!ptr)    {        return my_malloc(size);    }    struct block_meta *block = ptr - META_SIZE;    if (block->size >= size)    {        return block + 1;    }    uint8_t *newptr = my_malloc(size);    size_t i;    for (i = 0; i < (block->size < size ? block->size : size); i++)    {        newptr[i] = ((uint8_t *)ptr)[i];    }    block->free = 1;    return newptr;}

How realloc works:

If the pointer to reallocate is null, works just like malloc.

If the given size is bigger than the prior size, it allocates a bigger block and copies all data from the original block to the new block.

If the given size is smaller than the prior size, it allocates a smaller block and copies just the data that fits into the smaller block.

New challenge

In my implementation I used a linked list where each node holds a pointer to the next, but given that I have control over the _entire_ memory this actualy isn't necessary.My challenge to you is that you remove the pointer to next from the block_meta struct.

Resources

https://danluu.com/malloc-tutorial/

http://www.craftinginterpreters.com/chunks-of-bytecode.html

Grotsky Part 3: Interpreting

2020-04-01T00:00:00Z

Grotsky Part 3: Interpreting

It's slow!

My interpreter it's really, really, wait for it... _Really slow_.An example of a bad performing grotsky code:

# fib: calculates the n-th fibonacci number recursivelyfn fib(n) begin    if n < 2 return n    return fib(n-2) + fib(n-1)endprintln(fib(30))

Running the code

$ time ./grotsky examples/fib.g

Gives a wooping result of:

832040real    0m11,154suser    0m11,806ssys     0m0,272s

Almost twelve seconds!!! Comparing with a similar python code

def fib(n):    if n < 2: return n    return fib(n-2) + fib(n-1)print(fib(30))

Gives a result of:

832040real    0m0,423suser    0m0,387ssys     0m0,021s

That means, my interpreter is at least 20 times slower than Cpython.

Why is it so slow?

Here is an explanation.As the person from the first comment states, go garbage collector is not well suited for this kind of scenario with heavy allocation of objects.> Go's GC is not generational, so allocation requires (comparatively speaking) much more work. It's also tuned for low latency (smallest pause when GC has to stop the program) at the expense of throughput (i.e. total speed). This is the right trade-off for most programs but doesn't perform optimally on micro-benchmarks that measure throughtput.Setting the gc percent at 800 (100 by default) more than halves the time that the function takes to compute:

$ time GOGC=800 ./grotsky examples/fib.g832040real    0m5,110suser    0m5,182ssys     0m0,061s

Interpreting functions

Callable interface

type callable interface {	arity() int	call(exec *exec, arguments []interface{}) interface{}}

_All grotsky functions must be an object that implements the callable interface._For that I defined two kind of structs:

type function struct {	declaration   *fnStmt	closure       *env	isInitializer bool}type nativeFn struct {	arityValue int	callFn  func(exec *exec, arguments []interface{}) interface{}}

nativeFn

Let's you define standard functions available on all grotsky interpreters. Line println.

func (n *nativeFn) arity() int {	return n.arityValue}func (n *nativeFn) call(exec *exec, arguments []interface{}) interface{} {	return n.callFn(exec, arguments)}

From that, println would be pretty straight forward:

...var println nativeFnprintln.arityValue = 1println.callFn = func(exec *exec, arguments []interface{}) interface{} {    fmt.Println(arguments[0])    return nil}...

Ordinary grotsky functions

For ordinary grotsky functions the things are a little bit messier.First I got to introduce the environment that is an object that holds map[string]interface{} as a dictionary for variables in the local scope and a pointer to another environment that contains variables for the outer scope.

type env struct {	state *state	enclosing *env	values    map[string]interface{}}func newEnv(state *state, enclosing *env) *env {	return &env{		state:     state,		enclosing: enclosing,		values:    make(map[string]interface{}),	}}func (e *env) get(name *token) interface{} {	if value, ok := e.values[name.lexeme]; ok {		return value	}	if e.enclosing != nil {		return e.enclosing.get(name)	}	e.state.runtimeErr(errUndefinedVar, name)	return nil}func (e *env) define(name string, value interface{}) {	e.values[name] = value}

As you can see, the define method creates a variable on the local scope, and the get methods tries to retrieve a variable first from the local scope and then from the outer scope.Let's see how functions are implemented.

func (f *function) arity() int {	return len(f.declaration.params)}func (f *function) call(exec *exec, arguments []interface{}) (result interface{}) {	env := newEnv(exec.state, f.closure)	for i := range f.declaration.params {		env.define(f.declaration.params[i].lexeme, arguments[i])	}	defer func() {		if r := recover(); r != nil {			if returnVal, isReturn := r.(returnValue); isReturn {				result = returnVal			} else {				panic(r)			}		}	}()	exec.executeBlock(f.declaration.body, env)	return nil}

Function arity is pretty simple.The function call takes an exec object, that is no more than an instance of the interpreter, and the arguments to the function as an array of objects. Then creates a new environment the is surrounded by the environment local to the function definition and defines all the function parameters. Then comes the tricky part, first there is a deferred call to an anonymous function, let's ignore that for a moment, in the end, the function executeBlock gets called. Let's see what that function does:

func (e *exec) executeBlock(stmts []stmt, env *env) {	previous := e.env	defer func() {		e.env = previous	}()	e.env = env	for _, s := range stmts {		e.execute(s)	}}

What's happening here is that the interpreter steps into the new environment, saving the previous environment in a variable, and execute all given statements, after that it restores the environment to the previous one. Exactly as a function does.

What happens when you hit a return

type returnValue interface{}...func (e *exec) visitReturnStmt(stmt *returnStmt) R {	if stmt.value != nil {		panic(returnValue(stmt.value.accept(e)))	}	return nil}

When you get to a return node in the ast, the nodes panics with a return value. This has to do with the fact that you need to go up the call stack and finish the execution of the function, otherwise the function will keep it's execution.That's the reason of the deferred function we forgot a couple seconds ago:

func (f *function) call(exec *exec, arguments []interface{}) (result interface{}) {    ...    defer func() {		if r := recover(); r != nil {			if returnVal, isReturn := r.(returnValue); isReturn {				result = returnVal			} else {				panic(r)			}		}    }()    ...}

This function recovers from a panic. If the value recovered is of type returnValue it recovers successfully and sets the result value of the function call to the return value, else it panics again.

Hasta la vista, baby

That's it for now. There are a lot of nifty stuff to keep talking about. But I think it's enough for now.Remember to check out the source code. And stay tuned for more.

Grotsky Part 2: Parsing expressions

2020-03-15T00:00:00Z

Grotsky Part 2: Parsing expressions

Expressions

Parsing an expression like 1+2*3 requires a complex representation on memory. Just looking at it we think that it's pretty simple, but there is some hidden hierarchy that we have to pay attention to, like the fact that first we have to compute 2*3 and then add 1 to it.To represent that in a data structure the best thing we can come up to is a tree, as seen in the next figure:![image](/assets/images/grotsky-part2/AST.png)As you can see the leaves of the tree are literals and the root and intermediate nodes are operations that have to be applied from the bottom up. That means that we traverse the tree until we reach the bottom and start computing the results by going up.

Defining node types

> Not all operations are created equal.We have to define how each node fits into the tree.I'll use the following syntax: Binary -> left expr, operator token, right expr. Which means that a binary operation (as we have seen in the image before) links to 2 expressions (left and right) and stores 1 value (operator).

Let's define all posible operations on literals

Literal -> value object# 1, "asd", 5.2, true, falseBinary -> left expr, operator token, right expr# 1+2, 3*3, 4^2+1Grouping -> expression expr# (1+2)Logical -> left expr, operator token, right expr# true or false, false and trueUnary: operator token, right expr# not true, -5List -> elements []expr# [1, 2, 3, [4], "asd"]Dictionary -> elements []expr# {"a": 1, "b": 2, 3: 4}Access -> object expr, slice expr# [1, 2, 3][0], {"a":1}["a"]Slice -> first expr, second expr, third expr# [1, 2, 3, 4, 5, 6][1:4:2]

Traversing the abstract syntax tree

To traverse the syntax tree we need a pattern that's uniform and easily scalable when we have to add other types of expressions and statements.For that we'll use the Visitor Pattern.

Visitor Pattern

First we need an interface for the expression that allows a visitor to visit it.

type expr interface {    accept(exprVisitor) interface{}}

An expression visitor should have a method for each kind of expression it has to visit.

type exprVisitor interface {    visitLiteralExpr(expr expr) interface{}    visitBinaryExpr(expr expr) interface{}    visitGroupingExpr(expr expr) interface{}    visitLogicalExpr(expr expr) interface{}    visitUnaryExpr(expr expr) interface{}    visitListExpr(expr expr) interface{}    visitDictionaryExpr(expr expr) interface{}    visitAccessExpr(expr expr) interface{}    visitSliceExpr(expr expr) interface{}}

Then we have to define a type for each kind of expression that implements expr interface. For example, this is the implementation for a binary expression:

type binaryExpr struct {    left expr    operator *token    right expr}func (s *binaryExpr) accept(visitor exprVisitor) R {    return visitor.visitBinaryExpr(s)}

For all other expressions the definition is practically the same.

String Visitor

To finish this chapter, let's define a visitor that allows you to print the syntax tree in a lisp-like syntax, ex: (+ 1 2).Here is the implementation of the string visitor for a binary expression:

type stringVisitor struct{}func (v stringVisitor) visitBinaryExpr(expr expr) R {    binary := expr.(*binaryExpr)    return fmt.Sprintf("(%s %v %v)", binary.operator.lexeme, binary.left.accept(v), binary.right.accept(v))}

Grotsky expression

You can check out the state of the Grotsky project right here: https://github.com/mliezun/grotsky.Grotsky it's able to parse and print all types of expressions defined in this article right now.

Expressions

Examples of operations supported:

# Math operations1+2*3^2-(4+123)/2.6=> (- (+ 1 (* 2 (^ 3 2))) (/ (+ 4 123) 2.6))# Logical operationstrue or false=> (or true false)# Comparisons1 == 1 and (1 > 3 or 11/5.5 <= 3+2^2 and 1 != 2)=> (and (== 1 1) (or (> 1 3) (and (<= (/ 11 5.5) (+ 3 (^ 2 2))) (!= 1 2))))# Lists[1, 2, [3], "asd"]=> (list 1 2 (list 3) "asd")# List slicing[1,2,3,4][1:3][::2][0]=> (#0 (#::2 (#1:3 (list 1 2 3 4))))# Dictionary{    1: 2,    3: 4,    "asd": 3.14}=> (dict 1=>2 3=>4 "asd"=>3.14)# Dictionary key lookup{"key":0.6}["key"]=> (#"key" (dict "key"=>0.6))

That's it for now. In the next chapter we'll traverse the tree but instead of printing we'll execute the operations listed before.If you have questions or suggestions please get in touch.

Grotsky Part 1: Syntax

2020-02-21T00:00:00Z

Grotsky Part 1: Syntax

Syntax Restrictions

No use of semicolon ;

Block statements delimited by begin and end

Function definition using fn keyword

Logic operators in plain english or, and, not

Conditional statements use the following keywords: if, elif, else

There is no switch statement

Class definition with class keyword

Arithmetic operations: *, /, -, +, ^

Grouping with parentheses ()

Native support for python-like lists and dictionaries: [], {}

Support for enhanced for loop: for i, el in array

Keywords and identifiers can only use alphabethic characters

Primitives

nil

Integers

Floats

Booleans

Strings

Lists

Dictionaries

Example of functions and operations

## Arithmethicprint(2^10 - 2323*3)# Output: -5945print(2^(12*3+400/-4+10*5/2))# Output: 1.8189894035458565e-12## Logicprint(true or false)# Output: true (short circuit)print(false and true)# Output: false (short circuit)## Conditionalsif 3 > 2 or (1 < 3 and 2 == 2) begin    print('Condition is true')endelif 3 == 4 begin    print('Condition 2 is true')endelse begin    print('Conditions are false')end## Listsfor i in [1, 2, 3, 4] begin    print(i)endlet lst = [1, 2, 3, 4]lst[0] = -1print(lst) # Output: [-1, 2, 3, 4]print(lst[1:3]) # Output: [2, 3]## Dictionaries# (dictionaries and lists not allowed as keys)let dct = {    "Key1": "Val1",    2: "Val2",    true: false}for key, val in dct begin    print(key, val)end## Functionsfn square(x)begin    return x^2endfn operate(x, operation)begin    return operation(x)end## Clojurefn makeCounter()begin    let n = 0    return fn() begin        n = n+1        return n    endend## Classesclass Counterbegin    init(start) begin        self.start = start    end    count() begin        self.start = self.start+1        return self.start    endendclass CounterTwobegin    count() begin        return super.count()*2    endend

Syntax definition

Let's build a syntax definition in backus naur format that will be easy to parse with a recursive descent parser.

Expresions

expression       assignment;list             "[" arguments? "]";dictionary       "{" dict_elements? "}";dict_elements    keyval ("," keyval)*;keyval           expression ":" expression;assignment       (call ".")? IDENTIFIER "=" assignment | access;access           logic_or ("[" slice "]")*;logic_or         logic_and ("or" logic_and)*;logic_and        equality ("and" equality)*;equality         comparison (("!=" | "==") comparison)*;comparison       addition ((">" | ">=" | "<" | "<=") addition)*;addition         multiplication (("-" | "+") multiplication)*;multiplication   power (("/" | "*") power)*;power            unary ("^" unary)*;unary            ("not" | "-") unary | call;call             primary ("(" arguments? ")" | "." IDENTIFIER)*;arguments        expression ("," expression)*;slice            (":" expression)                | (":" expression ":" expression)                | (":" ":" expression)                | expression                | (expression ":")                | (expression ":" expression)                | (expression ":" ":" expression)                | (expression ":" expression ":" expression);primary          NUMBER                | STRING                | "false"                | "true"                | "nil"                | IDENTIFIER                | "(" expression ")"                | fnAnon                | list                | dictionary;fnAnon           "fn" "(" parameters? ")" block;

Statements

program         declaration* EOF;declaration     classDecl | funDecl | varDecl | statement;classDecl       "class" IDENTIFIER ( "<" IDENTIFIER )? "begin" methodDecl* "end" NEWLINE;methodDecl      "class"? function;funDecl         "fn" function ;function        IDENTIFIER "(" parameters? ")" block ;parameters      IDENTIFIER ( "," IDENTIFIER )* ;varDecl         "let" IDENTIFIER ("=" expression)? NEWLINE;statement       forStmt                | ifStmt                | returnStmt                | whileStmt                | exprStmt                | block;exprStmt        expression NEWLINE;forStmt         "for"  (classicFor | newFor) statement;classicFor      (varDecl | exprStmt | ",") expression? "," expression?;newFor          IDENTIFIER ("," IDENTIFIER)? "in" expression;ifStmt          "if" expression statement ("elif" expression statement)* ("else" statement)?;returnStmt      "return" expression? NEWLINE;whileStmt       "while" expression statement;block           "begin" NEWLINE declaration* "end" NEWLINE;

That's it! The next step is to build a lexer and a parser.

Sudoku Solver

2020-02-18T00:00:00Z

Sudoku Solver

I wanted to make my own sudoku solver to challenge myself.Im not a sudoku player so my approach is a brute force scan of possible combinations sort-of.I just know the basic rules:

Numbers 1-9 are allowed.

Numbers in the same row cannot be repeated.

Numbers in the same column cannot be repeated.

Numbers in the 3x3 square cannot be repeated.

The first thing i did was to build a some classes that calculates the possible values a cell can have if it's empty, based on the constraints.I came up with 3 classes:

Board that stores the entire board.

BoardSlice that stores a slice of a board. An object of this type is returned when a Board is sliced (method __getitem__).

Cell that stores the value of a single cell and calculates all possible values a cell can take.

The class Cell receives a board, the coordinates on the board, and the value that holds. Also has the method options that uses python set data structure to calculate the posibilites.If you look at the following snippet you can see that the method optionsgenerates the sets: options that contains all possible options (1-9), row that contains all the numbers that are in the same row, column that contains all the numbers that are in the same column and square that contains all the numbers that are in the same 3x3 square. The return value is options without all the used values.

class Cell:    def __init__(self, b, i, j, value):        self.b = b        self.value = value        self.i = i        self.j = j    def options(self):        if self.value != 0:            return {self.value}        options = set(range(1, 10))        row = set(map(lambda x: x.value, self.b[self.i]))        column = set(map(lambda x: x.value, self.b[:][self.j]))        def to_square(k): return slice((k // 3) * 3, (k // 3) * 3 + 3)        square = set(            map(lambda x: x.value,                self.b[to_square(self.i)][to_square(self.j)]))        return options - row - column - square - {0}

To make easier the implementation of the square I used the class BoardSlice that contains a slice of a board and implements the magic method __getitem__.

class BoardSlice:    def __init__(self, board_slice):        self.board_slice = board_slice    def __getitem__(self, items):        if type(items) == slice:            return (el for row in self.board_slice for el in row[items])        if type(items) == int:            return (row[items] for row in self.board_slice)        raise KeyError

The base class: Board contains the board and a copy method that copies all the values and creates a new Board object. This is necessary to avoid messing with object references and have a clean object when needed.

class Board:    def __init__(self, board):        self.board = [[Cell(self, i, j, value)                       for (j, value) in enumerate(row)] for (i, row) in enumerate(board)]    def copy(self):        return Board(((cell.value for cell in row) for row in self.board))    def __getitem__(self, items):        if type(items) == int:            return self.board[items]        if type(items) == slice:            return BoardSlice(self.board[items])        raise KeyError    def __repr__(self):        return repr(self.board)

With these tools the next step is to solve the problem!My idea was to generate a mixed iterative-recursive algorithm.The first pass will be iterative, and if needed, the second pass will be recursive.

Iterative pass

Iterates over the whole board and calculates the options that each cell can have. If a cell has only one option set that option on the cell and set a flag to repeat the iterative pass, if has 0 options return None meaning that the board has no solutions, and if has more than one option store the options for the recursive pass.If the loop ends and we found that no cell has more than one option then we solved the board!The idea of this first step is to solve an _easy_ board quickly.

Recursive pass

If the iterative pass ends and we found that a cell has more than one option then we try all that options and call solve again!If solve returns a board that means we've found the solution!If solve returns None (back at the iterative passs) we have to try with another options.

BoardSolver

The class is pretty straightforward.

class SudokuSolver:    @staticmethod    def solve(board):        b = board.copy()        # First pass: Iterative        board_map = {}        exhaust = False        while not exhaust:            exhaust = True            for i in range(9):                for j in range(9):                    cell = b[i][j]                    if cell.value == 0:                        options = cell.options()                        if len(options) == 1:                            cell.value = options.pop()                            exhaust = False                        elif len(options) == 0:                            return None                        elif len(board_map) == 0:                            board_map[(i, j)] = options        # Second pass: Recursive        for ((i, j), options) in board_map.items():            for op in options:                b[i][j].value = op                solved = SudokuSolver.solve(b)                if solved:                    return solved            return None        return b

Conclusions

Actually my implementation is not a brute force algorithm, is a search algorithm, that searches the path to solving a board. Because it doesn't try all values on all cells nonsensically, it rather tries _some_ options for a given cell and advances to the next option as _soon_ as it detects that it's not the correct path.

Source

Take a look at the source code.

Crafting interpreters

2020-02-12T00:00:00Z

Crafting interpreters

I've just finished the section 2 of the book _Crafting Interpreters_, and I wanted to upload it to github right away.Take a look at the source code.Beside the lox specification I've added:

The keyword until that is a variation of while loops (as in ruby). also

print is a function instead of a statement.

eval function that let's you eval source code in runtime.

Class methods.

I'll implement another language interpreter, this time using golang and with a syntax similar to ruby.

Reinventing the Wheel: PHP Generators

2020-01-24T00:00:00Z

Reinventing the Wheel: PHP Generators

First thing first. How a generator works?

Starting back at C

Let's create a function that each time we call it we get the next number of the fibonacci sequence.

int fibonacci(){    static int a = 0;    static int b = 1;    int aux = b;    b = a + b;    a = aux;    return a;}

If we call fibonacci(), the first time we'll get 1, the second time 1, the third 2, the fourth 3, and so on...This happens because we declared variables a, b to be static. This means that they mantain the value after the function returns. Normally, what happens (if we don't declare a variable as static) is that the variables inside the function don't mantain the values of the last execution.

First generator for PHP

The equivalent function in PHP is pretty similar to C's approach.

function fibonacci(){    static $a = 0;    static $b = 1;    $aux = $b;    $b = $a + $b;    $a = $aux;    return $a;}$out = [];for ($i = 1; $i <= 10; $i++) {    $out[] = fibonacci();}echo implode(', ', $out) . "\n";/*Output: 1, 1, 2, 3, 5, 8, 13, 21, 34, 55*/

Let's compare this to the real PHP version using yield.

function fibonacci($N){    $a = 0;    $b = 1;    for ($i = 0; $i < $N; $i++) {        $aux = $b;        $b = $a + $b;        $a = $aux;        yield $a;    }}$out = [];foreach (fibonacci(10) as $fib) {    $out[] = $fib;}echo implode(', ', $out) . "\n";/*Output: 1, 1, 2, 3, 5, 8, 13, 21, 34, 55*/

Creating a custom version of PHP yield

This is my own version using the library parallel and channels (probably uses yield internally).

class MyGenerator implements Iterator{    private $chan;    private $current;    private $iteratorFn;    private $runtime;    private $key = -1;    private $valid = true;    public function __construct($iteratorFn)    {        $this->iteratorFn = $iteratorFn;        $this->runtime = new \parallel\Runtime();        $channel = new \parallel\Channel();        $this->runtime->run(function() use ($iteratorFn, $channel) {            $iteratorFn(function ($val) use ($channel) {                $channel->send($val);            });            $channel->close();        });        $this->chan = $channel;        $this->next();    }    public function current()    {        return $this->current;    }    public function next()    {        try {            ++$this->key;            $val = $this->chan->recv();            $this->current = $val;        } catch (\parallel\Channel\Error\Closed $e) {            $this->valid = false;        }        return $this->current;    }    public function key() {return $this->key;}    public function valid() {return $this->valid;}    public function rewind() {}}function fibonacci($N){    return new MyGenerator(function ($yield) use ($N) {        $a = 0;        $b = 1;        for ($i = 0; $i < $N; $i++) {            $aux = $b;            $b = $a + $b;            $a = $aux;            $yield($a);        }    });}$out = [];foreach (fibonacci(10) as $fib) {    $out[] = $fib;}echo implode(', ', $out) . "\n";

Performance comparison: PHP vs Custom

Tested code

for ($i = 0; $i < 1000; ++$i) {    foreach (fibonacci(100) as $fib) {        $out[] = $fib;    }}

yield version

real    0m0,083suser    0m0,059ssys     0m0,023s

MyGenerator version

real    0m2,138suser    0m1,426ssys     0m1,363s

So, it's aproximately 26 times slower :-)