• As many other companies embracing AI, Microsoft/GitHub are training their models based on code and other data that does not belong to them. For example, GitHub Copilot is trained exclusively on projects hosted on GitHub, and completely ignores copyleft licenses (see also GiveUpGithub).

• Microsoft is in strategic partnerships with companies like Palantir and ICE, thus actively contributing to (AI-assisted) human-rights violations in the US and across the globe.

• Microsoft is perhaps the most complicit tech company in Israel’s illegal apartheid regime and ongoing genocide against 2.3 million Palestinians in Gaza.

• More generally, it makes sense for open source development not to be entirely dependant on the benevolence of one of the biggest tech companies on the planet.

As a result, I will be deleting my data from GitHub. For practical reasons I will keep my GitHub account to be able to contribute to projects hosted there, but hope that will become unnecessary long-term. In the future I want to host my own public instance of Forgejo, but for the time being I will be using Codeberg, which seems to be a good alternative.

If you’re interested in some other takes on why it makes sense to move off of the platform, check out this and this post.

The journey

I was trying to get some data from my NAS, and noticed that the connection seemed quite slow, around 8-10 MBit. I started out by making sure that this is not an issue with my internet connection. I ran the following from various machines on that network, and got consistent results:

$ docker run --rm moutten/speedtest-cli --accept-gdpr
Speedtest by Ookla

     Server: <redacted>
        ISP: <redacted>
    Latency:     1.65 ms   (0.04 ms jitter)
   Download:   389.15 Mbps (data used: 473.9 MB )
     Upload:   383.03 Mbps (data used: 425.9 MB )
Packet Loss:     0.0%

At first I suspected the issue was with my mounted SMB share, so I switched to SFTP and was seeing the same issues. I then tried to use iperf3 to get more insights into whats going on. On the NAS, I ran the following to start an iperf3 server:

docker run -it --rm --network=host networkstatic/iperf3 -s

Then, from the remote machine, I ran the following tests. First, in the direction from the remote machine to the NAS, which showed good speeds:

$ iperf3 -c 100.101.102.104

Connecting to host 100.101.102.104, port 5201
[  5] local 100.85.100.101 port 56494 connected to 100.101.102.104 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  54.7 MBytes   459 Mbits/sec  102   1.04 MBytes
[  5]   1.00-2.00   sec  26.2 MBytes   220 Mbits/sec    4    297 KBytes
[  5]   2.00-3.00   sec  22.5 MBytes   189 Mbits/sec    0    343 KBytes
[  5]   3.00-4.00   sec  26.2 MBytes   220 Mbits/sec    0    391 KBytes
[  5]   4.00-5.00   sec  28.8 MBytes   241 Mbits/sec    0    437 KBytes
[  5]   5.00-6.00   sec  32.5 MBytes   273 Mbits/sec    0    482 KBytes
[  5]   6.00-7.00   sec  36.2 MBytes   304 Mbits/sec    0    529 KBytes
[  5]   7.00-8.00   sec  38.8 MBytes   325 Mbits/sec    0    576 KBytes
[  5]   8.00-9.00   sec  43.8 MBytes   367 Mbits/sec    0    622 KBytes
[  5]   9.00-10.00  sec  46.2 MBytes   388 Mbits/sec    0    669 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   356 MBytes   299 Mbits/sec  106             sender
[  5]   0.00-10.01  sec   353 MBytes   296 Mbits/sec                  receiver

However, examining traffic in the other direction (note the -R in the iperf command) showed a very different picture, with slow and variable connection speeds as well as intermittent stalling (sometimes):

Connecting to host 100.101.102.104, port 5201
Reverse mode, remote host 100.101.102.104 is sending
[  5] local 100.85.100.101 port 37272 connected to 100.101.102.104 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.37 MBytes  11.5 Mbits/sec
[  5]   1.00-2.00   sec   803 KBytes  6.58 Mbits/sec
[  5]   2.00-3.00   sec   838 KBytes  6.87 Mbits/sec
[  5]   3.00-4.00   sec   620 KBytes  5.08 Mbits/sec
[  5]   4.00-5.00   sec   788 KBytes  6.45 Mbits/sec
[  5]   5.00-6.00   sec   680 KBytes  5.57 Mbits/sec
[  5]   6.00-7.00   sec  1.25 MBytes  10.5 Mbits/sec
[  5]   7.00-8.00   sec   548 KBytes  4.49 Mbits/sec
[  5]   8.00-9.00   sec   937 KBytes  7.67 Mbits/sec
[  5]   9.00-10.00  sec  1.08 MBytes  9.06 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  9.00 MBytes  7.54 Mbits/sec   68             sender
[  5]   0.00-10.00  sec  8.79 MBytes  7.38 Mbits/sec                  receiver

Notably, this was not an issue over UDP:

$ iperf3 --udp --client 100.101.102.104 --bitrate 350M -R
Connecting to host 100.101.102.104, port 5201
Reverse mode, remote host 100.101.102.104 is sending
[  5] local 100.85.100.101 port 38418 connected to 100.101.102.104 port 5201
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  41.7 MBytes   350 Mbits/sec  0.017 ms  0/35630 (0%)
[  5]   1.00-2.00   sec  41.7 MBytes   350 Mbits/sec  0.018 ms  21/35626 (0.059%)
[  5]   2.00-3.00   sec  41.7 MBytes   350 Mbits/sec  0.032 ms  0/35625 (0%)
[  5]   3.00-4.00   sec  41.7 MBytes   350 Mbits/sec  0.015 ms  0/35633 (0%)
[  5]   4.00-5.00   sec  41.7 MBytes   350 Mbits/sec  0.021 ms  0/35619 (0%)
[  5]   5.00-6.00   sec  41.7 MBytes   350 Mbits/sec  0.022 ms  0/35628 (0%)
[  5]   6.00-7.00   sec  41.7 MBytes   350 Mbits/sec  0.036 ms  0/35617 (0%)
[  5]   7.00-8.00   sec  41.7 MBytes   350 Mbits/sec  0.020 ms  59/35639 (0.17%)
[  5]   8.00-9.00   sec  41.7 MBytes   350 Mbits/sec  0.017 ms  0/35628 (0%)
[  5]   9.00-10.00  sec  41.7 MBytes   350 Mbits/sec  0.028 ms  0/35620 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.01  sec   418 MBytes   350 Mbits/sec  0.000 ms  0/356725 (0%)  sender
[  5]   0.00-10.00  sec   417 MBytes   350 Mbits/sec  0.028 ms  80/356265 (0.022%)  receiver

The solution

So it turns out that Tailscale tunnels all of its traffic over UDP, which was easy to see running tcpdump on either of the machines running iperf. Not being very versed in networking, I’m not really sure what’s at play here specifically, but I think it’s something along the lines of:

1. UDP upstream from my connection is slightly lossy
2. TCP tunneled over UDP interprets dropped / missed packages as congestion, and throttles

What helped in the end was to increase TCP socket buffers on my NAS:

sudo sysctl -w net.ipv4.tcp_rmem="4096 131072 50000000"
sudo sysctl -w net.ipv4.tcp_wmem="4096 131072 50000000"

The specific values I decided based on this page.

This worked okay for a comparatively low-latency scenario (~20ms). However, for higher latencies (>100ms) the I saw very slow speeds and stalls once more. Using BBR congestion control helped in that situation, which is apparently designed to work well for situations where TCP traffic is known to be tunneled. Unfortunately my NAS does not support it.

sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
sudo sysctl -w net.core.default_qdisc=fq

The problem: No matter how I mounted my photos, they were always mounted with rwx permissions only for the user. This didn’t work for me because the photos weren’t all owned by the same user. I’m not sure what’s at play here, but if I change the user ID of the photoview user to any of the user IDs on the host system, they were suddenly mounted with rwx permissions for the groups as well:

FROM photoview/photoview:2

USER root
RUN usermod -u 1026 photoview
USER photoview

After that, the compose.yml looks mostly like the default one from the Photoview devs:

services:
  photoview-prepare:
    build: .
    user: root
    entrypoint: []
    command: /bin/bash -c "sleep 1 && chown -R photoview /home/photoview/media-cache /home/photoview/database"
    cap_add:
      - CHOWN
    volumes:
      - "/etc/localtime:/etc/localtime:ro" ## use local time from host
      - "/etc/TZ:/etc/timezone:ro" ## use timezone from host
      - "${HOST_PHOTOVIEW_LOCATION}/storage:/home/photoview/media-cache"
      - "${HOST_PHOTOVIEW_LOCATION}/database:/home/photoview/database"

  photoview:
    build: .
    restart: unless-stopped
    stop_grace_period: 10s
    ports:
      - "127.0.0.1:40001:8000" ## HTTP port (host:container)
    depends_on:
      photoview-prepare:
        condition: service_completed_successfully
    security_opt:
      - seccomp:unconfined
      - apparmor:unconfined
    environment:
      PHOTOVIEW_DATABASE_DRIVER: ${PHOTOVIEW_DATABASE_DRIVER}
      PHOTOVIEW_SQLITE_PATH: ${PHOTOVIEW_SQLITE_PATH}
      PHOTOVIEW_LISTEN_IP: "0.0.0.0"
      PHOTOVIEW_LISTEN_PORT: 8000
    volumes:
      - "/etc/localtime:/etc/localtime:ro" ## use local time from host
      - "/etc/TZ:/etc/timezone:ro" ## use timezone from host
      - "${HOST_PHOTOVIEW_LOCATION}/database:/home/photoview/database"
      - "${HOST_PHOTOVIEW_LOCATION}/storage:/home/photoview/media-cache"
      - "${HOST_PHOTOVIEW_MEDIA_ROOT}/foo:/photos/foo:ro"

For completeness, here’s also my .env file:

HOST_PHOTOVIEW_LOCATION=/volume1/docker/photoview
HOST_PHOTOVIEW_MEDIA_ROOT=/volume1/SecurePhotos

PHOTOVIEW_DATABASE_DRIVER=sqlite
PHOTOVIEW_SQLITE_PATH=/home/photoview/database/photoview.db

WATCHTOWER_POLL_INTERVAL=86400
WATCHTOWER_TIMEOUT=30s
WATCHTOWER_CLEANUP=true

The full error message I was given was the following:

              .fallback(fallback_handler);
    |          -------- ^^^^^^^^^^^^^^^^ the trait `Handler<_, ()>` is not implemented for fn item `fn(Extension<State>, Request<Body>) -> impl Future<Output = impl IntoResponse> {fallback_handler}`
	|          |
	|          required by a bound introduced by this call
    |
    = help: the trait `Handler<T, S, B2>` is implemented for `Layered<L, H, T, S, B, B2>`

The error message didn’t help me in figuring out what’s going on. The most confusing part was that the problem seemed to be caused by instantiating a struct I had just written, but I hadn’t changed anything about the function signature or the return value.

My handler looked something like this:

async fn fallback_handler(
    Extension(state): Extension<State>,
    request: Request<Body>,
) -> impl IntoResponse {
	let mut errors: Vec<Box<dyn Error>> = vec![];
	// [...]
	let tera = state.tera.lock().await;
	// [...]
	StatusCode::OK
}

Now, if you’re familiar with async Rust, you might already see the problem. It’s the same issue described in detail here, namely that

1. std::error::Error is not Send, i.e. it cannot safely be sent between threads
2. Holding a non-Send type across an await results in a non-Send Future (explanation)

Axum’s Handler type, however, requires the resulting Future to be Send, as can be seen here:

pub trait Handler<T, S, B = Body>: Clone + Send + Sized + 'static {
	/// The type of future calling this handler returns.
	type Future: Future<Output = Response> + Send + 'static;
	// [...]
}

So, long story short: don’t try to hold non-Send types across await calls when you’re writing an Axum handler.

Pipeline structure

I’ll be talking about the following scenario:

The reason I would always include an intermediate environment (e.g. “staging”), is that I would need somewhere to run my automated acceptance tests¹ against, which try to make sure that the changes made do not brake any existing functionality. So in reality, the pipeline would look a little more like this:

Looking at the part of the pipeline after the unit test stage (blue) completes, it looks the same for both staging and production. First we want to deploy to the stage, then run our acceptance tests against said stage.

As far as I’m aware, for Gitlab pipeline definitions, we have three ways to handle this duplication:

1. YAML anchors and merge key
2. Gitlab CI extends directive
3. Gitlab CI child pipelines

I won’t go into detail about any of them, but I’ve found (3) to be the one that I prefer. It allows me to (mostly) separate the “deployment / acceptance test” logic into its own building block, making it easy to, for example, add new stages without having to copy and paste boilerplate.

The parent pipeline

Okay, so let’s look at some .gitlab-ci.yml (full source available here). It starts out simple enough:

stages:
  - "Analyze/Test"
  - "Deploy Staging"
  - "Deploy Production"

workflow:
  # For jobs which contain `rules`, removing this causes strange behaviour
  # when merge requests are created. In that scenario, additional detached
  # pipeline runs are started.
  # See also: https://gitlab.com/gitlab-org/gitlab/issues/34756#note_282820625
  rules:
    - if: $CI_COMMIT_BRANCH

test:
  stage: "Analyze/Test"
  script:
    - echo "Running test suite ..."

Not many surprises here, except for the workflow workaround (which will hopefully be fixed in the future). In this file you would add all stages that are independent of deployment, that is, all the checks that you wouldn’t want to re-run for each stage you’re deploying to. This would likely include your unit tests, integration tests, format checks, etc.

In addition, this is where we trigger the child pipelines, which will deploy to the individual environments:

deploy_staging:
  stage: "Deploy Staging"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
    - when: never
  trigger:
    include: .gitlab-ci/deploy-environment.yml
    strategy: depend
  variables:
    ENVIRONMENT_NAME: staging

In the rules section we define that the deployment should only be run when updating the main branch. In variables, we pass whatever information is needed for the deployment, which in this example case is only the name. Finally, the trigger starts the child pipeline, in a way that the current (parent) pipeline will depend on the result.

The child pipeline

Finally, the child pipeline we’re triggering in the example above will be as simple or complex as your deployment / acceptance testing logic:

# Contents of `.gitlab-ci/deploy-environment.yml`
stages:
  - deploy
  - acceptance

deploy_static_environment:
  stage: deploy
  environment:
    name: "$ENVIRONMENT_NAME"
  variables:
    WORKSPACE: $CI_ENVIRONMENT_NAME
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
    - when: never
  script:
    - echo "Deploying $ENVIRONMENT_NAME"

acceptance_tests_branch:
  stage: acceptance
  script:
    - echo "Deploying $ENVIRONMENT_NAME"

Because we’re triggering the child pipeline with strategy: depend, the parent will not continue to run until the child has completed successfully.

You may think that this is a lot of effort for a relatively simple pipeline. And, if we’re just looking at the example I wrote, I would agree. However, CI/CD pipelines tend to grow more complex over time, and, in my experience, a 1000+ line .gitlab-ci.yml file tends to be a bit cumbersome to understand, and especially to extend.

In a recent project I ran into a situation which had a relatively complex Spring Security configuration as well as some authorization logic sprinkled across it’s controllers. In order to gain more confidence in refactoring the security configuration, I wanted an isolated test telling me whether my modifications also changed the configuration’s behaviour. I came up with what’s documented in this post.

Let’s start out by looking at the Spring Security configuration we’ll be testing as part of this article:

@Override
protected void configure(HttpSecurity http) throws Exception {
    http
        .formLogin()
            .and()
        .logout()
            .and()
        .authorizeRequests()
            .antMatchers("/secure/**")
                .authenticated()
            .antMatchers("/admin/**")
                .hasRole("ADMIN")
            .anyRequest()
                .permitAll();
}

Here we are setting up a form login and a logout page, and then proceed to say that there are three separate areas:

• A “secure area”, consisting of all sub-pages of http://example.com/secure. All logged in users can see this content
• An admin area, containing all sub-pages of http://example.com/admin. Only users with the ADMIN role can see this content
• Everything else, which is world accessible

In order to test this configuration, I first created a dummy controller in my test code, which catches all GET requests:

@Controller
public class DummyCatchAllController {
    @RequestMapping(path = "/**", method = RequestMethod.GET)
    @ResponseBody
    public String catchAll() {
        return "DummyCatchallController#catchAll";
    }
}

Then I proceeded to set up a test which, using this dummy controller, tests the security configuration. First the boilerplate:

@RunWith(SpringJUnit4ClassRunner.class)
@WebMvcTest(DummyCatchAllController.class)
@Import({SecurityConfiguration.class})
public class SecurityConfigurationTest {
    private MockMvc mockMvc;
    @Autowired private WebApplicationContext webApplicationContext;

    @Before
    public void setup() {
        this.mockMvc =
            MockMvcBuilders
                .webAppContextSetup(this.webApplicationContext)
                .apply(springSecurity())
                .build();
    }

    [...]
}

Instead of @SpringBootTest (loads your full application context) we’re using @WebMvcTest (only loads whatever the controller needs) here, because they are commonly faster¹ and we’re trying to isolate the security configuration as much as possible. Since WebMvcTests don’t load the entire application context though, we need to manually import the security configuration with

@Import({SecurityConfiguration.class})

Next up, in the setup method, we’re initializing MockMVC, which will help us perform our test requests against the controller. We cannot use the @AutoConfigureMockMvc annotation here, because we need to make sure that springSecurity() is applied, since that’s what we want to test.

Now that we’ve got the setup finished, we can finally look at the actual tests. If you’re unfamiliar with how MockMVC works, and the examples are unclear, I recommend looking at this introduction.

First, let’s make sure that our world-accessible resources can be seen by any user:

@Test
public void asAnonymous_ICanAccessPublicResources() throws Exception {
    mockMvc.perform(get("/foobar")).andExpect(status().isOk());
    mockMvc.perform(get("/")).andExpect(status().isOk());
}

Next, let’s see if a loggged in user can see the “secure area”. Here we’re using the @WithMockUser annotation to simulate a logged-in user who has the USER role.

@Test
@WithMockUser(username = "user", roles = {"USER"})
public void asLoggedInUser_ICanAccessSecureArea() throws Exception {
    mockMvc.perform(get("/secure/foobar")).andExpect(status().isOk());
    mockMvc.perform(get("/secure/")).andExpect(status().isOk());
}

At the same time, we also want to make sure that a normal user can’t see the admin area:

@Test
@WithMockUser(username = "user", roles = {"USER"})
public void asLoggedInUser_ICantAccessAdminArea() throws Exception {
    mockMvc.perform(get("/admin/foobar")).andExpect(status().isForbidden());
    mockMvc.perform(get("/admin/")).andExpect(status().isForbidden());
}

Finally, an admin should be able to see the admin area:

@Test
@WithMockUser(username = "admin", roles = {"USER", "ADMIN"})
public void asAdminUser_ICanAccessAdminArea() throws Exception {
    mockMvc.perform(get("/admin/foobar")).andExpect(status().isOk());
    mockMvc.perform(get("/admin/")).andExpect(status().isOk());
}

Note that the usernames that we’re supplying to the @WithMockUser annotation in all of these examples does not actually have any meaning. I’ve only added it to make the test cases clearer.

That was it already. If you’re interested in a runnable example to play around with, you can find it here.

$ brew cask install xquartz
$ brew install homebrew/x11/bochs

When trying to run it, I got the following error though

Event type: PANIC
Device: [GUI   ]
Message: Cannot connect to X display

This happens because either XQuartz is not running or Bochs does not know how to communicate with it. You’ll need to do two things:

1. Start XQuartz.app
   • This will open an xterm terminal. Don’t close it
2. Now you have two options:
   • Run Bochs from within xterm
   • Run echo $DISPLAY inside xterm and copy the result. Then run export DISPLAY="<copied string>" in your normal terminal
3. Either way, Bochs will now know which X display to connect to

Jane: On a scale from one to ten, how proficient would you say you are in JavaScript?

Tom: Uhm, not sure. I’m not even sure what the numbers mean in this context.

Jane: Well, it goes from “I never heard of it” to “I know everything there is to know”.

Tom: I really can’t say… Maybe a five? …

And I really feel for Tom. Not because he is only “five good” in JavaScript, but because I consider this whole “How would you rate your $foo on a scale from $bar to $baz” thing to be nonsense. Back in school, my physics teacher would always tell me how I must never forget the units in my calculations, because otherwise I wouldn’t be able to interpret the result. But as it turns out, there is no unit for JavaScript knowledge, nor for any other programming language, because skill is very diffcult to quantify.

Let’s take another look at why this or any other arbitrary scale is no good for skill (self-)assessment. We could assume that Tom was brought up in Spain, where the school system rates you from 1 (worst) to 10 (best). While in theory this could be a linear scale, in practice it is far from it: 1-4 means that you’ve failed the class entirely and have to repeat it. The remainder is mapped as follows:

• 5 ≡ sufficient
• 6 ≡ good
• 7-8 ≡ notable
• 9-10 ≡ outstanding

As a result, Tom might be inclined to think that being “five good” in JavaScript signifies that you have only very basic knowledge of the language because this is consistent with how he’s been graded his entire school career. Now we could assume that Jane was brought up in Germany where school grades are commonly 1 (best) through 6 (worst), with a 5 and a 6 indicating a failed class. Since this is not really compatible with our one-to-ten-scale, she might simply assume that skill level is distributed linearly. Both of their assumptions are visualized in the figure below, colors ranging from red (poor skill) through yellow (mediocre) to green (skilled).

Note that this interpretation of the scale is consistent with 1 meaning “you’ve never heard of it” and 10 meaning “I know everything there is know”. The distribution in-between is very different though, such that being a “five” JavaScript coder for Jane would be roughly a “seven” for Tom, and so on. As a result, either Jane would think that Tom is massively overestimating himself or Tom would receive assignments that are way outside his own perceived skillset.

So now I’ve critized the one-to-ten-scale. “But which one should I use instead?”, you ask. The answer could be: whichever you like, just keep the scale to yourself.

Using binary search to position someone on a scale of your choice

The thing is, that if you ask a question such as “How good are you on a scale from […]”, then you already have an idea of what a “seven” (or any other number) means for a JavaScript programmer (at least in your own, very subjective scale). As a result, supposing that you are able to come up with a question for every number on your scale, you’ll be able to position any person you encounter on any scale in about log2(10) steps (3.32, for the 1-10 scale), by using binary search starting from the middle of the scale.
For example:

(1 2 3 4 5 6 7 8 9 10) - Pivot: 5. Answer: Correct
(          6 7 8 9 10) - Pivot: 8. Answer: Correct
(                9 10) - Pivot: 9. Answer: Incorrect
                       = Result: An 8

Since a scale is never communicated to the interviewee, no misunder­standings regarding the nature of the scale can arise. An additional benefit of this technique is that you can refine your questionnaire over time, based on your observations, thus gradually increasing the precision of your assessments. It should be noted that these must not be questions that can be memorized but could also be small programming excercises or the like, thus making this a viable technique for programming interviews as well. And it can even be automated!

What do you think about the one-to-ten-scale? Is it useful, and if so, in which situations? Could binary-search-based assessment be next big thing™?

Requirements

I worked in an Ubuntu 14.10 virtual machine with OS X being the host operating system (which should not matter, I guess). Your experience may differ on other flavors of Linux.

Additionally you’ll need to install minicom for initial communication with the board until an ethernet connection has been established as well as the toolchain to build applications for the target platform. Lastly sshpass will help us make testing our applications on the board a little easier. You can install all dependencies by running:

sudo apt-get install gcc-4.9-arm-linux-gnueabihf minicom sshpass

Getting the board up and running

I’ve used the official Zynq Linux 2014.4, which was the latest available version at the time. Download the release of your choice, extract the archive to an SD card and copy the boot.bin and devicetree.dtb files appropriate for your Zynq platform to the root of the SD card.

Now insert the SD card, connect the micro-usb cable to the UART port of your board and turn it on. If you’re working on Linux there should now be a device named something like /dev/ttyACM0. You can connect to the board using minicom:

sudo minicom -D /dev/ttyACM0

Note that you need root permissions for this operation. After connecting to the board you can then enter username and password (root/root by default) which gets you into a shell.

Next up is configuring your network interface. If you have a DHCP server at hand you can simply plug in the board and it’ll receive an IP address which you can find out by running ifconfig. If, like me, you have the board directly connected to your computer it may be more convenient to configure a static IP address than to install a DHCP server (which this tutorial does not cover). You can achieve this by opening the /etc/network/interfaces file and replacing

iface eth0 inet dhcp

with

iface eth0 inet static
address 192.168.1.2
netmask 255.255.255.0
gateway 192.168.1.1

and subsequently running ifdown eth0 && ifup eth0. Afterwards you should be able to SSH onto the board (don’t forget to configure your network interface locally to have the IP 192.168.1.1).

The first application

Okay, now let’s get the obligatory hello world application to run. I’ve created a file main.c with the following content:

#include <stdio.h>
int main() {
    printf("hello world\n");
    return 0;
}

To make our lives a little easier I’ve written the following Makefile which builds the application, transfers it to the board and runs it, displaying the output and the exit code on the local machine.

BOARD_IP=192.168.1.2
TARGET_DIR=/home/root/
CC=arm-linux-gnueabihf-gcc-4.9
LDFLAGS=-static
CFLAGS=-Wall -Wextra -O2 -std=gnu99 -g -mcpu=cortex-a9 -mfloat-abi=hard
DEPS=main.c
OBJ=main.o
NAME=hello-world

all: $(NAME)

run: $(NAME)
        sshpass -p root scp $(NAME) root@$(BOARD_IP):$(TARGET_DIR) && \
        sshpass -p root ssh root@$(BOARD_IP) "cd $(TARGET_DIR) && ./$(NAME); echo 'Return code: ' $$?"

%.o: %.c $(DEPS)
        $(CC)  $(CFLAGS) -c -o $@ $<

$(NAME): $(OBJ)
        $(CC) -o $@ $(CFLAGS) $(OBJ) $(LDFLAGS)

clean:
        rm -f $(NAME) $(OBJ)

The output of make run the example source code should thus be:

$ make run
hello world
Return code:  0

This concludes the “getting up and running” tutorial for the Zedboard (and similar Xilinx Zynq based platforms). If something doesn’t work for you or if you have suggestions on how this tutorial can be improved, feel free to leave a comment or drop me an email.

For this excercise I used DosBox on OS X, but it should work on any platform that supports DosBox. The mount commands will probably look different if you’re on Windows though.

Things you’ll need

• DosBox
• Borland C++ 3.1
• Keen Dreams Source
• Shareware version of Keen Dreams (for the game assets)

Preparing the environment

Download DosBox and copy it to your Applications folder. Then create a DosBox directory in your home directory (so the full path is /Users/foobar/DosBox) as well as a C and D directory inside it. Then go ahead and download Borland C++ 3.1 and extract it into the A directory.

Now we start up DosBox for the first time. You’ll see a prompt saying Z:\ which is the only drive currently mounted, so we execute the following commands to mount our created directories as drives:

Z:\> mount A /Users/lucas/DosBox/A
Z:\> mount C /Users/lucas/DosBox/C
Z:\> mount D /Users/lucas/DosBox/D

Now we can install Borland C++ by switching to the A: drive and running INSTALL.EXE:

Z:\> A:
A:\> INSTALL.EXE

Following the installation dialog and make sure to install Borland C++ into the C:\BC2 directory, because that is where the “Keen Dreams” Borland project expects it to be installed:

Lastly we need to adjust the PATH environment variable so that the OS knows where to find the Borland binaries. We do this by running SET PATH=Z:\;C:\BC2\BIN. It also contains the Z:\ drive because that is where DosBox stores its binaries! Running BC should now yield the following screen (which you can exit with Alt-X):

Compiling the source

The rest is pretty straight forward. First we need to create the MAKEOBJ.EXE inside the STATIC directory, which is later used to build some of the game’s static files. To do this, run the following:

A:\> D:
D:\> CD KEEN\STATIC
D:\> BC MAKEOBJ.C

This will open Borland C++ with the specified file opened. Now press F9 to compile and link it, thus creating MAKEOBJ.EXE. Exit Borland by pressing Alt+X. Now run MAKE.BAT in the same directory.

We’ve now created the static files, but the game remains to be compiled. So we move on directory up (CD ..) and run BC again. This should automatically find the Keen Dreams project file and open it. Now press F9 again, which builds the project. If everything works as expected, you’ll now have a KDREAMS.EXE binary in your source directory.

Can’t open KDREAMS.MAP

Unfortunately, as with many games that are open-sourced, the actual game content is still licensed differently and is thus not included with the source. As a result, you’ll see a Can't open KDREAMS.MAP error message when you try to start the compiled binary:

In order to run the game, you can download the assets from the shareware version of the game, though. Unpack the downloaded game and copy the following files to the Keen Dreams source directory (this is easier when done from outside DosBox):

KDREAMS.AUD, KDREAMS.CMP, KDREAMS.EGA, KDREAMS.MAP

And voilà, you can now run the game. Have fun playing or fiddling around with the source :)

So roughly 2/3 of my blog’s assets are basically.. junk. Not in general, but certainly for my use case. While Bootstrap might be nice, it’s certainly overkill for a small blog. And jQuery? I’m not even using that anywhere. I’d argue that Prettify.js is justifiable, because nobody wants to read non-highlighted source code, right? And those web fonts? We’ll see if they’re worth the additional 80KB.

How did it come to this?

When I created the blog layout, I was being utterly lazy. I snatched up Twitter Bootstrap and jQuery, added some web fonts, mixed in a little CSS of my own, shook it around a little and done was my blog layout. Did I think about how much unnecessary stuff I was adding? Hell no!

Honestly, thinking about this made me a little sad. When developing for a customer, you can’t always strive for perfection. There are deadlines, requirements and so on (you know the deal) and you usually have to stop somewhere and say “This will work, even though it could be better”. But for personal projects, this does not apply, so being sloppy is really not justifiable.

Let’s take out the trash!

So I set out to dramatically reduce the loading size of my blog. First off, I tried to simply removed Bootstrap, jQuery and Modernizr and see how it goes. You know what? It didn’t look that much different, but reduced the size from 350KB to around 100KB. See for yourself:

It’s not identical, but we just reduced page size by about seventy percent, which seems quite acceptable.

This is not what you would encounter on a large web application, of course. If I had used many of the Bootstrap components, it might have actually been worth to keep, because development time is usually costlier than a few dozen extra kilobytes. And as for jQuery: it’s an awesome library, and nobody really wants to deal with all the browser quirks and discrepancies that jQuery handles for you. But if you just have a couple of lines of JavaScript on your blog, chances are you simply don’t need it.

Web Fonts

The remaining large portion of possibly superfluous bytes were the web fonts at around 80KB, “PT Serif” for the body text and “Lato” for headlines, both served by Google Fonts. At first I thought that they’d be worth keeping, because reading an article in a nice font is so much more enjoyable. End then I found out that one of my favorite fonts is available on around 75% of all Windows and Mac system: Baskerville (chances are you’re seeing it right now). So I used CSS Font Stack to get a recommendation on fall-back fonts for Baskerville and was satisfied by how they look. I followed the same procedure for Lato, and thus was able to replace both web fonts.

Some more tweaking

I mentioned that I wanted to keep Prettify.js for syntax highlighting. But why load it if it isn’t used? So I made it optional per blog post. Oh, and pngquant helped me reduce the size of my picture by 50%, removing another 3KB.

The Status Quo

In summary, I managed to reduce the loading size of my blog’s homepage from initially 350KB to almost 10KB without (visually) sacrificing anything but the web fonts, and I’m fairly satisfied by the result. Let’s take a look:

Before	After

So what can we take away from this? It should once again remind us that there is no “one size fits all” solution. Every technology, framework and library has its place, but I’d argue that the following approach is the road to a heap of technical debt and maintenance misery:

In my last project I used Bootstrap, jQuery and [20 other cool things], so let’s add those and see what we can do with them.

Instead, ask yourself what you are trying to achieve and pick whatever fits the problem you’re trying to solve.

First of all, we’ll want to write a bit of C code which will invoke our assembly routine:

#include <stdio.h>

extern int my_pow(int base, int exp);

int main(int argc, char const *argv[]) {
    int base, exp, result;
    base = 2;
    exp = 8;
    result = my_pow(2,8);
    printf("Result: %d\n", result);
    return 0;
}

Using clang on Mac OS X (you should’ve the latest Xcode installed), you can compile (but not link) this piece of code using the command

clang -o test.o -c test.c

This assumes that your C source file is called test.c. Clang’s -c compiles the given source file to an object file for later linking, and defaults to 64-bit mode. If you’d rather create a 32-bit object file, you can supply the -arch i386 parameter.

So now we’ve compiled our C source, so it’s time to write some assembly. But how does the C function pass the parameters, and how do we return the result? To answer such questions, it is often useful to look at the assembly generated from our C code, which can be obtained with clang -S test.c. The interesting part is the following (this is AT&T syntax):

movl    $2, %eax
movl    $8, %ecx
...
movl    %eax, %edi
movl    %ecx, %esi
callq   _my_pow
...
movl    %eax, -28(%rbp)
movl    -28(%rbp), %esi
movb    $0, %al
callq   _printf

So our operands, 2 and 8, are moved into eax and ecx respectively, which are free-for-use registers. In 32-bit C functions, operands are passed on the stack, but in the System V AMD64 ABI the first integer or pointer operands are passed in registers instead. As a result, our operands are moved to edi and esi, and our _my_pow procedure is called. The return value of the procedure is always stored in the eax (or rax) register.

So now it’s time to write some assembly (this is NASM Syntax now, which uses the Intel assembly style):

global _my_pow
section .text

_my_pow:
    push    rbp             ; create stack frame
    mov     rbp, rsp

    cmp     edi, 0          ; Check if base is negative
    mov     eax, 0          ; and return 0 if so
    jl      end

    mov     eax, edi        ; grab the "base" argument
    mov     edx, esi        ; grab the "exponent" argument

multiply:
    imul    eax, edi        ; eax * base
    sub     esi, 1          ; exponent - 1

    cmp     esi, 1          ; Loop if exponent > 1
    jg      multiply

end:
    pop     rbp             ; restore the base pointer
    ret                     ; return from procedure

First of all, we have to define the _my_pow label to be global, because otherwise C won’t be able to call it. Note the underscore, since the C convention is to prepend one to all function labels. The “create stack frame” and “restore base pointer” parts are actually not necessary for our use case, since we do not use the stack, but I’ve added them for completeness.

As we’ve previously seen in the assembly generated from our C code, the “base” argument is passed in the edi register, and the “exponent” argument is passed in the esi register, so we start by moving the base to the result register eax, and then multiply eax and the base exponent - 1 times.

Assuming that you saved the assembly under the name asm.s, you can now generate an object file using the following NASM command, which will generate an asm.o object file:

nasm -f macho64 asm.s

After having generated both object files, you can now link them by invoking

clang -o pow asm.o test.o

This should generate an executable called pow, which prints “Result: 256”. For reference, here are my resulting files, including a simple Makefile.

If you don’t care about your history, simple use one of the solutions proposed in this Stackoverflow question.

In said question, there actually is an answer that preserves history, but it has a major flaw: paths from the submodule aren’t correctly updated, thus making the history less useful. Let’s say you have the following repository structure:

•
├── Makefile
├── lib/ (submodule)
│   └── bar.c
└── src/
    └── foo.c

If you try to merge this submodule or use git subtree add without modifications to the submodule, you’ll end up with commits like this:

commit 9f28e08ec8e1ae15823ec7ecb24bdf442a2d3581
Author: Lucas
Date:   Thu Aug 15 10:35:20 2013 +0200

    Do awesome stuff in submodule

diff --git a/bar.c b/bar.c
index b7c5cb8..997e3bb 100644
[...]

As you can see, the paths haven’t been updated, and the commit still points to the path where bar.c resided in the submodule. This is problematic, because git log lib/bar.c will not show the commits made in the submodule. In order to fix this, we’ll need to rewrite every commit made in the submodule, correcting the directory. The good news: git provides a tool for that: git filter-branch attached. The bad news: using filter-branch isn’t exactly straight forward.

But fear not! I’ve written a little bash script that’ll do all the heavy lifting for you, if you don’t want to get into the nitty-gritty details (download at the end of the post).

I will assume that you’ve cloned both your repositories into the same directory, one as parent/, and one as sub/.

•
├── parent/
└── sub/

First, change to the directory of your submodule (sub/ in my example), and execute the git-rewrite-to-subfolder script attached below. It’ll lead you through the necessary steps.

After the rewrite is completed (which may take some time if you have many commits), change into your parent/ repository and delete your submodule:

$ rm -r path/to/submodule
$ vim .gitmodules
    # Here you need to remove the submodule in
    # questions from your .gitmodules file.
$ git add -A . && git commit

Now that we’ve deleted the submodule, we can integrate it into the repository again (still from the parent/ directory):

$ git remote add sub ../sub/
$ git fetch sub
$ git merge --allow-unrelated-histories -s ours --no-commit sub/master
Automatic merge went well; stopped before committing as requested

We’re in the process of merging the history, but still need to add all the content from the submodule:

$ git clone git@domain.com:sub/module.git path/to/submodule
$ rm -r path/to/submodule/.git
$ git add path/to/submodule && git commit

And you’re done. If you take a look at any commit that happened in the submodule, you’ll see that it now points to the correct path, and that git log lib/ will correctly display all commits made in the submodule:

commit e4d5e423d81540b801079b5f644edf730ca477e9
Author: Lucas
Date:   Thu Aug 15 10:35:20 2013 +0200

    Do awesome stuff in submodule

diff --git a/lib/bar.c b/lib/bar.c
index b7c5cb8..997e3bb 100644
[...]

The code

Also available as a gist.

#!/bin/bash
# We need the TAB character for SED (Mac OS X sed does not understand \t)
TAB="$(printf '\t')"

function abort {
    echo "$(tput setaf 1)$1$(tput sgr0)"
    exit 1
}

function request_input {
    read -p "$(tput setaf 4)$1 $(tput sgr0)"
}

function request_confirmation {
    read -p "$(tput setaf 4)$1 (y/n) $(tput sgr0)"
    [ "$REPLY" == "y" ] || abort "Aborted!"
}


cat << "EOF"
This script rewrites your entire history, moving the current repository root
into a subdirectory. This can be useful if you want to merge a submodule into
its parent repository.

For example, your main repository might contain a submodule at the path src/lib/,
containing a file called "test.c".
If you would merge the submodule into the parent repository without further
modification, all the commits to "test.c" will have the path "/test.c", whereas
the file now actually lives in "src/lib/test.c".

If you rewrite your history using this script, adding "src/lib/" to the path
and the merging into the parent repository, all paths will be correct.

NOTE: This script might complete garble your repository, so PLEASE apply this
only to a clone of the repository where it does not matter if the repo is destroyed.

EOF

request_confirmation "Do you want to proceed?"

cat << "EOF"
Please provide the path which should be prepended to the current root. In the
above example, that would be "src/lib". Please note that the path MUST NOT contain
a trailing slash.

EOF

request_input "Please provide the desired path (e.g. 'src/lib'):"
# Escape input for SED, taken from http://stackoverflow.com/a/2705678/124257
TARGET_PATH=$(echo -n "$REPLY" | sed -e 's/[\/&]/\\&/g')


# Last confirmation
git ls-files -s | sed "s/${TAB}/${TAB}$TARGET_PATH\//"
request_confirmation "Please take a look at the printed file list. Does it look correct?"


# The actual processing happens here
CMD="git ls-files -s | sed \"s/${TAB}/${TAB}$TARGET_PATH\//\" | GIT_INDEX_FILE=\${GIT_INDEX_FILE}.new git update-index --index-info && mv \${GIT_INDEX_FILE}.new \${GIT_INDEX_FILE}"

git filter-branch \
    --index-filter "$CMD" \
    HEAD