This post has a protocol for putting DeepLabCut into GPU server production on github.

1. Using DeepLabCut to perform markless body tracking in mice

DeepLabCut is an open source machine vision package developed in Professor Mackenzie Mathis’s lab at EPFL and Harvard to harness two elements of machine vision into one tool. Firstly, they use convolutional neural networks (CNNS) to learn how to identify different parts of an object using image patches drawn from a single video frame. Next, they use that identification to then find all matching parts in a new video frame and extract the 2D coordinates of the matching parts. So for example, after manually marking a few dozen frames with coordinates for a mouse’s nose, tail tip, left and right hind paws, etc. DeepLabCut will quickly begin to identify the correct location of these parts in any new video run through the algorithm (tens of thousands of frames). This becomes incredibly powerful in basic neuroscience research, where tracking animal movements and behaviors can then be linked to activity in the nervous system and can help better define elements of disease. I am a researcher studying the neural circuit basis of Parkinsonism in mouse models of the disease, so accurately tracking mouse motor behavior is critical.

2. Training a good tracking model on a local computer with GPU

To train the model, you first need some representative example data. This requires thinking about all the major sources of variability you might have: different backgrounds? Different lighting? Different objects that might obscure the image? Different subjects? In my case, I had all of these! In addition I also had video from two different resolution cameras and either in color or B&W video. So I made sure to include example frames of each of these conditions to start with.

The experience of labeling frames is a bit mind numbing but can be fun. (Those immediately interested can see what the experience is like by labeling some animal datasets hosted online). DLC comes with some good Jupyter Notebook examples (e.g. Demo_labeledexample_Openfield.ipynb) that show how to extract example frames from one or more videos, and then use the included GUI to label the mouse body parts in each frame.

The process of hand labeling forces you to appreciate the possible difficulties the algorithm might also encounter in the data. In my case, we have a lot of data collected on low resolution cameras, and it could be surprisingly difficult to identify all of the mouse bodyparts accurately using the same criteria for each frame when the mouse was in the far corner, facing away.

Higher resolution images were much easier to mark correctly, especially if mouse was close to the camera and not facing away from the camera.

DeepLabCut is designed for this process to be iterative: you select some frames, train the model, check for frames likely to have mistakes (deviate greatly from nearby frames), put markers on the mistaken frames, train again, etc. The process is very similar for adding brand new conditions to an existing model, or tracking new body parts that were not previously labeled. So the strength of the iterations is that it adds a lot of important flexibility. Side note on adding body parts: make sure that the column order doesn’t change in your training set! That happened to me when I added a new body part to the model and it took a bit of time to realize this was the issue.

The downside of the iterations is that you might need quite a few to get something that works reasonably well across really diverse conditions. I ended up needing about 10 iterations, adding around 50 frames each time to get usable data (495 frames total). But some of those iterations were also to add new body parts or new environments.

Overall the tracking got pretty consistent. It’s not perfect (still needs some pre-processing, covered in Part 2 of this series), but this was a good enough start for my purposes, and took less than a day of my time to get the tracking to this point . In this case, each iteration trained for about 1 million cycles overnight on a single GeForce RTX 2070 GPU.

3. Scaling processing to a GPU cluster for production

So now that I had a reasonable model for tracking body coordinates, I had a lot of videos to process. Processing these data on a single GPU takes close to real-time, so a 30 minute video takes approximately 30 minutes to process. I have +1,000 hours of video from more than 1,000 experiments, which means one pass of the processing would take 42 days. Instead, I did it in 2 days. Here’s how!

Carnegie Mellon University has a close relationship with the Pittsburgh Super Computer, and I was able to get a starter grant funded for about 1500 hours of compute time on their GPU nodes. That was just about all I needed to get through my backlog of videos and test out a production for scaling up processing using DeepLabCut.

The full protocol is available here: https://github.com/KidElectric/dlc_protocol but in brief:

1. I realized that an easy approach would be to put DeepLabCut into a container. In essence, a container is like an entire computer in software form: all the files are in place and all it needs is a place to run. In this case, many places to run (in parallel). The PSC GPU nodes were set up to use Singularity images, which are similar to docker containers. So I found an existing docker image, made a few changes, and found the method for importing it to the supercomputer as a singularity image.

module load singularity
singularity build dlc_217.simg docker://kidelectric/deeplabcut:ver2_1_7
#Then, for example, you can run a Linux shell from inside that image:
singularity shell dlc_217.simg

2. I wrote a protocol for uploading all of my videos to the supercomputer using ssh and uploaded the trained DLC model onto the supercomputer.

3. Three scripts were necessary to make the process work in parallel. The trick was to use the SLUM job array call on the supercomputer–this method automatically sends your singularity image out to as many nodes as become available and starts processing new data. So for that I needed: 1) a SLURM job script to initialize the Singularity images and pass the Job ID 2) a Linux .sh script to start python in the image and pass the job ID along and 3) a python script to run inside the Singularity image to translate the SLURM job ID into a video file to analyze and send that to DeepLabCut to begin processing. It also became useful to write code that checked that all videos were processed correctly.

4. Conclusion

This method allowed me to analyze MANY videos simultaneously. Nodes become available whenever other jobs are finished, at certain points I remember seeing a dozens jobs running simultaneously, and sometimes many dozen! Altogether it took about 2 days to analyze the 1,000 hour backlog. That’s a 20x improvement over running on a single GPU computer.

Would it be possible to analyze 1000 hours of video on a single GPU PC? Certainly. However, waiting 42 days is a big risk to take if you might find out that something didn’t work correctly. And it becomes even worse still if you realize the model needs to improve under a new condition. Lastly, if a lab ever wanted to scale up towards a full pipeline, it would only take a few more lines of code: check the lab server for new videos, send them to the supercomputer! Now that I’ve typed that out, maybe I’ll do that this afternoon ;-).

Next up, now that we have some tracking coordinates, what do we do with them??? Part 2: Pre-processing the data. Part 3: classifying the data. Part 4: verification and final tweaks!

BE: How did this project start?

BI: In 2013, I was taking a neural networks class for my PhD and encountered the phenomenon of “multistable perception,” which I found to be incredibly cool. Some information in the world is so convincing, and yet so ambiguous, that we will actually notice our perception spontaneously switching between two very different interpretations [see example below]. There is something “true” constrained by the alternating perceptions, but that truth is also inaccessible. This idea really resonated with me. I was in a writing group at the time and I started writing poems in the shape of a 3 x 3 grid of lines that could be read both left-to-right and top-to-bottom. Thus, given sufficient grammatical ambiguity, the same text could pivot to allow two perspectives to emerge from the same text.

BE: Basically “The dress” of gold vs. blue fame.

BI: Exactly. It’s obviously blue [laughs]. And also obvioulsy gold.

Does the cylinder turn clockwise or counterclockwise? Ambiguity allows your perception to spontaneously switch how the movements are interpreted.

BE: What did this form allow you to do, poetically?

BI: All experience is open to multiple interpretations, but the grid poems allowed me to experiment with that property explicitly. It allowed me to express and examine the repercussions of interlocking, pivoting subjectivity. For example, these pivots broaden the range of possible identities and experiences each of us can have. But at the same time, the same pivots expose us to miscommunication, disappointment and disillusionment.  This trade-off became a common theme throughout the book.

BE: When did you two first decide to collaborate on this project?

JS: Brian and myself had collaborated years ago on a hybrid poetry and design blog, called Selective Synthesis. As most blogs do, they fizzle out in time, but we had a blast doing it and always knew that we shared a collaborative energy and complementary skill set.

Last July, we both attended a wedding in the English countryside and commiserated how we needed to find a path for making work together again. Brian, shared that he had been working on a series, called “Grid Poems” for quite some time. The moment I heard his concept for this form of poetry, I was captivated. As a designer and Josef Müller-Brockmann proselytizer, the grid is sacred. I was immediately inspired by the potential of how this could work in tandem with a visual system.

BI: All true! I sometimes forget about Selective Synthesis but that was definitely the first design/poetry collaboration we had. I had been working on these grid poems for a while and I knew that the form would really benefit from design considerations that were beyond what I could do myself. John was a natural choice of people to approach.

BE: You have very different expertise and live in different states–was it difficult to collaborate?

JS: Working from NY and CA actually proved to be much easier than I expected. When the only thing you are sharing is text docs and pdfs, it’s easy to review and iterate. Throughout the process we provided constant critique, but ultimately because we are working in different mediums, we allowed the other to express themselves how they saw fit. Lengthy emails became second nature to the process. I’m pretty sure I’ve emailed Brian more in the last year than anyone, ever.

BI: Like John said, it proved to be pretty easy. One thing I worried about was whether John would have time to dive deep into the poetry. Not only did he do that, he gave in depth feedback on every single poem that I found incredibly useful. It ended up being much more collaborative and productive than I expected, to great effect. 

BE: Why did you decide to design and publish this book yourselves?

BI: We wanted complete creative control over the project in order to bring it into the world exactly how we imagined. I think the design of the book was a huge part. Typical publishers don’t like taking risks and use in-house designers. At the same time, there is a lot of marketing and distribution overhead in any successful project–a publisher is a huge asset for these parts of the equation.

JS: Immediacy. We honestly didn’t even seek out a publisher. From the beginning, we had a pretty clear vision for how we wanted the book to be executed, and did not want to wait on anyone else for approval.

BI: We’ve discussed approaching a publisher if the project seems to warrant a second edition.

BE: Had you ever designed a book before?

JS: Never. Quite a bit of trial and error through the process. Constantly printing out versions on the home inkjet. However, I’m incredibly inspired to do it again.

BI: I actually designed one other book previously: a self-help book that my dad wanted to put out there (“Think Right, Feel Right”). That book was a success, so I think that experience showed me the DIY approach could work. It was also good because technical aspects like a book’s “gutter margin,” were already on my radar. This central margin of the book changes size depending on the type of paper you use and how many pages the book requires, making it a common source of DIY error.

BE: What were some book design considerations you had to make that weren’t on your radar before this project?

JS: Initially, the printing on internal pages were multi-colored, but because of cost restrictions, we limited the book to a single-color black. Ultimately, I think this limitation provided a stronger structure and viewpoint for the entire piece. This distilled everything down to be judged on the purity of its form.

BI: The cover cloth and foil-ink selection were new to me. [The ‘foil’ is the title and design stamped into the cloth of a hardcover book.] John created a ton of cover mock-ups but then it turned out we were pretty constrained to using a subset of cloths that could hold the right amount of ink. We also couldn’t justify the cost of a full mock-up book being made so we didn’t know for sure what it would look like until they shipped us all 150. But I was so happy with how it turned out! We relied heavily on John’s design intuition and Conveyor Edition’s patience and expertise.

BE: John, how did you use the poetic form and content to inspire your visual work in the book?

JS: The gridded form of the poetry determined everything. This defined the book’s grid, as well as the exact structure each illustration needed to live on—becoming probably the most painstaking and annoying set of restrictions I’ve ever imposed on myself during a design project. Early in the process, I was “editor” for the individual poems. This allowed me to really dig into the content and many layers each poem possesses. With that insight, I then began illustrating a complementary visual component. Having the poem side-by-side with the visual, allows the context of the poems to drive much of the perception. Each illustration became a meditation on restraint and creating the most reductive, but expressive element possible. This was not an attempt at extreme minimalism, but rather, reducing a concept to a few simple shapes allows for an array interpretations, much in the same way the poetry does.

BE: Will there be a Grid Poems Vol. II?

JS: I really hope so! The intent going into the project, was to build a system which evolve and grow. At this point, we are looking to distribute and publish Grid Poems Vol. I on a larger scale. If this volume is well received, I feel it would definitely inspire a desire to continue making these books. We are infinitely more well prepared now to create a second volume.

BI: Such an ambitious title, right? If we do, I could imagine it being a complete re-imagination: using a new set of constraints, or switching roles so that I write poetry in response to John’s work… I think there are a lot of cool possibilities. Like John said, if this project is successful, I think we would both be excited to brainstorm what a second project would look like.

BE: Thanks for reading about the project and thanks to John for making this Q & A with me! -BE.

During my thesis I built a 1D virtual track that responded to mouse locomotion in order to naturalistically deliver tactile shapes and textures in a controlled environment:

The heart of this technical challenge was building the right running disk. In this post we will discuss how to make a low-weight running disk with high-speed, high-resolution positional readout. Along the way I discuss rotary encoders, quadrature, coding interrupts, and real-time responsive environments like the stimulus wall responding to mouse running shown above.

Edit: And if you end up using the design described below, please consider citing the paper which led to its creation!

1. Introduction: running rodents
2. Constructing a running disk
3. Wiring the encoder
4. What is quadrature?
5. Decoding quadrature on an Arduino Uno
6. Need for speed: closed-loop control

1. Introduction: running rodents

When a rodent explores its environment, it increases sensory information acquisition, and this facilitates primary cortical areas like visual cortex (Niell & Stryker 2010).  In the case of the whisker system, it is well known that whisking frequency is highly correlated with running speed (Fig. 1).

In other words, rodents increase the acquisition rate of tactile information as they increase their running speed. This might be a byproduct of increased breathing rate, or it may be a way of making sure a lot of surrounding areas are still sampled by the whiskers at high running speeds. In either case: it’s pretty cool! Since so many behaviors are synchronized with running, it is often useful to allow rodents to run during experiments.  And it’s even better if this running information can be used to gather position and velocity estimates for analysis and even “closed-loop” systems (more on that later).

Many scientists have implemented strategies for keeping track of rodent running speed and position, including the infamous(e) “Tank Ball” named after Professor David Tank at Princeton (a version of which is featured in this glorious Wired article). A Tank Ball consists of a Styrofoam ball floating on air (think ping-pong ball trick for scientists) with a computer optical mouse on one or two sides for reading ball movements.

While this works great for visual stimuli (as shown above), we had concerns that the air blowing around could interfere with mouse whiskers in their normal movements. They are very fine hairs, and this could introduce unpredictable movements. Thus we settled on a simpler design: a 1D running wheel made of a 6″ plastic disk on a rotary encoder.

2. Constructing a running disk

Back to top

The disk can easily be laser-cut from an online company like Ponoko, or you can do what I did and make it yourself using a Dremel circle-cutter and a drill press (be sure you are trained in using these tools before any such effort!). However, you aren’t done yet: acrylic is too slippery for mice feet, so you will need to add a nice texture. I found attaching a fine polypropylene mesh (McMaster Carr) via epoxy at the edges provided a good solution. As for attaching the disk and encoder, we ordered encoders with precision ground 1/4″ shafts, so I found a matching 1/4″ hub and used a small set-screw for mounting to prevent imbalance (i.e. avoid a large machine screw hanging off the side). The hub I originally used isn’t available but this hub (Sparkfun) should work. Other good resources for these items are: Pololu, ServoCity. The wheel up to this point is shown in Fig. 4.

I later epoxied a thin 1/4″ plastic lip around the edge of the wheel to further aid traction (Fig. 5).

But constructing the wheel is really only half the battle, the next section will describe how we interpret information coming from a running disk / 1D track.

3. Wiring the encoder

Back to top
First thing is to wire up the encoder as per the datasheet. This will depend on whether your encoder is single-ended (one ground) or has differential output (two pins for each output–> this can be converted to single-ended). I used a single-ended encoder with 1000 pulses per rotation, with A and B outputs for quadrature (H5-1000-IE-S, US Digital).

In this configuration, we do not need the indexing pin (which can give an absolute reference pulse, once per rotation). If we want to know when the wheel moves forward vs. backwards, we need to interpret the quadrature. One possibility is to use a chip like the LS7184 chip to do this, but I found it was simpler to implement a real-time quadrature decoder on an Arduino, which I will describe below.

4. What is quadrature?

Back to top
Let’s first take a second to describe what signals a rotary encoder provides!

Inside an optical encoder, you will find a very finely-marked transparent piece of plastic (symbolized by blue disk in Fig 7). By reading light occlusion from two neighboring sensors, different phases of light/dark will occur depending on whether the wheel moves clockwise or counter-clockwise (imagine starting from the arrow and reading left or right in Fig. 7). Thus, whenever either pin changes state, a known increment has occurred. To reveal the direction of this increment, we can check the state of the opposite pin of the pin that changed state. I phrased this in a particular way to make it more clear how we might write code, but to be clear: there are 4 possible states with particular transition paths and these can be used to create a state machine. If this all doesn’t quite make sense yet, draw out the possibilities and how to get to each state for yourself (if you are thorough, it should look something like this). Also, there are many good tutorials, including this one, which I adapted for the decoder described below.

5. Decoding quadrature on an Arduino Uno

Back to top
There are many ways to do this, but the one I liked most was to use 2 interrupt pins: one each for Channel A and B. Interrupt pins will momentarily leave whatever code is currently being processed and call a designated interrupt function. As I alluded to above, one way to implement this decoder would be: any time Channel A (B) changes state, call an interrupt and check the state of Channel B (A). Depending on the state of Channel B (A), increase or decrease a counter (or set a direction flag) as appropriate. Parenthetical letters are to show that we do this for when either channel A or B goes changes state.

To set up the interrupts on an Arduino Uno (there are only certain pins that operate this way), we do something like:

 #define chB  1      // Pin 3 is Int 1 on UNO
 #define chA  0      // Pin 2 is Int 0 on UNO 
 volatile long clicks // Variable used to store wheel movements. Volatile is important
                      // for variables that can change in an interrupt
 void setup() {
   //This can be done here or at a relevant point in void loop().
   attachInterrupt(chA, quadA, CHANGE); // CHANGE = whenever input goes HIGH or LOW on chA, call function quadA
   attachInterrupt(chB, quadB, CHANGE); // And for chB
 }

A illustrative example of possible quadA and quadB functions would be something like this:

void quadA() {
  // If pin 2 (int 0) state CHANGES, this interrupt is called.
  if (digitalRead(chA)==HIGH) { //If this pin (DIO2) is HIGH
    //Check state of DI03:
    if (digitalRead(chB) == HIGH) { //if DI03 is high 
      clicks++; // Wheel is turning backwards/clockwise (mouse running forwards)
    }
    else {
      clicks--; // Wheel is turning forwards / counter-clockwise (mouse "running" backwards):
    }
  }
  else { //IF DI02 is LOW
    //Check state of DI03:
    if (digitalRead(chB) == HIGH) { //if DIO3 is high
      clicks--; // Wheel is turning fowards
    }
    else {
      clicks++; // Wheel is turning backwards
    }
  }
}

void quadB() {
  // If pin 2 (int 0) state CHANGES, this interrupt is called.
  if (digitalRead(chB)==HIGH) { //If this pin (DIO3) is HIGH
    //Check state of DI02:
    if (digitalRead(chA) == HIGH) { //if DI02 is high 
      clicks--; // Wheel is turning forwards / counter-clockwise (mouse "running" backwards):
    }
    else {
      clicks++; // Wheel is turning backwards/clockwise (mouse running forwards)
    }
  }
  else { //IF DI03 is LOW
    //Check state of DI02:
    if (digitalRead(chA) == HIGH) { //if DIO2 is high
      clicks++; // Wheel is turning backwards
    }
    else {
      clicks--; // Wheel is turning forwards
    }
  }
}

Two things become clear from this example: 1) Notice that the state (chA==HIGH, chB==HIGH) has opposite meaning depending on which pin changed. Thus, quadrature is largely about capturing transitions and not states themselves. The quadA() and quadB() functions are exactly inverted, as they capture complementary sets of transitions. 2) Quadrature gets 4x higher step resolution than the listed “Clicks Per Rotation” of the encoder (which is the pulse resolution of a single A or B channel by itself). In this case, the encoder is 1000 CPR, and I used quadrature to read 4000 CPR. Thus we can convert clicks into a real distance using 2*π*r / clicks, where r = mean radius of the mouse’s position on the running disk.

6. Need for speed: closed-loop control

Back to top
To prevent interrupts from disrupting your other Arduino code excessively, the interrupt functions need to be very fast. With higher encoder resolution, this becomes quite relevant, as higher CPR means more interrupts to handle. There are several ways to make interrupts faster:

1) Do little more than increment a counter / set pin states in the interrupt, and address the implications of these state changes outside of the interrupts
2) Use the digitalWrite2() package available here for any pin state manipulations within the interrupt.
3) For an extra boost, consider checking pin states using hardware-specific code as opposed to using the native digitalRead( ) function. The downside to this is that the hardware can no longer port between different Arduino types, so this is generally not recommended. But for my purposes, it ended up being necessary.

In addition to being fast, I wanted to use the same interrupts to optionally yoke the movement of a stepper motor to the incoming quadrature pulses, thus creating a closed-loop interface where mice ran in a 1D virtual track. The combination of these goals leads to a slightly more complex quadA() and quadB() function:

void quadA() {
  // If pin 2 (int 0) state CHANGES, this interrupt is called.
  if ((PIND & (1 << PIND2)) == 0) { //If this pin (DIO2) is HIGH
    //Check state of DI03:
    if ((PIND & (1 << PIND3)) == 0) { //if DI03 is high (c code digital read of DI03 on UNO )
      clicks++; // Wheel is turning backwards & set direction pin state;
      // Set direction pin state:
      digitalWrite2(stimWheelDir, HIGH); // PORTD |= (1<<PIND5); //HIG
    }
    else {
      clicks--; // Wheel is turning forwards & set direction pin state
      // Set direction pin state:
      digitalWrite2(stimWheelDir, LOW); //PORTD &= !(1<<PIND5);//
    }
  }
  else { //IF DI02 is LOW
    //Check state of DI03:
    if ((PIND & (1 << PIND3)) == 0) { //if DIO3 is high  (c code digital read of DI03 on UNO )
      clicks--; // Wheel is turning fowards
       // Set stepper motor direction pin state:
      digitalWrite2(stimWheelDir, LOW); //PORTD &= !(1<<PIND5);//
    }
    else {
      clicks++; // Wheel is turning backwards
      // Set stepper motor direction pin state:
      digitalWrite2(stimWheelDir, HIGH); // PORTD |= (1<<PIND5); //HIGH
    }
  }
  if (syncMove) {
    //Serial.println(clicks);
    digitalWrite2(stimWheelStp, HIGH); // but faster: PORTD |= (1<<PIND4); // (1<<PIND4); //Equivalent to digitalWrite2(stimWheelStp,HIGH); but faster
    delayMicroseconds(stepPulseDur);// 
    digitalWrite2(stimWheelStp, LOW); // but faster: PORTD &= !(1<<PIND4); // Equivalent to
  }
}
void quadB() {
  // If pin 3 (int 1) state CHANGES, this interrupt is called.
  //If DI03 is HIGH
  if ((PIND & (1 << PIND3)) == 0) {
    //Check state of DI02:
    if ((PIND & (1 << PIND2)) == 0) { //if DI02 is high  (c code digital read of DIO2 on UNO)
      clicks--; // Wheel is turning forwards
      // Set direction pin state:
      //Pin 5
      digitalWrite2(stimWheelDir, LOW); //PORTD &= !(1<<PIND5);//
    }
    else {
      clicks++; // Wheel is turning backwards
       // Set stepper motor direction pin state:
      digitalWrite2(stimWheelDir, HIGH); // PORTD |= (1<<PIND5); //HIGH
    }
  }
  else { //IF pin B is LOW
    if ((PIND & (1 << PIND2)) == 0) { //if DI02 is high  (c code digital read of DIO2 on UNO)
      clicks++; // Wheel is turning backwards
      // Set direction pin state:
      digitalWrite2(stimWheelDir, HIGH); // PORTD |= (1<<PIND5); //HIGH
    }
    else {
      clicks--; // Wheel is turning forwards
       // Set stepper motor direction pin state:
      digitalWrite2(stimWheelDir, LOW); //PORTD &= !(1<<PIND5);//
    }
  }
  if (syncMove) {
    digitalWrite2(stimWheelStp, HIGH); // but faster: PORTD |= (1<<PIND4); // (1<<PIND4); //Equivalent to digitalWriteFast(stimWheelStp,HIGH); but faster
    delayMicroseconds(1);// 
    digitalWrite2(stimWheelStp, LOW); // but faster: PORTD &= !(1<<PIND4); // Equivalent to
  }
}

Which leads to closed-loop behavior like this:

That’s about it for running disks! In the future this post will be paired with stepper motor stimulus control…