Evan Kiefl

Northern Mongolia: a journey into the Taiga

2025-03-09T00:00:00+00:00

Day 1: The black market

The text of a nearby ad caught my eye as we waited at the baggage carousel. It was written in two scripts, and to my surprise, neither of them used the Latin alphabet. I could have at least learned the Mongolian phrase for “thank you”, I thought to myself.

We grabbed our bags and found the exit. Amongst the people waiting in arrivals, I saw someone holding a sign that said “Evan and Kourtney”. I could tell it was Anuka, our tour guide, based on the pictures from her website. Thank God she was real.

Anuka fastening a reindeer during the Eastern reindeer herder Autumnal migration (Day 9).

After brief introductions she led us outside to the parking lot. She introduced us to a tall skinny man named Baagii, her husband. We would get to know him very well over the next two weeks. It seemed he didn’t speak any English, but with a shy smile, he motioned for us to hand him our bags.

Kourtney (left), Baagii (center), and me (right) on Baagii's friend's speedboat on Khuvsgul Lake (Day 13).

On our way to Ulaanbaatar, Mongolia’s capital, Kourtney and I calibrated to Anuka’s accent as she shared bits of information about Mongolia and Ulaanbaatar. She talked quietly with a friendly demeanor. I could tell she loved her country and was eager to share it with us. We were in good hands.

“We are a quickly developing nation,” Anuka started. 1.8 million people lived in Ulaanbaatar. Including undocumented people, it might be over 2 million, which is well over half of Mongolia’s total population. Mongolians were contending with a blazingly fast speed of urbanization. Smartphones, 4G networks, TV, 9-5 jobs, fast food, and high-rise apartment buildings were pervading the urban lifestyle, and Ulaanbaatar was the hearthstone of this drastic cultural shift.

As we drove through the rolling hill landscape, littered with thousands of free-roaming cows, goats, and sheep, we crested over a hill and in the distance, we saw Ulaanbaatar sprawled out beneath the backdrop of the Khentii mountains. Along the outskirts of the city, white circular tents called gers (similar to yurts) sat interspersed among modern buildings and city streets.

As we drive toward the city center, we watch the view outside transform from scattered gers to increasingly dense clusters of shacks, eventually giving way to high-rise buildings. Our journey along the road's transect through the city, a symbolic timeline of Ulaanbaatar's rapid urbanization. Photo credit: Hector Retamal

In the not too distant past, nearly 100% of Mongolians lived in gers, and traveled nomadically with their livestock, perpetually moving their gers with the change of seasons, conditions, and resource availability. Today, many gers are equipped with satellite dishes and solar panels, showcasing the conveniences of modernity knocking on the doorstep of ancient tradition.

I was taken aback by the sheer chaos of the traffic. Faded road paint feebly demarcated lanes, traffic lights were sparse, and where they did exist, they seemed to serve more as suggestions than mandates.

As a “fix” to Ulaanbaatar’s traffic problem, the government had instated police-enforced access restrictions at critical bottlenecks in the city traffic flow. This is done by categorizing cars into two groups based on whether the first digit of your license plate number is odd or even. If you’ve got the wrong license plate for a given time, you aren’t allowed to enter the region of the city.

Eventually, we made it to our hotel, which would serve as a launch point for the trip, which started the next day. With the remainder of the day, our plan was: have a nap, buy supplies for the trip, go to a museum, and get a good night’s rest.

After our nap, it was 1 pm and we headed downstairs to meet Anuka and Baagii. To buy supplies for the trip, we headed to the black market. “You can buy anything and everything at the black market”, Anuka explained. Clothes, shoes, electronics, containers, tools, building materials, cabinetry, camping gear, school supplies, plastic, art, horse saddles, livestock, meat, fruit, dry goods—the list went on and on, and included more nefarious things like bullets and presumably drugs. Few needs couldn’t be met at the black market.

(left) A horse accessory market stall featuring colorful saddles with intricate embroidery, braided leather reins and ropes hanging prominently, and horse tack and accessories. (right) An eclectic market stall of Buddha statues, hanging necklaces and rosaries, ornate ceremonial knives and sheaths, miniature statues of animals and creatures, and a variety of leather and metal accessories.

Before long, we were in the heart of the bustling bazaar. Mongolians all around us were buying and selling, trying to score the best deals they could find. Sellers showcased how well their products worked, while buyers, hiding their interest, pointed out identifiable flaws in the product. After successful deals, everyone walked away happy. After failed deals, vendors returned to their smartphones, shaking their heads, perhaps getting in one last verbal jab, while the unsuccessful customers tried their luck with the neighboring vendor, who sold nearly identical products.

Anuka expertly dipped and dodged through the densely packed stalls. To give a sense of scale, it took two minutes of walking just to get out of the shoe section. Exchanging words with the nearest vendors for directions, we finally arrived at the camping section. Kourtney and I bought two sleeping bags, and Kourtney bought a warm fleece. Encouraged by Anuka to haggle, we weakly negotiated, but in truth, since the prices were so cheap compared to our Western standards, we decided the few dollars we could have shaved off the price were better off in the vendors’ hands anyway.

Next, we wanted to bring gifts for the reindeer herders and perhaps any nomadic families we stayed with. Relying on Anuka’s knowledge of the locals, we settled on pencil bags that we stuffed with ornate pencils, erasers, and pencil sharpeners for the kids. For the adults, we bought light bulbs powered by AC battery cables, flashlights powerable via a multitude of methods, battery-powered headlamps, miniature binoculars, and medium- and large-sized thick plastic containers for milking and berry picking. Against Anuka’s advice, we also purchased fifty .22 caliber bullets, which are particularly useful, if not illegal, in the Taiga for hunting.

Afterward, we headed back to the car. On the way, an old drunkard tossed an empty vodka bottle in the direction of a woman sitting on a concrete block. As he shouted, the bottle slid and tumbled on the dirt, coming to a stop near her feet. “Do you want to drink with me? No one can drink more than me,” Anuka translated. The woman didn’t bat an eye, indicating her battle-hardened experience in dealing with drunks. Alcoholism is a major issue in Mongolia, particularly in the city. Anuka offered to us her opinion that the solution is a more stringent allocation of welfare. It made me want to get out of the city.

We spent the rest of the day driving throughout the city, visiting the Buddhist monastery, getting some classic Mongolian fast food, and visiting the Chinggis Khaan Museum.

A Mongolian family beside us in traffic singing along to a traditional Mongolian ballad that blasted from their car speakers. Their baby, standing on the lap of someone in the back seat, hangs his head out of the rolled-down window, staring at Kourtney and me smiling back at him.

Well-fed and sleepy, we were dropped off at the hotel by Anuka and Baagii, and we agreed to reconvene the next day at 10:00 am for the start of our big journey.

Day 2: Journey to the monastery

Now might be a good time to give an overview of our planned itinerary.

Roughly speaking, the plan was to drive deep into Northern Mongolia to the small village of Tsagaannuur. This is where Anuka lives. We would then treat the village as a basecamp, setting off on two distinct horseback journeys. The first to visit the reindeer herders in the Western Taiga, and the second to visit the reindeer herders in the Eastern Taiga. Then we would make our way back to Ulaanbaatar, stopping at some touristic sites along the way.

Details about selecting an itinerary

When planning the trip with Anuka she gave us the reins to create whatever sort of itinerary we wanted. But since she was the expert of her land, it felt more appropriate to let Anuka suggest some sample itineraries, and have us select the one that interested us most. Here is the one we selected:

Day 1: Welcoming day
Day 2: Travel to Amarbaysgalant Monastery
Day 3: Drive to Murun town
Day 4: Off-road drive to Tsagaannuur village
Day 5: Horse riding to West taiga
Day 6: Exploring day with the Western Reindeer Herders
Day 7: Back to the village
Day 8: Horse riding to East taiga
Day 9: Exploring day with the Eastern Reindeer Herders
Day 10: Back to the village
Day 11: Off-road drive to Khuvsgul lake
Day 12: Rest day - boat trip through the lake or hiking to the High mountain (horse riding) which you can see whole pure water lake
Day 13: Drive to Uran togoo
Day 14: Khustai national Park

Add in a spattering of cursory Google searches, and you now know as much as we did going into this.

Our plan for the day was to log some distance in our multi-day goal of reaching Tsagaannuur, and then spending the night by a historical monastery.

We drove through the morning traffic slowly but surely.

The rapid growth of the city was glaringly obvious when looking at the construction of high-rise apartments. For every occupied high rise, there was another under construction. And building contracts were clearly carried out in bulk, whereby as many as 12 identical high rises were erected concurrently, side by side.

Just 15 minutes outside Ulaanbaatar and already Mongolia's natural balance seems restored.

We stopped at a roadside restaurant. The kitchen was within the ger, and outside was customer seating, underneath an awning protruding from the ger’s entrance. The furniture was patchwork, best explained by Baagii’s seat being an extricated car seat.

Anuka explains the menu to Kourtney at a roadside restaurant ger. The food is prepared behind the orange door in the background.

Kourtney got Хуушуур (Khuushuur), a stack of flat, deep-fried meat pastries, and I got Цуйван (Tsuivan), a very popular meat-heavy noodle stir fry. They were both delicious, if not lacking in vegetables.

Alongside our meals, the waitress, a 14-year-old girl on summer break, brought us milk tea, a staple beverage in Mongolia. It’s black tea leaves and salt, brewed in a mix of water and most commonly cow’s milk. I didn’t realize it at the time, but milk tea is more than just a beverage to Mongolians. It’s embedded in their lifestyle, served to guests as a way of welcoming someone into their home, a way to keep warm and hydrated. It’s like their version of “breaking bread”.

Anuka brewing some milk tea in an ort. Whether at a fastfood restaurant in Ulaanbaatar, a roadside restaurant, a family's ger, or an ort, milk tea is always on the menu.

In the car, Anuka gave us our second Mongolian lesson. Yesterday, we learned sembano (hi) and bayarlala (thank you). Today, we learned tawny nerr himbe? (What’s your name?) and mini nerr (my name is). We practiced on Baagii, who somehow struck a balance between laughing at us while also providing genuine encouragement.

At a nondescript location somewhere along the highway, we slowed down, turned onto a dirt road, and left the asphalt behind us. Off-roading would become the new norm for our trip.

After getting turned around once or twice in the labyrinth of dirt roads, the car got stuck during a creek crossing. After an all-hands-on-deck scenario in which Kourtney, Baagii, and I pushed in unison while Anuka manned the wheel, we finally got the car on the right side of the creek. Kourtney remarked that it was the most exciting part of the trip so far. Anuka assured her there was a lot more where that came from.

The lexus gets stuck in a creek just a few kilometers from the monastery.

We finally arrived at the monastery, but it was closed. While Anuka cooked a flavorful vegetable soup on a portable stovetop, Kourtney and I climbed up the hill that had a big golden Buddha statue that overlooked the Monastery.

In the twilight, the moon shone brightly, lighting up the monastery and the surrounding pastures, accentuating the calmness of the valley.

Our evening hike to the viewpoint of the monastery.

After eating dinner, we retired to the tourist camp, where a ger was waiting for the four of us. In bed, Anuka explained a little bit about the simple, yet effective ger architecture.

The design of a ger. The ceiling is supported by a central wooden support with a circular window on top. The woodstove sits underneath this, and the chimney feeds through a segment of the window. From the centerpiece emanates 80 spokes. Layers of plastic and fabric lay on top of these spokes, forming the roof.

Then we all quieted down and let the silence envelope us. I was enjoying this so much more than the busyness of Ulaanbaatar.

Day 3: Journey to Muurun

In the morning we woke up and headed to the car, where Baagii and Anuka were already busy at work. Baagii was doing car maintenance and Anuka was preparing a scrambled egg breakfast with watermelon.

Allured by the smell of our cooking breakfast, a herd of curious cows invades our space around the car.

Three cows eye the breakfast on our fold out table.

When we finished eating, we headed to the monastery, seeing it for the first time in the morning light.

Exploring the Amarbayasgalant Monastery grounds. The monastery was built in 1727-1736 in honor of Undur Gegeen Zanabazar, the first Bogd, Buddhist leader of Mongolia.

Three monks, the oldest no more than 14, emerged from a ger residing within the monastery grounds. At the entrance of the central building they formed a small huddle, their heads almost touching. At first, I thought they were engaged in a prayer or ritual of some sort. But soon I realized they were huddled around a smartphone, watching a video that was making them laugh. The fun cut short by their responsibilities, at the turn of the hour, they opened up the monastery doors, and motioned for us to follow them in.

Three monk children huddle around a smartphone moments before their monasterial duties of opening the monastery.

After turning on the lights and lighting candles and incense, the three apprentices sat in meditative poses around the center of the room and the eldest began speaking his morning recitations into a microphone. As he spoke I explored the beautifully ornate interior full of gongs, statues of gods, gorgeous drapery, and colorful symbology. His monotonous mantra echoed endlessly and unwaveringly throughout the chamber. This is not a performance because we are here, I kept reminding myself, this is their life.

Outside, Anuka explained that many of these children are either orphans or were sent here by their parents because the monastery serves as a refuge from poverty and provides educational opportunities they couldn’t afford otherwise.

It was time to get back onto the road and continue our journey to Tsagaannuur. The plan was to spend the night in Muurun, the administrative capital of Northern Mongolia, north of which one can only find nomads and small villages. So onwards and upwards, into Northern Mongolia.

I woke up to Baagii braking. A weather-seasoned nomadic man was herding his horses across the road with a dirt bike. Anuka and Baagii exchanged some dialogue and then Anuka explained we would go say hi to the nomad and his family.

We parked the car on the side of the road, climbed up the dirt bank, and onto the prairie. The man and his wife were corralling a herd of at least 30 horses, some young, some old. The young ones were tied to a line pegged to the ground. This was to make sure the mothers never strayed too far.

Baagii took the lead and started talking with them. Kourtney and I kept a tentative distance. Weren’t we intruding? But the conversation was going well, and it became obvious they knew each other, so we got closer and started snapping pics.

(left) Baagii makes himself at home while talking to the couple, playing absent-mindedly with one of their horse leads while they milk two horses within earshot. (right) Baagii and the lady by the stable.

Baagii and Anuka spoke with the nomads while they milked all the horses. The procedure was as follows: they released the foal from the rope line, it started to suck on the mare’s teat, at which point the foal was removed, the woman came in with her bucket, and milked. Meanwhile, the man steadied the mother and, if it was a feisty one, he held its leg bent, so it didn’t kick the bucket or his wife.

Holding the foal by its mane, the man allows it to feed in order to promote the mother's milk production. A solitary basketball hoop lies in the distance.

The woman milks the mare, filling the bucket while the man steadies it while keeping the foal nearby.

(left) Kourtney milking the mare. (right) Hitching a ride from Handa's husband.

Every two hours they milked the horses, Anuka told us. It was unendingly exhausting work, and you could tell that to them it was clockwork. They processed the dairy in multiple ways, ate what they could, and sold the rest at market, Anuka continued. In the traditional nomadic lifestyle, summertime was a season of preparation. To survive the upcoming winter, that meant no meat was to be eaten. With little going on agriculturally due to the intense dry climate of Mongolia, that meant a primarily dairy-based diet.

“Таны нэр хэн бэ?” I asked casually, with my barely noticeable accent. Just kidding. With my heart racing, after practicing three times in my head, I managed to squeek out the phrase: “TaWny neRr HimbE?”. She stared at me blankly, probably deciphering what the fuck I just said, then, after figuring it out, snapped back to reality and responded (in Mongolian), “[My name is] Handa”. Handa was short, stout, and had deep red cheeks. Anuka explained to me that her dark rosy cheeks were a product of being outside so many hours a day. When winter came, her skin would turn even darker due to a dramatic change. Not only would the intensely cold and dry climate suck the blood out of her face, but apparently her diet would shift to primarily meat. Anuka looked me straight in the eyes and told me with a very serious tone, “You cannot imagine how much meat we eat in winter.” Apparently, this dramatically affected their body.

After some language-transcending humor, including Kourtney being asked to milk the horses and me catching a ride on the back of the man’s dirt bike, we were invited to their home, a white ger around 100m down the hill. As we sauntered down, I inquired about how Anuka knew these nomads.

“I’ve never met them, and neither has Baagii.”

“Anyone is welcome to stay with anyone 24 hours today and they are treated like family”, Anuka explained after seeing our surprise. In our culture, you don’t just walk onto someone’s land and expect the red carpet to be rolled out. But in Mongolia, no one owns the land. Perhaps this can in part explain a fundamental cultural difference. Since there is a shared conception of the land, this may have a profound influence on the interrelationships between Mongolian people and how they congregate. Kourtney also pointed out that because life is so difficult here, people need to band together in a sort of “we’re all in this together” mentality. In other words, people rely on their literal and proverbial neighbors. Fostered over millennia, this generosity and hospitality has penetrated deep into nomadic culture and persists today as an overwhelming welcoming attitude to strangers. This is just a theory.

Before entering the ger, I noticed some fermenting cow milk being pressed down by two large rocks. Anuka explained that they were making a sort of hard, thin cheese. In fact, some was already fully processed and baking in the sun. Anuka and I grabbed a chunk to share. It was very dry, sour, and salty. It was an intriguing taste, far outside my realm of familiarity. I didn’t ask for seconds.

(left) Horse milk in the process of being turned into cheese. The milk is being pressed in sacs of cheese cloth while baking in the sun. (right) The finished product which tastes very chalky, sour, and salty.

Inside, we sat on some stools. To the right of the entrance was a bed that Handa was sitting on. At the far end of the entrance, the man, whose name I asked but could not pronounce, was sitting near a small table. They laid out some bread, surcream (a thick, creamy, and slightly tangy butter), and candies.

Near the man was a highly decorated cabinet with a telephone, some seemingly Buddhist items, and a mirror. My eyes were drawn to the left side of the ger, where a very large sack hung from a strong beam of wood. It was made of an old, large animal hide that had been stitched together in two pieces. It looked like it had a capacity of at least 100L. Handa walked over and poured her newest batch of mare milk into the sack. From the sound of the pour, I could tell the sack was nearly full, presumably with more mare’s milk.

Inside Handa's ger, a large cow-skin hide has been sewn into a vessel for fermenting horse milk into kumis. It hangs from a sturdy wooden frame that leans against the ger's interior wall. Hundreds of liters of milk stretch the leather to its limit, exerting tremendous force on the two small wooden toggles that solely hold back a catastrophic milk tsunami.

After giving the contents a quick stir, Handa filled a plastic jug by dunking it into the sack, and then placed the jug on the table. Wasting no time, the man procured a decently sized wooden bowl that he filled to the brim, and then took a long pull. After wiping the milk from his mustache, he replaced what he had drunk by pouring more into the bowl, and then passed it to me. To me!

Remembering what Anuka had told me previously about receiving with my right hand, I tentatively took the bowl and looked inside. The sides of the bowl had small chunks of rancid dairy, and on top of the liquid was a thin film that I had watched the man gently blow to one side of the bowl before sipping (gulping).

Everyone hushed to watch what would happen next, with curiosity and anticipation. How would the foreigner react to the taste? Emulating the man as closely as possible, I blew the film off to the side and touched the bowl to my lips and started drinking. Sour and bubbly, it tasted nothing like I expected. Sort of like kefir. I think I liked it! I had several gulps and passed back the bowl, and the ritual continued. The man filled the bowl, looked at someone, and that special someone would receive the bowl and sip from it, then pass it back.

I asked Anuka what I was drinking. “This is Mongolia’s national drink, kumis,” she explained. When offered kumis, it is a sign of respect that you must not refuse, and the more you can drink in one sip, the better. But never drink the whole bowl, she continued. Perhaps it’s just a superstition, but supposedly this is a guarantee for a bad milking yield the next day.

Handa and her husband were very interested in our situation. Anuka translated everything as the questions rolled in: What is your itinerary? Why did you come to Mongolia? Do you think it’s weird we drink from one bowl? I explained that we were in Mongolia to learn about cultures that are different from ours, so we can be more mindful about which cultural practices we engage in within our own culture. It was a complex concept to translate, but I could tell that whatever Anuka said resonated with Handa, as she paused her normally talkative and smile-filled demeanor to listen and nod.

I told them they were very friendly and explained that from our countries, this level of hospitality would require years of knowing each other. How bizarre must that seem to them?

Handa’s youngest son came in and shyly sat down beside her. He loved Michael Jordan and had a Michael Jordan jersey. After learning Kourtney was from Chicago, Handa said he wanted to write a letter that we would give to Michael Jordan. All the adults laughed at the impossibility.

Depending on the strength and stage of fermentation, kumis can be as alcoholic as a light beer.

We needed to get back onto the road, so we said our goodbyes. Handa gifted us a big bottle of kumis, not expecting or asking for anything in return. But before leaving, we gifted them 20000 MNT as a thank you for the kumis and their hospitality. At first, she declined, so I handed it to her son. They stood outside their ger and waved to us as we drove back onto the road.

We drove for another couple of hours until we made it to just outside Murun, where Anuka had arranged for us to spend the night in a tourist camp. As we approached our ger, a very loud and drunk man stumbled out of the neighboring ger and Anuka immediately turned around. “You will have a very bad night. You will not get any sleep,” she said. Back at reception, she threw some assertive words towards the employee, and before we knew what was happening, Kourtney and I were spending the night in a small, one-room cabin in a more secluded section of the camp. One of many instances where Anuka had our back.

After a game of killing all the flies, we settled into the calm of night, far away from the ruckus of our neighbors (and flies).

Day 4: To Tsagaannuur

Today would be our biggest driving day, with our plan being to make it all the way to Tsagaannuur. This would not be as simple as following a road–the main issue being the lack thereof.

Before the beginning of the end of the road, we made a quick pitstop into Muurun so Baagii could get the car checked by a mechanic.

We drove on the only road out of Muurun for about 20 minutes. At a nondescript portion of the road, Baagii veered left onto a dirt road. And that would be the last asphalt we’d see for hundreds of miles.

'Dirt road' would be a generous term to describe what we were driving on. It was essentially a conglomerate of tire tracks through terrain suitable for a car ad. There were river crossings, tire-puncturing rocks littered like landmines, worn out ditches and trenches filled with mud. The tire tracks split and merged over the uneven terrain like capillaries, their patterns serving as remnants of drivers' decisions past.

Baagii navigated the obstacles expertly. In the rare instance that an oncoming car was sighted, the windows would invariably roll down to discuss conditions: Which ways are muddy? How high is the water level of the upcoming river? Is the bridge passable?

We started up a mountain hill and after some discussion in Mongolian between Anuka and Baagii, we turned around, deciding that the path would be too muddy given the recent rainfall. As a Tsagaannuur local, Anuka has traveled to and from Murun many times in her life. “There are many paths to Tsagaannuur”, she said before adding with a smile, “And you can of course always make your own path”.

Cut to 20 minutes later and I was clutching the overhead handlebar with unexplainable strength to avoid bouncing out of my seat as the car barged forward. The GPS sat unused while Anuka directed Baagii, pointing to distant landmarks and presumably saying things like “turn left at the three-peaked mountain,” or “follow this dry riverbed until the standing stone.” The more remotely we traveled, the more ancient knowledge proved to be infinitely more valuable than the GPS.

The relentless pursuit of perfection. Lexus.

After around an hour on the road, the landscape was beginning to change. Away from Murun and its sought after asphalt, the true expansiveness of Mongolia was beginning to reveal itself. Ancient mountains covered in nothing but short grass and rocks. And interestingly, no matter how far away we drove from “civilization”, we saw gers scattered everywhere.

A typical landscape in northern Mongolia. Scattered and free-roaming livestock graze the green rolling hills. The white canopies of gers are sprinkled all throughout each valley. Evergreen forests blanket the distant hillsides.

The “road” was taking a toll on the Lexus, leading Baagii to make frequent stops to cool the engine. (It goes without saying that cars don’t last long here). During one of these stops, we were approached by a man walking alongside his horse. Baagii started talking with him, and we did our ritual of asking what his name was (pretty much the only thing we know how to say), and then unsuccessfully repeating their name back to them. Before long, the gentleman’s friend pulls up on his motorcyle, in masterful style. They wanted to drink with us.

Twenty minutes later we stopped to eat lunch while Baagii fixed the car suspension yet again. Anuka prepared a pasta with fresh veggies and rehydrated beef. Rehydrated meat is amazing. It’s light, doesn’t expire, compact, easy to rehydrate, and delicious. When Mongolians kill an animal, usually in November, they often dedicate two legs just for dried meat. By cutting it into thin strips and hanging it for winter, it’s ready to eat the following April.

During lunch, Kourtney asked if Anuka and Baagii ever disagree about which way to go. Anuka said they frequently bicker about which way is best, and when the car ends up stuck in the mud there is always hell to pay for the person who was wrong. “Fighting is in every culture”, I said, which Baagii found especially funny. The friendship was growing.

Rolling hills cloaked in evergreens cradle a few grazing horses.

Unmonitored livestock grazing as free animals. Four different nomadic families in the background.

Hundreds of sheep and goat litter a hillside, free to roam and graze.

Later in the day we found ourselves enjoying the scenery as we drove alongside a gorgeous mountain. We crossed a muddy river, and then ascended up a rolling mountain. We arrived at a fork in the proverbial road. The car stopped. There were two potential ridgeline crossings. To be honest, both looked exposed and steep. I looked at the left pass and imagined the Lexus barrel-rolling down the side of the slope. After some hemming and hawing, Baagii opted for the rightmost mountain pass, which my imagination was thankful for. We let the engine cool and drank some kumis. It was a hold your breath moment with slopes to our left and right. When we successfully mounted and traversed the ridgeline, everyone was relieved and smiling.

After the mountain pass, we descended into a valley, where a small village lay. It’s a common place to stop for food for passersby traveling to and from the remote north. Day or night, the restaurant will open for a bus full of travelers. An old man sat in a security booth manning a rudimentary car gate that used a defunct chainsaw as a counterweight. A few words were spoken and the conversation ended with a za from both parties. He released the drawstring and the chainsaw pulled the gatekeeping log upwards, allowing our passage.

We were in the middle of Autumn and the more north we drove, the more the grass and trees were showing a spectrum of Fall colors. “It will be even better when we return”, Anuka said.

We were slowed by a herd of about 40 horses who were blocking the road, their owners nowhere in sight. I can imagine no better life for captive animals than in Mongolia, where they live free and nearly wild. Seeing all the horses inspired us to stop for a quick kumis break. As we sipped, Anuka, as she often did, spoiled us with a little tidbit of information. Apparently, Chinggis Khaan proclaimed that a proper Mongolian should drink more horse milk than water.

A curious foal explores, but not too far from its mother's side.

Horses are branded to establish ownership. More on that later...

At the top of a mountain was a collection of 13 ovoos. An ovoo is a shrine composed of branches, rocks, and multicolored silk fabrics, each color representing a natural element (sun, sky, water, etc.). They are typically found at the top of mountains and mark border crossings, and serve as checkpoints either in the beginning or the end of a journey. I couldn’t make heads or tails as to whether this was the beginning or the end of our journey.

Following Anuka’s example, we grabbed three rocks and made our way around the central ovoo, walking three laps in a clockwise motion. While you loop around the sacred space, it’s customary to throw the rocks onto it, adding to its stature. After that, you are to send a wish out into the universe. Since this ovoo site has so many ovoos, we move to the ones linked with our respective Chinese zodiac sign and repeated the process.

After making a wish, I watched Baagii pour out 3 cups of kumis, flinging each skyward. Finishing his ritual, he looked my way and noticed I had been watching him. He smiled. Back in the car, he spoke with Anuka and she translated to us that he was sharing his kumis with nature. It gave me pause. Baagii’s spiritual connection to the land is palpable. He believes in these practices. This is how he lives.

The full journey from Murun to Tsagaannuur is around 9 hours. Understandably, improved road conditions drive much of the election cycle. Candidates visit all the villages by helicopter, and then sing about promises of road infrastructure. Famous people are ushered in and perform concerts, and free vodka is handed out to restore voter faith from the last election. But post election, the promises of roads fade like the distant hum of their departing helicopters.

(left) An average road. (right) A rickety bridge crossing just outside Red Mountain Village, a small village that neighbors Tsagaannuur.

Perhaps it is for this reason that motorbikes reign supreme in Mongolia. They are cheap on gas, cheap to purchase, easy to repair, and can traverse roads and bridges that cars struggle with. With the introduction of the Chinese motorcycle in the 1990s, and its widespread proliferation throughout the country in the following decades, horseback riding has greatly diminished. Increasingly so, Mongolians are traveling, transporting goods, carrying out errands, and herding by motorcycle rather than by horseback. It’s an example like many others found in Mongolia: modern technologies threatening to overwrite a lifestyle maintained for thousands of years, and the Mongolian people and government trying to strike a balance between modernity and tradition.

Past Red Mountain Village we saw some activity outside a cluster of three gers. The men and women were saddling up horses, and someone was drinking. “They are going to race horses”, Anuka determined. “But we can’t go. If we do, we can’t leave. Evan not be able to walk, and Evan no translator”. After a moment, I understood her riddle: if we visit them, we’d be culturally obligated to drink the night away and we wouldn’t make it to Tsagaannuur. We drove on.

A tranquil mood envelopes us as Baagii drives towards the darkening horizon. The car squeaks with each passing bump as we're thrown from side to side. In the twilight, an adolescent herds her family's sheep on motorcycle towards an out of sight ger.

Thirty minutes later, after a journey that seemingly never ended, we made it to White Lake. This is the lake upon which Tsagaannuur village was formed. Anuka explained that no one swims, bathes, or washes their clothes in the lake out of respect for its purity. This respectful relationship with the lake allows Tsagaannuurians to drink its water and maintain a healthy ecosystem that teems with a famous “white fish”, known throughout Mongolia for being tasty. Though Anuka noted that high winds are often blowing roadside trash into the lake.

The entrance to Tsagaannuur. Anuka's father is the artist who created the metal reindeer statues.

In the moonlight, we arrived at Anuka’s childhood home. We were greeted by her parents and Anuka’s precious little girl. Just 18 months old, she squealed with energy at the reunion. After some brief introductions, Kourtney and I retired into an offshoot of their home where Anuka’s mother used to operate a foreign guesthouse. It was during Anuka’s childhood where she interacted with foreign guests that her ideas of becoming a Mongolian tour guide started forming. Without taking our surroundings in with any great detail, Kourtney and I clambered into our beds and fell asleep nearly immediately. What a day.

Day 5: Horseback to the Western Reindeer Herders

The shower of Anuka's childhood home. Since there is no running water in the village, showering is quite the event. An open water basin sits on the roof that can be filled with water from the lake (hence the ladder). This basin feeds into an electric water heater that's bolted to the ceiling of the shower room, that can be powered by running an extension cord from the house. A simple spigot controls the water flow. This complexity, coupled with bone-chilling temperatures in winter, means that showering is a weekly, not a daily occurrence.

It was September 1st, the first day of school in the village. After breakfast, we headed to the school to check out the special occasion. As we approached, we heard speeches and singing performances emanating from the center of the village through shoddy speakers. Within the school grounds, parents and children had gathered around the makeshift stage: a square of concrete embedded in the otherwise grassy school grounds. Here, songs and ceremonies were taking place. A teacher accepted a Teacher of the Year Award, a 6-year-old child sang a song about September 1st, the village leader asked for donations. Children and parents swayed to the rhythm of the music during this joyous day.

A schoolgirl sings a welcome back to school tune. A banner celebrating the school's Tsaatan pupils hangs in the background.

The pupils ranged from 1st to 12th grade. Independent of age, the boys wore ties and white shirts tucked into dress pants, and the girls wore pleated skirts and cardigans. About half of the students lived within the village, whereas the other half lived in nomadic families. Throughout Mongolia, villages with schools host students coming from the nomadic families who live in the surrounding mountains, plains, and valleys. However, unlike other villages, Tsagaannuur also hosts children of reindeer herding families in the Taiga.

Whether their families are classic nomads or reindeer herding nomads, the nomadic children are boarded in a partially government funded dormitory that resides on the school grounds. It is the only brick house I saw in the village. Beside it is the old dormitory, a single story wooden building. This is the dormitory many of Anuka’s classmates lived in when she was a child. The living conditions have improved greatly between old and new. There are dormitory teachers who take care of the children, a cook, who feeds them 3 times a day, and a custodian who keeps the fire stove burning from October through May.

(left) The new government-funded dormitory on the school grounds. It's the only two-story brick house in Tsagaannuur that I saw. (right) The old and new school toilets. When Anuka was in school, the toilets were just a pit with planks stretching across to stand on. Now they have seats to sit on.

It was obvious that education is extremely important to the villagers, the nomads, and the reindeer herders.

“Are the nomads or herders sacrificing their culture by sending their children to school?” I asked Anuka.

“No. They stay with their family for the first 6 years of life. After this, they already know the lifestyle”. Her response illustrated to me how capable children can become when thrust into such a hard-pressing lifestyle. They are capably riding horses by 4-5 years old, and are already taking over familial responsibilities such as goat and sheep herding. Meanwhile in North America tire swings have been removed from playgrounds.

After around 40 minutes, during which time Anuka caught up with all the village folk, we stripped ourselves away from the festivities of September 1st. A big day lay ahead of us: we were heading into the Taiga to meet some of the western reindeer herders.

20km outside the town we met our horse guide, Lhagwa. He’s a nomad with hundreds of livestock, including many horses–some of which he uses for horse trekking services. After waiting a bit at the meetup spot, Lhagwa showed up on one of his horses and ushered us to meet him at an alternative location. At the alternate spot, his wife and a handful of horses were hanging out.

(left) Lhagwa smiling through the antlers of a massive bull reindeer (Day 9). (right) Lhagwa's wife sitting down with the soon-to-be baggage horse, sucking on a candy.

As we geared up for our journey into the Taiga, there was a natural rhythm to the preparations. Anuka boiled drinking water, Lhagwa and his wife saddled the horses, and Baagii packed the saddlebags. Kourtney and I helped where we could but felt relatively useless. Soon, some of us sat down in the grass and enjoyed a snack.

Baagii helped me into his deel, showing me how the buttons and silk sash work.

Kourtney and I in Anuka and Baagi's deels that they lent us for the trip. Deels are a time-proven garb that's ideal for nomadic life. They shield against the sun, they're insulative when it's cold, and they protect your body from branches when on horseback through forested regions. Moreover, a natural pocket is created between the two flaps and above the sash, which can store a surprising amount of stuff.

Baagii (left), Anuka (center), and Lhagwa (right) load up the luggage horse, who is responsible for carrying all of our luggage and equipment. Strong and intelligent horses are chosen for this job because the packed gear rests on either side, and the horse must have the physical awareness to account for this when choosing which path to take through the forest. And if a path is too skinny the horse must be smart enough to back up and find another path instead of freaking out.

When everything was ready, Anuka, Kourtney, Lhagwa, and I set off on horseback heading due North. The herders live northwest of Tsagaannuur and to get there, we needed to ascend up a mountain by horseback. After cresting the mountain, it descends into a flat, marshy bog. From there we would cross the bog, and then descend further into the valley where the reindeer herders were currently settled at their Autumn camp location.

We ascended up the gentle mountain slope and stopped for lunch while overlooking a great view of an expansive bog that lay ahead of us. From far away, the bog looked like a cake walk, but as we would soon learn, it was anything but gentle.

Lhagwa ties up our horses as we break for lunch. The foreboding bog beckons in the distance.

After lunch we saddled up. Being exceptionally tall and of generally low fortitude for Mongolian standards, I was experiencing some knee pain from the stirrups being too short, so Lhagwa lengthened them to the last notch in the leather. Still too short, but it was better. We descended down to the bog and it became clear how treacherous it really was. Extending for kilometers, the bog was a patchwork of soft moss, mud pools, and oily bog water.

As my horse heroically navigated waist deep bog water, I came to realize that we aren’t just riding horses for the hell of it. This ain’t no trail ride. This was in fact, the only way to get to the Taiga. Traversing by foot would be impossible, though technically possible in Winter when land freezes over. Cars are an obvious no, and motorcycles are no better. That leaves horses and reindeer. It was no easy task, even for the horses, who regularly “broke through” the surface layers, ending chest deep in mud.

Throughout this arduous journey for the horses, us humans were enjoying a relatively peaceful journey. The only sounds were the squelching of our horse’s legs navigating the thick bog. After an hour or two of marching, the landscape started to change. The terrain steepened, the ground firmed, and the treeline began to thicken.

We descended further and further into a large river valley. After a quick traversal along the basin of the valley, far in the distance we saw a plume of smoke. As we got closer, we could see a clump of orts. After a four hour trek, we had arrived.

Triangulated between the orts lies more than 30 reindeer. Some young, some old, some white, some brown. The babies are tied to logs, and their concerned parents are close by.

As we came into view, the herders’ dogs spotted us and started barking, running over to check out the new visitors.

We dismounted our horses, and Anuka ushered us into one of the tents while Lhagwa tended to the horses.

Inside we met Sartlong, one of the herders. She kindly gave us some milk tea to drink and was otherwise very shy. After warming up by the stove, Anuka and Sartlong’s husband, Godla, peeled back the animal skin door and entered.

This camp contained three families and three orts. First there was Godla and his wife, Sartlong. Strangely, they met because of a film that was made about a reindeer herder who could catch wild reindeer. They wanted it to be authentic, and Godla could genuinely catch wild reindeer, so he was cast into the role. Sartlong was suggested as the love interest role. Fast forward, they are happily married with a kid, who just left for school. She was staying in the dormitories in Tsagaannuur and would have been at the festivities we had witnessed that morning.

Then there is Sartlong’s brother and his family, who lived in the tent next door. Tragically, their son passed away last month, just a year old. The baby was running a high fever, so they rushed him by reindeerback to the village. Unfortunately, there was no doctor there, nor was there a nurse. So the baby died, and apparently the police were investigating. During our visit they were resupplying in Tsagaannuur.

Finally, a single man lived in the third ort. Since they knew Anuka would be bringing tourists to come visit, he had rented his ort to us for two nights.

Anuka explained that we could ask some questions to Godla, and she would translate.

I inquired, “What do you love about Mongolia?”

Godla responded, “The freedom. In the village, it’s too busy. Out here, there are no rules. We can live with our animals and do whatever we want.”

Godla holding onto his reindeer steed.

Observing how the nomads live, I began to truly understand the depth of his answer. Back in our hometowns, we reside in bustling cities governed by countless regulations.

“Are you concerned about the upcoming winter?”

Before sharing his thoughts, I’d like to provide some context for this question. Every November, the women migrate south, closer to where the weather is milder. This enables them to stay nearer to their children, who attend the village school if they’re of school age, purchase supplies more conveniently, and reach a hospital faster in case of emergencies. Meanwhile, the men venture deep into the Taiga, tracking the reindeer, which thrive in the harsh winter conditions. They remain here with the reindeer, either in solitude or alongside others if their reindeer herds happen to converge. Generally, reindeer are not herded; the only reliable way to manage these particularly wild livestock is to tether their young near the campsite.

Anyways, Godla confidently answered the question about his concerns for winter, stating that he wasn’t afraid at all—in fact, he was excited. He added that if he misses his wife, he’ll return after a month; otherwise, he’ll extend his stay. They exchanged knowing glances and shared a laugh.

“How has technology changed since your father was a herder?”

“So many things. We have TVs to tune into the weather station (and watch soap operas), solar panels, chainsaws, too many things to count.”

(left) Godla's neighbor's ort is equipped with a solar panel and satellite dish. (right) Inside, Lhagwa and Sartlong watch a soap opera.

“Do you want more technology or do you want to balance tradition with modernity?”

“We want more technology but nothing works here except solar.”

We needed more fire for the stove, so Godla slung a chainsaw over his back and gathered two reindeer. He mounted a saddle on the first and hopped on the second. He crossed the river and disappeared into the adjacent woods, leading the first reindeer with a rope. A few minutes later we heard the chainsaw busy at work, and soon after he re-emerged from the woods with our wood for the night.

Godla crosses the stream back to camp, riding his reindeer bare back. A second reindeer, exhausted by its payload, drags two large logs on either side of its saddle through the creek bed.

To show our gratitude, I chopped enough wood to get both orts through the night.

Ready to chop wood.

Reindeer herding is a life consumed by following–and most importantly–protecting the reindeer, as they are constantly being hunted by bears. Godla and Sartlong had lost five just in the last two weeks.

Sure enough, right as twilight was turning to night, when were back in our ort with Anuka, we heard the dogs barking relentlessly. Then, a gunshot! Rushing outside, we met Godla outside his tent, rifle in hand.

He was looking across the river, where the dogs were still barking. In the tent, Anuka translated that he wasn’t aiming to kill, as it’s against shamanistic religion to kill it unless it threatens your life. He showed us his gun, a dated Russian rifle. I realized it would be a great time to gift Godla with a pair of pocket binoculars we had bought at the black market. When I gave it to him, he took it with both hands, touched it to his head out of appreciation, and inspected every feature. He looked up at me, grinned, and said, bayarlala (thank you).

Day 6: Shaman ritual

This morning it was raining so we played a card game called 13 for hours in the ort.

When the rain stopped we went and picked blueberries. Anuka paid for her university tuition by picking blueberries for 8 hours a day for 20 days of each summer. She said 500g could be sold for about 5000 MNT. Near the end Anuka asked if we were bitten by mosquitoes. She said that since we took from the forest, it’s only fair that we give back.

Our diet with the herders consisted of beef ribs that Anuka brought and hung in the ort, bread and fried bread with blueberry jam, milk tea, and noodles, all prepared fresh. Anuka is a more capable cook with limited ingredients, a pot, a wood fire stove, and a river of fresh water than I am with a grocery store and an electric appliance kitchen.

(left) Anuka making a blueberry jam with part of our harvest. (right) A Tsaatan kitchen.

Kourtney and I getting comfortable at camp.

That evening we were told that Godla was going to perform a ceremony. It was a special occasion because he only does them once or twice a month. It is an incredible burden, physically, emotionally, and spiritually. While we didn’t come for a ceremony, we felt honoured that Godla was willing to perform one.

Before the ritual, the evening started off extremely casually. We played cards. We were laughing and chatting. In the finale of one particular game, Lhagwa was coaching Sartlong on a final hand over her shoulder. Anuka, mirroring Lhagwa, was behind Godla’s shoulder, pointing at suggestions as to how he should play his hand. It was husband versus wife. Who would take the prize money? In the end, Sartlong won. Shy as she was, she couldn’t help but laugh. She said she was glad we came because she hadn’t played in months.

At the head of the ort, Godla had prepared his ceremonial stage. Hundreds of torn cloth pieces spanned across the altar, with white pieces on the left, and black and red pieces on the right. Each strip represented a ceremony that he’s done. White strips represented people who he’s helped within the first half of the moon cycle. These are sanctioned, whereas the red and black strips were rituals that took place in the second half of the moon cycle. Such ceremonies are considered taboo, and are traditionally paid for with blood sacrifice–though that itself is now considered taboo.

An incandescent lightbulb hung from above, powered by a clunky portable battery, casting its glow across the sacred space. Beside the stage stood his magnificent drum. Crafted from the skin of a young deer, the drum’s wooden frame had been fashioned from special timber—either a bough struck by lightning or wood from ancient trees deemed sacred.

With Godla turned away from us, Sartlong and Lhagwa started dressing him in a weighted and heavily ornate collage of animal skins, furs, and fabrics upon fabrics. He needed the help of two people to get it on. It was big, voluminous–even asymmetric, giving him the appearance of a hunched back. He looked like a yak with dreadlocks.

On his head he wore a vulture and eagle feather mask and something that covered his eyes. Although he was a shorter man, silhouetted by the dimly lit ort, he looked large and powerful.

Then he started drumming. Softly at first. Each hit generated a deep, muted thud. He moved to and from the altar as he drummed. At the altar he held the drum low and by the center of the ort he held it high. His movement was controlled. The drum beats got progressively louder. He then started speaking in Tuvan. Like the drumming, it got progressively more intense. His voice became strained, high pitched, and mantric. As the ceremony proceeded, he began disengaging with reality. His movements became more erratic and animalistic. Often he would cock his head and turn it swiftly from side to side, the movement exaggerated by his mask’s feathers. The cadence of his voice, distorted. Hissing through gritted teeth he chanted–ever more stressing and ever less human. And throughout all of this, the drum beat continued relentlessly. At times he was completely out of his own control and would throw himself backwards, raising the drum above his head, as if puppeteered by skyward spirits. Lhagwa was behind him the whole time, ensuring he could catch him if thrown back.

I’d never witnessed someone have such an unrelenting and burdensome out-of-body experience before. It was beautiful and terrible simultaneously, holding me in a sort of trance-like purgatory. I think since I’d never experienced anything this different from my own cultural bubble, I caught myself perpetually disassociating from what was happening. “This is what a shaman ceremony is like”, I would think to myself. I’d have to remind myself that I wasn’t watching a re-enactment, or viewing through a TV screen. I was watching a shaman ritual. He was, according to his religion, convening with spirits. And it was happening in front of us.

The ceremony was relentless and lasted around an hour and a half. It was an incredible feat of stamina, to do all of that in heavy and restrictive garb. Even as observers, we found the experience incredibly draining.

Whether you believe in shamanism or not, history has been shaped by these rituals. War decisions of humanity’s largest empire under Chinggis Khaan were decided by shamans. Although Kourtney and I to this day question the extent that our tourism exploited Godla, Mongolians seek his guidance and he delivers his message through similar ceremony.

After a crescendo I almost couldn’t bear, he stopped suddenly, and slouched as if dead. He was dragged outside by Sartlong and Lhagwa, all the while his mantra whispered autonomously from his lips.

No one said anything. I could hear the river outside and for the first time in 90 minutes the world felt bigger than our cacooning ort. I was shell-shocked and speechless. In the wake of what we witnessed, Anuka softly explained that the ceremony was over, and that he was outside deliberating with the spirits. “Didn’t he seem so strong?”, she admiringly whispered.

Five minutes later he returned, not as a medium of spirits, but once more as Godla. We had sought answers to questions we had about directions to take in our currently tumultuous lives, and he divulged the answers he had learned from within the spirit world.

As he spoke, I strained to listen to the sounds of his voice. Mongolian is such a beautiful language. I felt so fragile after the ritual that even without hearing Anuka’s translation, I wanted to cry.

After the ceremony, we went straight to our ort. It was 2am and the night was bitterly cold. Anuka started a small fire for us to fall asleep to. What just happened? As I lay curled in my sleeping bag in the silence of night, the sounds of his rhythmically distorted voice replayed in my mind at a deafening volume, like I was on a powerful hallucinogen. I drifted off into a feverish dream. The dogs started barking far in the distance.

Day 7: The bear strikes again

In the morning Anuka came back with news from the other ort. A reindeer had been killed by the bear last night. Godla and Sartlong had already processed the body. I took the camera, kicked on my boots, and rushed outside. By the river I found what remained of the reindeer.

(left) The reindeer's stomach resting where the animal was dressed. (right) One of the dogs picking at the intesines.

The portions of desirable meat uneaten by the bear hang from an impromptu wooden tripod, out of reach from the dogs. Strands of sinew sway in the gentle breeze.

The reindeer's rib cage partially encases its skull, its eye fixated skywards in an immortal gaze.

The hide is staked taut so a nice leather forms.

Last night was the first day after the full moon, making last night’s event a “blood” ceremony. Did the ceremony cause this to happen? Is this our fault? I asked Anuka. She said it’s possible, but she tried to make light of it that the dogs are happy because they will eat well. I felt so guilty.

But Godla and Sartlong seemed to be in good spirits, unfazed by what transpired. Anuka said that a reindeer costs 1M MNT, but that well crafted reindeer boots, which require the hide of one reindeer, also sell for 1M MNT. It’s better to keep the reindeer alive, but not all is lost. Before, the loss of a reindeer was much worse but now with urbanizing economy, where reindeer boots are fashionable (as well as useful), the loss is not as bad. It is a blatant example of a globalizing world threatening the Tsaatan: their reindeer are worth as much dead as alive.

After giving them some gifts we had prepared, a multi use flashlight, a headlamp, a lightbulb, we departed. To rid myself of guilt, I told Godla I was very sorry for the loss of his reindeer. Anuka translated, “don’t worry, it became sick yesterday”. Reading between the lines, I think he was telling me he doesn’t think it was because of the ceremony.

Me, Sartlong, Godla, and Kourtney.

Godla, the movie star, posing coy by Kourtney's horse.

The trip back to Tsagaannuur was peaceful.

On the journey into the Taiga, Lhagwa rode his horse while pulling the baggage horse with a leash. But on the return trip, because my ineptitude at riding was slowing the group down, this time me and the baggage horse got swapped. How embarassing. And just as I'm feeling most down about it, Lhagwa turns back, points at me with a shit-eating grin, and says one of the only English words he knows: 'baby'.

At the meeting point where Baagii was supposed to be, he wasn’t. But a bunch of reindeer herders were. According to one of them, they were waiting for a group of Mongolian tourists, so we waited with them. A young kid, his father, and his grandpa in law.

The most common forms of transportation: horse, motorcyle, reindeer.

The grandpa was an incredibly small man with a crinkled face and few teeth. He brought over three reindeer and invited us to pose beside them.

Afterwards we gave the kid a gift packet. Kourtney exchanged the Mickey Mouse eraser for a Spider-Man eraser, since he was wearing Spider-Man sneakers. He was very rambunctious and full of energy. We asked him what he wants to be when he grows up. His answer: a reindeer herder, just like his dad. Although I already knew the answer, I asked whether or not he was old enough to ride a reindeer.

With a boastful confidence, the boy articulately commands his reindeer with a dynamic range of rein maneuvring, heel kicks, and a stick he picked up off the ground to whip its ass with. His velcro Spiderman sneakers, the only thing resembling his age.

Then, a car pulled up, and out popped a bunch of people. Anuka recognized all of them. After a while Anuka started to piece it together: Gambut was just joking when he said he was waiting for tourists. He was in fact, just waiting for his family, and here they were.

Commence the chaos. Out of the car they came, pulling and carrying around various supplies and gear. A large rack of meat was transferred from a thick plastic bag into a large rice sack. Saddles, guns, axes, food, and miscellaneous bags with unknown content were all being arranged on the ground. Clearly it was a very successful trip into the village. One by one the reindeer were getting saddled with all the gear. Everyone was working hard and hardly working. Older siblings were taking turns wrestling with the young kid while saddling their reindeer. Everyone was casually chatting and laughing as they readied their gear. In the midst of the activity someone had made a small fire and brewed up milk tea that got passed around with sour chalky cheese pucks. In contrast to every road trip I went on as a kid, they had no apparent anxiety about whether they were leaving on time, whether everything was packed, or whether the front door was locked. They were booming with a confidence gained only through a lifetime of nomadic living, where both everywhere and nowhere is a place to call home.

And when they were ready, they mounted their reindeer and left as casually as they came. As I watched them disappear into the forest, I couldn’t help but feel a pang of envy for their carefree spirit.

Loaded with supplies from Tsagaannuur, the reindeer herders start their journey back to the great wilderness, a place they call home.

The little boy riding on the back of his father's motorcycle back to Tsagaannuur.

Day 8: To the eastern reindeer herders

In the morning we went to the kindergarten. Unlike the jovial ceremony we witnessed a few days ago, this time there was no song and dance. This was the first real day of school.

Outside we were greeted by a happy, plump woman in a short red dress, and her makeup done. She has been the headmaster since Anuka went to the kindergarten over 20 years ago, and it was the 30th year anniversary of the kindergarten. Before that, there was no school in the village, and people sent their children to boarding school at a distant town–or didn’t educate them. Needless to say, the headmaster was in a great mood, and her pride showed in her unwipable smile as she opened the door for us.

However, upon crossing through the door’s archway, we were greeted by terrorized screams of around 30 kindergarteners who had just been stripped away from their families, dumped in a town full of strangers, and here to stay for the foreseeable future. I’m sure the first day of school is tough for most kids, but for many of these children, this was a boarding school, and they knew they wouldn’t be seeing their parents for a long time. Needless to say, they were completely inconsolable, and as horrible as it was, I couldn’t help but smile at the stark contrast between the joyful, celebratory mood of the headmaster and the sudden, overwhelming outbreak of tears and cries from the children.

(left) The entrance to the kindergarten. (right) Four boys playing on an airplane in the kindergarten playground.

Then we went to the store. There again happened to be the kindergarten headmaster. Anuka wanted to buy her gifts, so we agreed to pay for some wine and chocolate, and Anuka gave her the equivalent in cash. She was so happy. Later in the car, Anuka reflected on how grateful the headmaster was. She had had a really tough year. Her husband passed away last Winter. He was a jack of all trades, very successful. And one day he was helping a husband and wife cross a body of water that had frozen over. Since it was some technical offroad driving, he offered to drive the car, and suggested that the wife should get out of the passenger side seat because it was too dangerous. She didn’t, the ice broke, and they both drowned.

The Tsagaannuur store.

After our morning stroll through the village, we headed out for our second adventure into the Taiga, this time to the Eastern Reindeer Herders.

On our way to the starting point, Baagii points out a Mongolian Irish Pub, which is Baagii's joke for a group of Mongolians who, through happenstance, find themselves on a hillside drinking vodka, a not all too uncommon occurrence. Beside them are parked three motorcycles and a horse. And upon further inspection, one of them is Lhagwa, our horse wrangler! Down rolls the windows and Baagii and Anuka yell at him. Presumably something like, 'get your ass to the start point!'

After some practice rounds of 13, where Baagi demonstrated some advanced strategies by pointing out at various groupings of cards within his hand, Lhagwa finally arrived and we began the trek. I had the same temperamental horse that I was riding yesterday. When we stopped for lunch, Lhagwa explained I was riding a sheep horse, which is what Mongolian children use to herd sheep. No wonder he called me baby.

The journey was thematically similar to the first one. Through some rolling mountains, a descent into a valley, and then follow the river to the camp. There was no trail, just suggestions that probably start as animal trails and then are reinforced by mounted horse and reindeer. But the terrain is so spongy and wet that by the time a trail is formed, it’s already a mud pit.

Given this, we established our own way through the marsh, bushland, creeks, bogs, and small timber forests. Once again Lhagwa led the charge with instincts he’s been honing since he himself was a 4 year old child herding sheep.

Lhagwa and Anuka reattaching the bags after they fell off the horse. I take the opportunity to stretch my legs.

Interestingly, there exists a cultural divide between reindeer herders depending on the region they inhabit, either the western or eastern Taiga. In the west, herders like Godla and Sartlong live in small communities, congregating in small nuclear families, or if their reindeer herding patterns happen to coincide. This was in stark contrast to the Eastern reindeer herders, which live in just one or two large nomadic communities.

I’m not sure of the exact reasons, but perhaps owing to the large community that the Eastern reindeer herders live in, they are much more commonly visited by foreign and domestic tourists than their Western counterparts. Tourists are coming all the time. And when guests arrive, they visit whoever they want to. It’s no doubt an over generalization to say that everyone wants guests, but it certainly seemed that way because there is an expectation to bring gifts, and of course to pay for their room and board.

The Eastern reindeer herders Summer camp.

Anyways, we opted to stay with an elderly woman. Her daughter was classmates with Anuka and just had a baby a month ago, so she was all alone, except for her grandson who just finished high school. Her name is Buyantogtokh–or just Buyan for short. She was very nice, immediately asking if we were warm enough, and if we wanted tea. She’s also incredible, being completely self sufficient at managing her reindeer and living in the harsh conditions of the taiga.

Buyantogtokh.

She housed Lhagwa, Anuka, Kourtney and I in her ort next door. Once we settled, she came over for dinner that Anuka made. Inevitably the cards got pulled out, along with the vodka. After several days of playing, I finally won my very first game of 13. Maybe it was those lessons Baagii gave me. Feeling pretty rich with the 5K bill Lhagwa just handed me, I was quickly reminded that Anuka had beaten me the prior round, and so the money really belonged to her. Everyone laughed at my short-lived riches. Before handing it over I decided to take a picture of the bill, serving as perhaps the only proof that I ever won a game. Everyone thought it was hilarious that I took a picture of it.

I went to bed feeling thankful for all the laughs we had shared that night. I felt amongst friends. The crackling sound of the fire was overlaid with Lhagwa’s drunken snoring. The flickering flames illuminated the animal skin walls. The last thing I remember before falling asleep was looking at a bright star through the chimney hole of the ort.

Day 9: Autumnal migration

We arrived at a very lucky time. The herders were undertaking their annual Autumnal migration to their final camp before winter. This happens because the reindeer need to move into the forest, where they can plump up before winter, where there is a lot of grass. Also in the valley where we spent the night, the reindeer travel far and can be difficult to herd, whereas in the new location they stay closer.

Rise and shine. A curious reindeer pokes its head into the ort as Anuka prepares breakfast, before eventually being shooed away.

Morning light outlines a peacefully grazing stag, haloing its antlers in pure white.

Outside Buyantogtokh's ort, freshly cut antlers hang from a tree. Some of them still have velvet fuzz attached to them. It is through this velvet that arteries supply the antlers with ample blood flow, allowing them to grow quickly.

From around 9 AM the community was bustling with activity.

Buyantogtokh leads Kourtney and I to her reindeer milk stash in the forest. Dug under the moss near a slowly moving creek, she looks around, and in a eureka moment peels off a nondescript clump of moss, revealing 40 plastic bottles of milk. Together we bring back her milk reserves to camp where her grandson starts bagging them.

Each herder was deconstructing their ort and placing the entirety of their possessions into plastic, fabric, and leather bags. Solar panels, lightbulbs, mattresses, furnaces, ropes, dishes, antlers, leather, reindeer milk, chainsaws, cutting boards, axes: everything these people owned was to be transported to the their new home.

Bit by bit, the Tsaatan pack up camp. With the canvases covering their orts now stripped away, the entirety of their material belongings now lay in just a few piles next to the frames of their once homes.

A battle worn solar panel lays next to the frame of an uncovered ort.

While some were busy breaking camp, others were herding their reindeer together for the migration. Each and every reindeer had to be accounted for.

(top-left) An elder corrals a pair of her reindeer, as indicated by the branded letter 'A' on each of their torsos. (top-right) A huge stag is led to the staging area. Its antlers have been cut to avoid infighting within the herd. (bottom-left) To discourage endless wandering of the mother reindeer, their babies--which they will not abandon--are often tied down. (bottom-right) A calf gets one last suckle of milk before the migration starts.

Once the reindeer were gathered, a core set of each family’s herd would be saddled for the purpose of either riding or carrying possessions.

(top-left) Two men carefully secure an ort canvas to a reindeer’s flank, ensuring a balanced load on both sides. (top-right) Anuka holds the lead of a small caravan of loaded reindeer, while their owner finishes their final preparations. (bottom-left) A modern solar panel, neatly tied to the reindeer’s pack, offers a glimpse of contemporary influence (bottom-right) A striking white reindeer, adorned with matching white packs, stands poised.

The wood-fire stove, just like the reindeer that bear its weight, is a truly indispensible asset to the Tsaatan.

They tied the saddle-backed reindeer in lines, with the biggest in front, and the smallest in the back. The majority of reindeer, who were not saddled, were herded on the fly by skilled horsemen and reindeermen.

Then the herders took off down the valley in the direction from which they had migrated from a few months prior.

After finishing his cigarette, a gun-slinging nomad mounts his horse, grabs the rope tethering his reindeer caravan, and rides over the hill and out of sight.

A horse contends with entrencheningly deep mud as its rider holds tightly to his tethered reindeer. Compared to horses, reindeer navigate this terrain more easily because they weigh less.

Within a few hours, the camp was mostly deserted, except for a few people who were still waiting for their reindeer to return, Buyantogtokh included.

(left) As she waits for her unaccounted for reindeer to return, Buyantogtokh gathers her remaining, scattered possessions. Her loyal companion follows behind. (right) Lhagwa and I load our gear back onto the horses. It feels good to be trusted by our horse wrangler with the responsibility.

Anuka and Buyantogtokh waiting patiently for Buyantogtokh's reindeer to show up. At this point, almost all of the other families have left.

After hanging out with Buyantogtokh for a bit, Anuka, Lhagwa, Kourtney and I saddled up our horses and followed suit down the valley.

In the marshy valley, our path converges with a herd of 20 reindeer and their owner heading the same direction. For a fleeting minute, while Kourtney frantically gets the camera out, we find ourselves engulfed in the center of their herd, and I feel like I've animorphed into one of them and joined their herd entirely.

The journey took around 2 hours. In the final half hour, reindeer with empty saddles were being led back to the original camp to pick up what couldn’t be carried on the first trip.

We arrived at the new camp and it was chaos. Piles of belongings were piled besides orts, which were all in the process of being erected. All the while, horses, reindeer, dogs, and humans were active all throughout the sparse forest.

From the moment we arrived at the new campsite, until well past dark, we joined the chaos and helped however we could. Primarily, we helped erect orts for newly arrived herders, since it requires the coordination of many hands to properly cover the teepee-like structure with the canvas.

(top-left) Stuff, everywhere. (top-right) Anuka, Kourtney, and Lhagwa assembling the ort for a family that returned to Summer camp to fetch the rest of their reindeer. (bottom-left) Getting full coverage of the canvas over the ort is half art, half science. (bottom-right) The ort construction begins with a central knot that ties 3 logs together. Once the tripod is assembled all of the other logs are leaned against the formed center.

Day 10: Making the rounds

The plan for the day was to hang out with the herders and then ride our horses back to Tsagaannuur.

A very curious calf exploring us and its new home.

Kourtney saying hello.

After breakfast, we made our way around and gave away the rest of our gifts to families that we had met. I really liked how well Anuka knew everyone because all of our gifts ended up being quite personalized.

We met with the tribal leader and his wife, who were very interested to hear I was from Canada. They knew that reindeer were in Canada and asked if we had reindeer herders. I wasn’t sure, but I guessed yes. Turns out I wasn’t totally wrong either. In the 1930s there was an effort to introduce reindeer herding to northern communities, a programme dubbed the Canadian Reindeer Project. It wasn’t as successful as projected, but it’s still practiced today by very small number of people.

Husband and wife.

Then we found Buyantogtokh. She got in late last night but had clearly settled well because she was out and about. We gave her some durable rubber twist ties that my sister bought me for Christmas a couple years ago. We thought it could save her arthritic hands from some of the endless rope tying necessitated by herder lifestyle. We also gave her a waterproof sack.

Buyantogtokh resting easy after another successful migration.

One of the herders came into our ort with a branding iron and placed it in the coals of the wood fire stove. Lhagwa followed in soon after, pointing at the iron with a grin on his face, making sure we understood something exciting was going to unfold. Anuka filled us in: a horse branding was underway.

Four men wrestle the unrelenting chestnut horse to ground. With its front legs tied together, the horse bucks, thrashes, and whinnies as the men gradually restrict its movement. Eventually, after considerable effort from both parties, the horse submits and the branding can begin.

One of the men fetches the branding iron from the coals of a nearby fire, using his sleeve to protect his hands with the heat. While two of the men restrain the horse with taut ropes, Lhagwa and the wielding man drive the iron into the horse's hindquarters. After a second of motionless obedience, the horse kicks hard and unexpectedly, destabilizing the men and freeing its hind from the branding.

The men reconfigure for a second attempt. The horse's owner hovers the branding iron above the horse's hind until it superposes with the partial branding, and then presses in with an ironclad grip, finishing what he started. Exhausted, and with no potential recourse, the horse lays dutifully motionless.

We were planning to leave in the next couple of hours, but we had yet to give away our potentially most prized gift: the bullets we had bought in the Ulaanbaatar black market. As a gift for hosting us in his ort, we gave our host 25 bullets, and as we were deciding what to do with the remaining 25, someone entered the ort to play 13. Then another, and then another. Clearly the news was out that a tourist couple knows how to play. Typically each loser pays 5,000 MNT, but we had the idea to offer either 5,000 MNT or 2 bullets. And then it was off to the races. The energy was high. People were smoking, slamming their cards down, laughing, and most importantly, trying to win some prized bullets. After a couple games, we had to call quits and head back to Tsagaannuur. Since there wasn’t enough time for the herders to win all the bullets legitimately, we ended up divvying the remaining bullets to the players as gifts.

Before we left, our host called us over. To thank us for the bullets we gave him, he had saddled one of his reindeer and allowed us to ride it.

Then as quickly as we came, we were gone. The horse ride back was beautiful, and we went fast. We trotted for at least an hour through beautiful forest. I cantered with my horse, marking the first time during the trip that my horse and I synergized. Back in Tsagaannuur Anuka’s mom made white fish soup. Just like turkey dinner, it’s known to make you tired. It sure did because we were completely knocked out in no time.

Day 11: Sheep dumplings

In the morning Baagii went to buy a sheep because we had run out of meat. We wanted to check it out, so we went out back where Baagii had a sheep tied up by the car.

He dragged it over to the back of the house, positioned it on its back, and with a relatively small knife tried to make an incision through its underbelly. But the knife wasn’t sharp enough so he called over Anuka’s mom who brought a bigger knife. He got through all the wool and made an incision without the sheep making a fuss. Then he maneuvered his hand through the sheep, at which point the sheep started to kick, and then with a mighty force he ripped his bloody hand out. He showed us later, during the dissection, that he had plucked a main artery that runs just below the sheep’s spinal cord. Without any fight, the sheep was dead within a minute.

(top-left) Baagii maneuvering his hand through sheep's innards, searching for the artery that runs below the sheep's spinal cord. (top-right) After plucking the artery, Baagii steadies the sheep in its final moments. (bottom-left) Baagii beckons for me to pass the front right leg through an incision in the upper chest as a way of peeling back the sheep's underside, better exposing its cavity. Yet because I have no idea what I'm doing, I for some reason go to shake his hand. He howls with laughter, nearly falling over. Later, he tells Anuka that he will think of that moment every time he kills a sheep and laugh. (bottom-right) With the innards removed, a large pool of blood remains, which Baagii scoops into a bucket with a ladle.

Baagii's daughter stands quietly behind him, pacifier in mouth, while he processes the freshly slaughtered animal. In Mongolia, there's no separation between meat and its source, and children grow up seeing the entire process from an early age. Observing her father provide for his family is a casual, everyday experience, one that fosters a deep respect for where their food comes from.

We had some left over gifts to give the children, so we made a stop by the school. A group of girls were fascinated with Kourtney’s hair, and indiscreetly whispered to each other, wide-eyed and shy. We found a group of three girls and two boys who all shared a room together. Kourtney laid out the remaining pencil cases that fashioned Disney or superhero characters for the kids to select from. Without any arguing they all picked their favorites. We talked with them for a bit. They were very shy so to break the ice we had Kourtney count to 10 in Mongolian, which conjured some laughter. Then one of the girls counted to 10 in English. We asked them what they wanted to be when they grow up: basketball players, singers, and teachers.

We returned to pack up our stuff. Anuka’s mom gave us some pendants made of reindeer antlers. Anuka prepared us a plate of sheep heart, liver, intestine, and stomach.

(left) A small plate of sheep innards prepared for us, including heart, liver, intestine, and stomach. (right) A bowl of the innards being shared with the neighbors.

This was truly a delicacy amongst Mongolians. Thighs and ribs can be smoked and stored. But fresh innards last for just a few hours. We tried the heart. It was very different but palatable. Then we tried a piece of the stomach which sent me into gag territory. Embarrassed by our foreign palates, we returned the plate mostly untouched.

The sheep meat hangs to dry alongside a clothesline outside Anuka's parents' front porch while an innards feast takes place inside.

Inside, there was an innards boil and the energy was high. Everyone was gobbling it up like Christmas dinner. People were picking up the small intestine and knifing off pieces. Anuka explained to us that the younger generation, especially from the city, considers innards to be dog food, and that it’s uncivilized. But it is a part of Anuka’s culture, and although Kourtney and I found the taste and texture to be disgusting, the practice of making full use of the animal was admirable and beautiful. Equally, I found the experience to serve as powerful evidence that taste preference is entirely relative and dictated by cultural norms.

Afterwards Kourtney and I had a long discussion about our meat consumption habits. In our Western society, there is an increasingly perpetuated disconnection between the meat we buy at the grocery store and the animal who supplied the meat. This disconnect both protects us from questioning the morality of our meat consumption habits and turns a blind eye to increasingly unethical factory farming practices that drive our meat supply. It was eye-opening to observe a way of life where animal and meat are inextricably linked. Where animal’s are given free-roaming lives, never killed prematurely, and where no part goes unused when they are slaughtered.

During the feast, someone came over to borrow Anuka’s mom’s weigh scale. They picked a bunch of blueberries and wanted to know how much. They were invited in, and they gladly took part in the feast. Afterwards, they weighed the blueberries. 58kg at 8,000 MNT per kilo, or 464,000 MNT ($131 USD). Not too shabby. When they left, the gate was left open and three cows came onto the property, happy to have found fresh grass. Anuka shooed them away.

The plan for the night was to visit Lhagwa’s Autumn home near Tsagaannuur. It was a beautiful little one room cabin.

(left) The view from Lhagwa's cabin. (right) Kourtney petting a plump little cat, cats being somewhat uncommon in Mongolia. It seemed that Anuka was slightly superstitious about cats. I wonder if Mongolians think this way in general.

Inside, his mother-in-law prepared some surcream with bread, chalky sour dairy pucks, and of course, milk tea.

Two of the four cabin walls were covered in innumerable horse racing medals mounted on the walls that Lhagwa had won. Clearly Anuka chose the right horse wrangler for us.

(left) A section of the cabin wall dedicated to Lhagwa's esteemed horse racing track record. (right) Wrestling medals earned by one of Lhagwa's sons.

We ate some surcream and bread then wandered outside to catch a view of Tsagaannuur and Lhagwa’s animals.

(top-left, top-right, bottom-left) Happy, free-roaming livestock enjoying their lives. (bottom-right) Kourtney and Anuka messing around in typical, unhurried, nomadic fashion. Tsagaannuur in the distance.

The hindquarters of the sheep Baagii killed in the morning being hung and smoked by a dung pie. Once sufficiently smoked, the meat can be preserved for around a week unrefrigerated. In Mongolia, and especially in the Steppe, where trees aren't bountiful, it's common to use dried livestock poo as a source of fire. Nearby, a sheep belt hangs over the handlebars of a motorcycle.

Last spring was a terrible spring, with bitter winds and no grass. Mongolia had 72 million livestock. Now Anuka thinks it could be less than 60 million. When there is such profound levels of death, it’s too much meat to eat, and the quality of the meat is so poor, so herders either bury the bodies or cut open their abdomens and let the vultures and various other birds dissect the carcasses. Lhagwa, with his 400 livestock, suffered great loss during this tough year, losing about 200 goats and sheep, Anuka predicted. This uncertainty comes with the territory of being a nomad. You can’t put your money into a bank, it roams freely.

After a few hours Lhagwa called. It was getting dark and he needed some help taking down some tourist gers with his wife. So Baagii, Kourtney and I jumped in the car to help.

Lhagwa was loading up two gers into his old Russian truck.

After around thirty minutes the truck was loaded, but Lhagwa couldn’t get the truck started. As he tried, Baagii pointed to the truck and said, “danger car, big danger car” while laughing. Our path forward was a relatively steep slope on wet grass, and so after Baagii’s observation, Kourtney used her body language to convince Lhagwa’s wife to ride with us, rather than in the death-mobile. Eventually Lhagwa got the car into a chortling start, and we followed behind him.

Helping Lhagwa deconstruct a guesthouse ger I assume he manages. The tourist season is winding down in anticipation of the cold Autumn to come.

Lhagwa's UAZ-452, known as a 'Bukhanka' in Russian (bread loaf). This iconic Soviet-era utility vehicle is a modern convenience in Mongolia, with a simple, serviceable design and high ground clearance. A heavy mist hangs in the air, shielding the looming mountainous wilderness from the simple pastoral grounds that the mixed flock in the foreground lounge within.

We followed behind as Lhagwa barrelled across the slope side, slipping several times in what I thought would be a tip, potentially rolling into the lake. His wife, safely in our car, would go “ooooooooo!” and then nervously laugh each time it happened. To cap off the adrenaline rush, Lhagwa undertook a short horse herding adventure by veering brazenly off-course in an attempt to herd a group of his horses towards the direction of his cabin.

When we got back, Anuka was in the middle of preparing sheep steamed dumplings from one of the hind legs. Before, during, and after eating dumplings, we played cards on a fold up table under a flickering hospital white light bulb. I won the first round but Lhagwa reigned supreme for the rest of the night with an impressive winning streak. Must have been home court advantage.

Day 12: Farewell Lhagwa

Our plan today was to make it to Khövsgöl lake, where Anuka had rented a small tourist cabin.

In the morning we said goodbye to Lhagwa, his wife, his mother-in-law, and their 400 goats, sheep, cows, and horses. We gifted Lhagwa and his family a flashlight setup that can be powered multiple different ways and a pair of pocket binoculars. And inadvertently, my sandals.

Thank you Lhagwa.

Although Kourtney and I shared zero linguistic overlap with Lhagwa, he had become a dear friend of ours over the last week. He guided us safely through the Taiga. We broke bread with him, assembled orts, and disassembled gers. We found the overlap in our humour. He inspired us with his selfless attitude towards hard work that benefits those around him. And what he’s no doubt most proud about, he beat us in cards many times, and laughed hard at the wins and losses.

“How long is the drive to Khövsgöl lake?” I asked.

“250km. The time I cannot say”, Anuka answered while laughing.

A question impossible to answer, no doubt. It was a rainy day, which will affect which mountain passes we choose. We would get further information from cars passing from the other direction. Perhaps this river would be too high, this steep section too muddy. And thus from our available options, our path would emerge. Whatever would come to be, we knew it would be a long day, as even during the rarest and most ideal stretches of land, our speed wouldn’t surpass 30 mph.

Our journey to Khövsgöl lake (I).

Our journey to Khövsgöl lake (II).

After car pushes, countless river crossings, failing power steering and subsequent calls to Baagii’s car mechanic, we stopped to let the car cool down for the 5th or 6th time.

Seemingly every Mongolian is an amateur mechanic, and Baagii is no exception to this generalization.

(top) A nomad rolls up. Seeing Kourtney's camera, he wants her to take a picture. (bottom) When I introduce myself in Mongolian, he looks at me with a mix of bewilderment and amusement.

At 7:30, 10 hours since our departure, we made it to asphalt. Just two minutes away was the longitude-latitude coordinates 50’100’, with an architectural monument pinpointing the spot. Even more incidentally, Anuka’s father, an artist, was commissioned by the mayor of the nearby village to design the monument, and was promised some property on Khövsgöl Lake as his commission. But after the monument was built and before the keys were handed over, the election cycle brought a new mayor into office, who never acknowledged the contract. Kourtney and I hiked up to the spot begrudgingly, while Baagii cooked instant ramen. In solidarity of this grievance, I won’t share a picture, but it was a cool monument.

For the last 45km to Khövsgöl Lake, Baagii, with unwavering concentration, navigated the two-lane highway. By western standards it was shoddy, but after 8 days of roadless mountain driving, it felt like the Autobahn. As the road signs flew past us at a blistering 40mph, I daydreamt about taking Anuka and Baagii to an American amusement park. We arrived at Khövsgöl Lake in the dark and clambered into our sleeping bags.

Day 13: Batsur

Our plan for the day was to hang out at the lake, kick our feet up, and relax.

In the morning we went for a motorboat ride. The boat driver was Batsur, one of Anuka and Baagii’s university friends. He worked hard as a boat driver for 10 years until he could buy his own speed boat. Now he makes good money driving his own boat.

(left) Baagii in his friend's speed boat. (right) Batsur, Baagii and I playing a rock throwing game, in which the goal is to throw up one rock, and then strike it with a second before the first hits the ground.

Khövsgöl lake is a big deal for Mongolians. Visiting the landlocked country’s largest lake is a tourist destination for foreigners and natives alike. And for Mongolians, most who can’t swim, ripping around on a motorboat is a thrilling novelty. While foreign tourism has mostly finished for the year, there was still plenty of domestic tourism in mid September. Lining the shore were mini cottages, houses, and shacks for rent.

After breaking the ice with Batsur with a self-explanatory rock-throwing contest, we got it on our heads that the 5 of us would go to a nearby karaoke bar. To prepare, we went to the shop and bought 15 beers of various Mongolian brands. But after calling the establishment, we realized they closed early, and so we drank the night away in our cottage house. For dinner we had more sheep dumplings, this time prepared by Anuka and Kourtney.

(left) Round two of sheep dumplings, prepared by Kourtney and Anuka. The worst constructed one was made by me. (right) The sheep bone picked clean by Baagii.

The night was a mix of laughter and emotional conversations. I found it very impressive how communicative Baagii was despite having a miniscule English vocabulary. We learned that Batsur and Baagii used to play Counterstrike together at university. And that Baagii once drank 6L of kumis. Jokingly, he claimed that from the center of the ger he was projectile vomiting outside the door.

On a separate note, Baagii thanked us for respecting his culture by participating in the sheep slaughter the previous day. As we drank, Baagii took the remainder of the sheep leg that Anuka used to make dumplings, grabbed a knife, and picked the bone clean. Anuka clarified to us that it wasn’t because he was necessarily still hungry, she said. Instead, it’s because it is both practical an honorable to revere the animal’s sacrifice by letting nothing go to waste. She also said if you are a guest and you similarly pick the bone clean, the owners will be “so, so appreciative” because it means you’re fully respecting their hospitality.

Day 14: Coming full circle

In the morning we drove to Murun to get the car fixed. On the way we visited some ancient deer stones. Just like stonehenge, the meaning of these ancient monuments remains a mystery to this day, although there are some pre-historic traditions in the region that remain steadfast, such as burying Murun people in mountains surrounding these deer stones. I wonder if it’s related?

Stuck behind a huge flock of sheep being herded down a road on the outskirts of Murun, Anuka, Kourtney and I jumped out of the car to visit the monastery to our right while Baagii went to fix the car. We went for a stroll through the monastery. Anuka brought us inside the building where some kind of Buddhist reading was taking place. It was jam packed with people, with no place to sit. Everyone’s staring gaze pushed us back through the doors we came. Anuka thought it was a funeral procession of some sort.

Then we walked to the supermarket where we would meet up with Baagii. Along the way we saw some statues, street art, illegally high apartment buildings, and a congregation of people attending a wedding event. Half the men were in suits, half in deels.

At the market Baagii had bought us gifts. For Kourtney, he noticed her phone screen was broken so he bought her a new screen protector. And for both of us he bought us a deck of novelty cards, where each card was a picture of a famous Mongolian wrestler in their traditional (and revealing) garb. As he placed the cards in my hands he winked and said, “Mongolian playboy”.

Then we were back on the (paved 🙏) road. We decided we would stay with Handa and her husband, the nomads we encountered a week ago, who had invited us into their ger. That doesn’t mean we arranged this with them–our plan was to just show up. Anuka wasn’t kidding when she said Mongolians have no word for “go away”. When we arrived, Handa was just finishing a horse milking session and seamlessly transitioned into hostess mode, preparing milk tea and cookies with surcream, and of course, kumis.

Their kid had left for school, living in a 50 bed dormitory in a village school around 100km away. He was the kid who wanted us to send a message to Michael Jordan. In his place, was Kourtney’s newest obsession: a small pregnant cat gnawing a sheep’s thigh bone.

It felt full circle to be spending our last rural night at the very place where we experienced the greatest culture shock of the trip. A week ago, we had no idea about nomadic life when we were invited into Handa and her husband’s life. Without context, I was handed a bowl of fermented horse milk from a large cow skin full of unrefridgerated dairy product, and eating rock-hard cheese completely unique to my palette. Just moments before that, Kourtney was milking a horse and I was hitching a ride on a stranger’s motorcycle. The experience gave me such a culture shock that I was fighting back a bit of an adrenaline rush.

But this time was different. This time we felt like seasoned veterans of this lifestyle. In the last week, we’d travelled across roadless mountain passes, traversed the treacherous Northern Mongolian landscape on horseback to visit the Tsaatan peoples. We’d had our prophecies told during a life-altering Shamanic ritual, witnessed a bear-attack that killed one of Godla’s reindeer. We’d participated in a seasonal reindeer herding migration, and lived in orts underneath the stars. We’d butchered a sheep, watched a backyard horse branding, and rode reindeer.

So in comparison to all of that, we felt comfortable with Handa and her family. I played a friendly game of 13 while sipping kumis. Kourtney helped herd the baby cows. Then we stirred the newest milk contribution into the massive cow skin bag.

To transform the fresh horse milk into kumis, the batch is whisked vigorously with a wooden stick for an hour continuously. Over this time, yeast alcoholizes and carbonates the milk while Lactobacillus acidifies it.

Although we felt we were fitting in, that’s not to say we’re fit for this lifestyle. I cannot overstate how tremendously hard nomadic life is compared to most Western lifestyles. This is a non-controversial opinion for any “Westerner” who has spent as little time here as we have. Case in point: Handa’s daily summer routine. Every two hours she milks the horses. From 5am to 9pm. And at the end of the day, she needs to mix the new milk with the forever soup of kumis by stirring it vigorously for 90 minutes continuously. This helps incorporate the yeast and kickstarts the fermentation. Needless to say, she was visibly happy that Baagii, Kourtney, Anuka and I took shifts so she could rest her body.

Late in the night I got a bad case of the dairy trots from the kumis that I was so confidently sipping earlier in the night. A further reminder of my un-Mongolianness.

Day 15: Home

On our final day together, we visited Khustai National Park, where we witnessed herds of the only truly wild horses left on earth. We spotted around 40 of them, representing around a tenth of the entire population.

As dusk fell we began our final journey back to Ulaanbaatar.

“What would Baagii do for work if he moved to Canada?” I wondered aloud as we drove, trying to imagine this horseman, herder, and all-around handyman transplanted into our world.

“Butcher,” we all agreed after some discussion. He would excel.

“And he could live with us!” Kourtney and I suggested enthusiastically.

Anuka translated, then added with laughter, “And pick the meat off the bones to eat for free!” referencing our conversation from days earlier. Through the rearview mirror, I watched Baagii’s eyes crinkle with amusement.

The landscape gradually transformed as we approached the capital. Gers and petrol stations gave way to taller buildings and metropolitan development. Neon signs in Korean and Cyrillic punctuated the darkness, advertising restaurants and karaoke bars.

After maybe an hour navigating, we finally arrived at the hotel where our journey had begun two weeks earlier. The symmetry felt right, somehow. We had traveled in a great circle through Mongolia, and here we were all over again.

For our final meal together, we spoiled ourselves to the Korean restaurant attached to the hotel. We sat around the table and shared our food family-style, without the awkwardness that had characterized our first meal together.

Back in Anuka’s apartment, surrounded once again by the trappings of modern comfort, I felt a strange displacement. Part of me still wished to be in the Taiga, still riding across boggy terrain, still waking to the sound of reindeer hooves outside an ort.

I thought about the word “nomad”—how it implies constant movement, no fixed home. Yet the nomads we’d met seemed more rooted, more connected to their place in the world than people with permanent addresses, myself included. What, then, did home mean to them? I couldn’t pretend to know, as I’d only glimpsed fragments. But perhaps the answer can be found somewhere within Handa’s unquestioning hospitality to strangers. Or in how Lhagwa’s family arranged their treasured medals on cabin walls they’d soon leave behind. Or in how the Eastern reindeer herders dismantled their entire camp in one morning. Maybe it was in Godla’s words when asked what he loved about Mongolia: “We can live with our animals and do whatever we want”.

Whatever the case, I realized that Anuka and Baagii had given Kourtney and I a profound gift: they had made Mongolia feel, however briefly, like home to us. Not by chaperoning us through the “must-see” attractions of Mongolia, but by providing us an unadulterated lens into Mongolian culture.

Indeed, as I reflect on this trip now, nearly two years later, it wasn’t the spectacular landscapes or exotic experiences that I regularly think of, but the simple moments centered around people: Baagii’s laughter when I mistakenly reached to shake his blood-covered hand during the sheep butchering. Or the way that Anuka looked up at Godla, admiringly, and in a hushed whisper remarked to me how powerful he seemed. Or the bunch of us in Lhagwa’s cabin, huddled around a card game underneath a flickering lightbulb.

Mongolia revealed itself to us as a way of being proud, generous, and profoundly connected to what matters most. And for that lesson–to Anuka, Baagii, Lhagwa, Godla, Sartlong, Handa, Buyantogtokh, Batsur, and everyone else who welcomed us into their lives–no amount of bayarlala could ever be enough.

Kourtney, Baagii, Anuka, and I share a final countryside meal together, talking about relationships, marriage, and what makes a good life.

If you want to travel to Mongolia, we would highly recommend getting in touch with Anuka. Her website is here and her Instagram is here.

Northern Mongolia: a journey into the Taiga was originally published by Evan Kiefl at Evan Kiefl on March 09, 2025.

Pooltool is alive and well

2023-05-03T00:00:00+00:00

It’s been a while

It’s been almost 2 years since my last update on pooltool. Usually that’s an indication that the project is dead, but I’m here doing CPR. In fact, despite the radio silence, I’ve been working on pooltool alongside what has been an eventful 2 years with finishing my PhD, starting a new job, moving countries, yada, yada, yada. During that time I could never find the energy to give a formal update on the project, but that’s coming to an end right now!

Since it’s been so long, there’s really no hope of giving any satisfyingly detailed report with commit hashes, correlating features with code snippets, etc. Instead I’m keep things relaxed with a devlog update. Please enjoy:

Pooltool is alive and well was originally published by Evan Kiefl at Evan Kiefl on May 03, 2023.

Pooltool just underwent a massive graphics overhaul

2021-10-26T00:00:00+00:00

Backstory

In the last post I turned pooltool from a code-based simulation into a 3D-rendered interactive game. In this post I extend this direction of the project by transitioning from primitive cartoon graphics to something a bit more realistic.

Sitting down to write this post has made me realize just how far down the rabbit hole I am. When I first started pooltool, I never even expected it to be rendered in 3D, let alone with eye-catching graphics. Fast forward to now, and I’ve inadvertently produced a video game that’s almost ready for an alpha release. Honestly, I don’t even know how I ended up here, but it’s been a pleasure to let the project evolve so organically.

Before this post, pooltool was full of primitive, textureless, and ugly assets, and I wanted to change that.

I found assets online, but none of them fully suited my needs. I wanted full control, which meant I would have to learn some basic 3D modeling. And so began my journey into the amazing open source modeling software, Blender.

Cue

To start out, I decided that modeling a cue would be relatively easy because of its simple geometry. After countless youtube tutorials, I ended up with the following model.

Then, at the cost of some casualties, I managed to get it into the scene.

(Browse code)

Table

Taking a big leap in complexity, I started a several week process of trying to model a pool table, which involved many iterations.

As a practice round, I just tried to freestyle something that looked like a pool table, without trying to adhere to specific shapes and dimensions. What I ended up with was not half bad in my opinion.

(Browse code)

With that under my belt, I iterated the process, this time with physically accurate measurements. I thought it would be most fun to use my own pool table as a reference, so I took about 30 reference photos. To get an idea, here is a few of them.

The next day I had an 8-hour flight, which I spent every minute of modeling. By the end of the flight, I had a physically accurate representation of my pool table inside pooltool.

(Browse code)

Soon after that I figured out how to apply some UV textures.

(Browse code)

At this point, it should be kept in mind that the simulation is completely disjoint from the model. For example, the simulation has its own set of parameters that define the cushion geometry, and I’m rendering this geometry using simple grey lines.

As you can tell, they don’t line up that well with the model. To make sure they do, I created a INI formatted file that specifies all of the model’s geometry in terms of the parameters used in the simulation. It looks like this:

[7_foot]
type = pocket
model = models/table/7_foot.glb
table_length = 1.9812
table_width = 0.9906
table_height = 0.708
cushion_width = 0.0508
cushion_height = 0.036576
corner_pocket_width = 0.118
corner_pocket_angle = 5.3
corner_pocket_depth = 0.0398
corner_pocket_radius = 0.0612
corner_jaw_radius = 0.02095
side_pocket_width = 0.137
side_pocket_angle = 7.14
side_pocket_depth = 0.00437
side_pocket_radius = 0.0645
side_jaw_radius = 0.00795

These parameters define the exact cushion geometry for the model. If I add more models in the future, I would simply add their geometrical parameters to this file and pooltool would be able to create the simulation geometry from these parameters.

Now, the simulation geometry matches the model geometry nearly perfectly.

Room

I wanted pooltool to be an immersive experience, so I decided to create a room. I saved this for last because it is the most involved. Thanks to Kourtney’s interior design skills, I think it turned out really well.

Here is an in-game screenshot of the room when it was about 75% finished.

(Browse code)

And here is an interactive model of it fully completed:

If you look around, you should be able to spot Dexter.

This is actually a super realistic pencil crayon drawing we have in our apartment. The artist is Renaissance Pup. All of the other art in the room is in the public domain.

Balls

Then I turned my attention to the pool balls, which thus far have been black blobs.

To give my game a unique feel, I designed my own ball set. The strategy is to first design an image for each ball that will eventually get wrapped onto a sphere. This is called a UV image and here is the collection of UV images I created.

If you’re curious, the font is ITF Devanagari Marathi. Using Blender, these images can be wrapped around spheres which is how all of the ball models are made. For example, here is the 10-ball.

Eventually I’ll have to model more balls for games like 3-cushion. But for now this is good.

Want these files? SVG, PNG, BLEND, and GLB file formats for all of these balls can be found here

Optimizing

After adding the balls in, this was the state of things:

(Browse code)

It looks good, but I was starting to think the table textures looked kind of bad. So I went to ambientCG, a database of public domain physical-based rendering (PBR) materials, and replaced the wood and cloth with textures I thought looked more realistic.

(Browse code)

Then I did the same thing with the floors. Also also replaced one of the walls with brick, and added some trim so the contrast between floor and wall wasn’t so stark.

(Browse code)

I also added some lights under the bar to add a nice ambience.

Performance

With PBR materials, lights, and shadows, performance is starting to become an issue on my poor macbook. So I created a little INI file config/settings (introduced in this commit) that can be used to toggle various graphical settings. Right now you have the option to toggle the room, the floor, lights, shadows, shaders, whether PBR materials are used, and the target FPS.

As much as a like the high quality graphics I’ve been illustrating, I also think the following settings create a great cartoony vibe:

[graphics]
room = 0
floor = 0
lights = 0
shadows = 0
shader = 1
physical_based_rendering = 0
fps = 30

Cue avoidance

Once the graphics started to look quite realistic, I found it no longer acceptable for the cue to intersect geometry:

This unrealistic behavior is in stark contrast to the graphics.

To fix this, I created a routine that runs every frame and calculates the minimum cue elevation required to avoid intersecting any rails or balls. If the cue elevation is less than this, the cue is raised. While all of the collision detection is handled by built-in Panda3D capabilities, all of the collision resolution was handled by my own equations, which almost caused me to go insane, as evidenced by my notebook during this 3 day period.

The reason why I found this so difficult is because the cue stick doesn’t always aim at the center of the cue ball, which introduces several degrees of freedom. Nevertheless, I made it through, and the result looks good and feels intuitive.

All of the changes required for this feature can be found in this pull request.

Gallery

All in all, I think the game looks great now. Far better than I ever had expectations for. Here are some screenshots I’ve taken.

Next up

While pooltool is looking good, the physics engine needs some work. So next up on the agenda is increasing the physical realism.

Pooltool just underwent a massive graphics overhaul was originally published by Evan Kiefl at Evan Kiefl on October 26, 2021.

Pooltool is now an open source billiards video game rendered in 3D

2021-04-17T00:00:00+00:00

Goal

In the last post I implemented a prototype that visualizes shots with pygame. But my plans are much more ambitious than visualizing shots on a pixel-art table. My goal is to turn this simulation into an interactive, 3D game/tool. That’s what this post is about.

My credentials for making a game

None.

Game engine selection

If I was a game developer, I would have written this in C# or C++, using Unity or Unreal Engine as my game engine. But I’m not, so I decided to use a game engine with Python support.

It turns out, there is only 1 option that deserves consideration: panda3d. Fortunately, it’s by no means a bad option. Originally developed by Disney, it was the game engine for the critically acclaimed ToonTown (among a few other titles), and then they later open-sourced it in a collaboration with Carnegie Mellon University for the purposes of using it as an educational tool.

Today, panda3d is a fully-featured, open-source game engine that remains under active development. It has good documentation and a relatively small, yet charitable and welcoming community. I must say, I feel a small sense of home every time I visit the Discourse and in general, I’ve had a wonderful time using panda3d so far.

Getting something going

After successfully completing the panda3d tutorial, I had created somewhat of a masterpiece:

But Michelangelo didn’t stop painting after he completed the Sistine Chapel. After some time, I managed to squeeze a pool table into the scene.

[Browse code]

The animation is the wrong speed, there is no camera controls, and the balls are rendered as little smiley faces. Nevertheless, pooltool is officially 3D.

Next, I started working on a mouse-based camera system. To orient the camera based on mouse movements, I needed to track mouse movements. To no one’s surprise, panda3D has a mouse-watcher object called mouseWatcherNode that can query the mouse’s current position. So I wrote a class called Mouse that uses mouseWatcherNode and a ClockObject to track changes in mouse movement over time:

class Mouse(ClockObject):
    def __init__(self):
        ClockObject.__init__(self)

        self.mouse = base.mouseWatcherNode
        self.tracking = False


    @staticmethod
    def hide():
        props = WindowProperties()
        props.setCursorHidden(True)
        base.win.requestProperties(props)


    @staticmethod
    def show():
        props = WindowProperties()
        props.setCursorHidden(False)
        base.win.requestProperties(props)


    def track(self):
        if not self.tracking and self.mouse.hasMouse():
            self.last_x, self.last_y = self.get_xy()
            self.last_t = self.getFrameTime()
            self.tracking = True


    def get_x(self, update=True):
        x = self.mouse.getMouseX()

        if update:
            self.last_x = x
            self.last_t = self.getFrameTime()

        return x


    def get_y(self, update=True):
        y = self.mouse.getMouseY()

        if update:
            self.last_y = y
            self.last_t = self.getFrameTime()

        return y


    def get_xy(self, update=True):
        x, y = self.get_x(update=False), self.get_y(update=False)

        if update:
            self.last_x = x
            self.last_y = y
            self.last_t = self.getFrameTime()

        return x, y


    def get_dx(self, update=True):
        last_x  = self.last_x
        return self.get_x(update=update) - last_x


    def get_dy(self, update=True):
        last_y  = self.last_y
        return self.get_y(update=update) - last_y

[Browse code]

The most important methods in this class are get_dx and get_dy which return how far and in what direction the mouse has moved since last queried.

Next I setup a task called update_camera that’s called every single frame. update_camera gets the change in mouse position by calling get_dx and get_dy and then associates changes in mouse position with changes in the camera’s position relative to a focal point.

    def update_camera(self):
        if self.keymap[action.fine_control]:
            fx, fy = 2, 0
        else:
            fx, fy = 10, 3

        alpha_x = self.dummy.getH() - fx * self.mouse.get_dx()
        alpha_y = max(min(0, self.dummy.getR() + fy * self.mouse.get_dy()), -45)

        self.dummy.setH(alpha_x) # Move view laterally
        self.dummy.setR(alpha_y) # Move view vertically

[Browse code]

Here it is in action.

Next, I added a cue stick that follows the camera’s orientation:

[Browse code]

I also added the ability to stroke the cue by holding ‘s’ and moving the mouse up and down:

[Browse code]

Just like in real pool where you often walk around the table to check out the angles, the camera shouldn’t be pinned to a single ball. So when you hold down ‘v’, you can around the table. And by holding down left-click, you can zoom in or out with the mouse:

[Browse code]

Robust, object-oriented design

If I keep moving at this speed the codebase is going to turn into spaghetti. I need to start carving out a modular, object-oriented code structure.

(I) Mode management

The first problem I noticed is that tasks are not being properly managed.

A task is just a method that’s called each frame. For example, update_camera shown above was a task that controlled camera movement with the mouse. If you wanted, you could flatten the code complexity and have just a single task carries out all your desired functionality. For example, here’s a task that accomplishes all of the above functionality:

def master_task(self, task):
    self.update_camera_rotation()
    self.update_camera_pan()
    self.update_camera_zoom()
    self.update_stroke_position()
    return task.cont

This task delegates to respective methods and all is good. Except, depending on different scenarios, the mouse should do different things. Holding ‘s’, it should control the cue’s displacment; holding ‘left-click’, it should zoom the camera; holding ‘v’, it should pan the camera, etc. Yet in the above example, it does all of these things simultaneously. Let’s be a little more careful:

def master_task(self, task):
    if self.keymap['v']:
        self.update_camera_pan()
    elif self.keymap['left-click']:
        self.update_camera_zoom()
    elif self.keymap['s']:
        self.update_stroke_position()
    else:
        self.update_camera_rotation()
    return task.cont

Good, now what if I want to allow panning and zooming at the same time?

def master_task(self, task):
    if self.keymap['v'] and self.keymap['left-click']:
        self.update_camera_pan()
        self.update_camera_zoom()
    elif self.keymap['v']:
        self.update_camera_pan()
    elif self.keymap['left-click']:
        self.update_camera_zoom()
    elif self.keymap['s']:
        self.update_stroke_position()
    else:
        self.update_camera_rotation()
    return task.cont

The logic is becoming more branched and complex, and the game barely does anything. Further down this ill-conceived design, this method is going to start blowing up with nested conditionals. So my solution was to define game modes.

A game mode, or mode for short, represents a state that the game can be in. For example, the menu system is a mode. Aiming the cue stick is a mode. Viewing a shot is a mode. All modes manage their own tasks, that turn on and off depending on which mode is active. Whenever the game switches modes, an exit method is called for the current mode that tears down all of the current tasks, rendered objects, etc., and then an enter method is called for the next mode that builds up all of the new tasks, rendered objects, etc.

All modes inherit from this simple base class:

from abc import ABC, abstractmethod

class Mode(ABC):
    keymap = None

    def __init__(self):
        if self.keymap is None:
            raise NotImplementedError("Child classes of Mode must have 'keymap' attribute")


    @abstractmethod
    def enter(self):
        pass


    @abstractmethod
    def exit(self):
        pass

Mode inherits from ABC (from the abstract class library abc) that offers some defensive coding relating to inheritance. For example, any and all modes must possess enter and exit methods, and the @abstractmethod decorator ensures that. So an inheriting class without enter and exit methods can’t instantiate without error:

class NewMode(Mode):
    pass
NewMode()

TypeError: Can't instantiate abstract class NewMode with abstract methods enter, exit

The solution is to define enter and exit methods:

class NewMode(Mode):
    def enter(self): pass
    def exit(self): pass
NewMode()

Some more defensive coding: each mode must possess a dictionary called keymap that defines actions and whether or not they are active. By defining keymap as a class attribute and setting it to None, and then complaining in the __init__ if keymap is None, it is ensured that any inheriting class will fail object instantiation unless it explicitly defines its keymap:

class NewMode(Mode):
    def enter(self): pass
    def exit(self): pass
NewMode()

NotImplementedError: Child classes of Mode must have 'keymap' attribute

The solution is to define a keymap either as a class attribute or instance attribute

class NewMode(Mode):
    keymap = {'zoom': False}
    def enter(self): pass
    def exit(self): pass
NewMode()

Here is a currently implemented game mode, ViewMode. This refers to the mode in which the player possesses a free-form camera that can be rotated, panned, and zoomed.

class ViewMode(CameraMode):
    keymap = {
        action.aim: False, # rotate camera
        action.move: True, # pan the camera
        action.quit: False, # exit to menu
        action.zoom: False, # zoom the camera
    }


    def enter(self):
        self.mouse.hide()
        self.mouse.relative()
        self.mouse.track()

        self.task_action('escape', action.quit, True)
        self.task_action('mouse1', action.zoom, True)
        self.task_action('mouse1-up', action.zoom, False)
        self.task_action('a', action.aim, True)
        self.task_action('v', action.move, True)
        self.task_action('v-up', action.move, False)

        self.add_task(self.view_task, 'view_task')
        self.add_task(self.quit_task, 'quit_task')


    def exit(self):
        self.remove_task('view_task')
        self.remove_task('quit_task')


    def view_task(self, task):
        if self.keymap[action.aim]:
            self.change_mode('aim')
        elif self.keymap[action.zoom]:
            self.zoom_camera()
        elif self.keymap[action.move]:
            self.move_camera()
        else:
            self.rotate_camera(cue_stick_too=False)

        return task.cont


    def quit_task(self, task):
        if self.keymap[action.quit]:
            self.keymap[action.quit] = False
            self.change_mode('menu')
            self.close_scene()

        return task.cont


    def zoom_camera(self):
        with self.mouse:
            s = -self.mouse.get_dy()*0.3

        self.cam.node.setPos(autils.multiply_cw(self.cam.node.getPos(), 1-s))


    def move_camera(self):
        with self.mouse:
            dxp, dyp = self.mouse.get_dx(), self.mouse.get_dy()

        h = self.cam.focus.getH() * np.pi/180 + np.pi/2
        dx = dxp * np.cos(h) - dyp * np.sin(h)
        dy = dxp * np.sin(h) + dyp * np.cos(h)

        f = 0.6
        self.cam.focus.setX(self.cam.focus.getX() + dx*f)
        self.cam.focus.setY(self.cam.focus.getY() + dy*f)


    def rotate_camera(self):
        if self.keymap[action.fine_control]:
            fx, fy = 2, 0
        else:
            fx, fy = 13, 3

        with self.mouse:
            alpha_x = self.cam.focus.getH() - fx * self.mouse.get_dx()
            alpha_y = max(min(0, self.cam.focus.getR() + fy * self.mouse.get_dy()), -90)

        self.cam.focus.setH(alpha_x) # Move view laterally
        self.cam.focus.setR(alpha_y) # Move view vertically

From the keymap you can see there are 4 different actions that this mode supports

    keymap = {
        action.aim: False, # rotate camera
        action.move: True, # pan the camera
        action.quit: False, # exit to menu
        action.zoom: False, # zoom the camera
    }

Whenever this mode is entered, enter is called:

    def enter(self):
        self.mouse.hide()
        self.mouse.relative()
        self.mouse.track()

        self.task_action('escape', action.quit, True)
        self.task_action('mouse1', action.zoom, True)
        self.task_action('mouse1-up', action.zoom, False)
        self.task_action('a', action.aim, True)
        self.task_action('v', action.move, True)
        self.task_action('v-up', action.move, False)

        self.add_task(self.view_task, 'view_task')
        self.add_task(self.quit_task, 'quit_task')

So what happens when enter is called? First, some changes to the mouse are made. The mouse is made invisible, and set to a relative mode where the cursor is repositioned to the center of the screen so it never moves out of the game window. Then, mouse tracking is turned on so changes in movement can be associated to camera movements. Next, the actions in keymap are explicitly associated to mouse/keyboard inputs. Upon defining these key-bindings, the values in keymap will be set to True or False every frame depending on which keys are being pressed.

And finally, 2 tasks called view_task and quit_task are added to a panda3d built-in task manager. If you look at the guts of view_task and quit_task, you’ll see they carry out operations depending on which keys are being pressed.

Whenever this mode is exited, exit is called:

    def exit(self):
        self.remove_task('view_task')
        self.remove_task('quit_task')

Quite simply, the two tasks that were added to the task manager in enter are removed from the task manager in exit.

In total, I currently have 5 game modes which can be found in the pooltool.ani.modes module.

To manage switching between modes and reliably carrying out build-up and tear-down operations, I wrote a ModeManager:

class ModeManager(MenuMode, AimMode, StrokeMode, ViewMode, ShotMode):
    def __init__(self):
        # Init every Mode class
        MenuMode.__init__(self)
        AimMode.__init__(self)
        StrokeMode.__init__(self)
        ViewMode.__init__(self)
        ShotMode.__init__(self)

        self.modes = {
            'menu': MenuMode,
            'aim': AimMode,
            'stroke': StrokeMode,
            'view': ViewMode,
            'shot': ShotMode,
        }

        # Store each classes current keymap as its default
        self.action_state_defaults = {}
        for mode in self.modes:
            self.action_state_defaults[mode] = {}
            for a, default_state in self.modes[mode].keymap.items():
                self.action_state_defaults[mode][a] = default_state

        self.mode = None
        self.keymap = None


    def update_keymap(self, action_name, action_state):
        self.keymap[action_name] = action_state


    def task_action(self, keystroke, action_name, action_state):
        """Add action to keymap to be handled by tasks"""

        self.accept(keystroke, self.update_keymap, [action_name, action_state])


    def change_mode(self, mode, exit_kwargs={}, enter_kwargs={}):
        assert mode in self.modes

        self.end_mode(**exit_kwargs)

        # Build up operations for the new mode
        self.mode = mode
        self.keymap = self.modes[mode].keymap
        self.modes[mode].enter(self, **enter_kwargs)


    def end_mode(self, **kwargs):
        # Stop watching actions related to mode
        self.ignoreAll()

        # Tear down operations for the current mode
        if self.mode is not None:
            self.modes[self.mode].exit(self, **kwargs)
            self.reset_action_states()


    def reset_action_states(self):
        for key in self.keymap:
            self.keymap[key] = self.action_state_defaults[self.mode][key]

change_mode changes modes by carrying out build-up and tear-down operations for the respective modes. Because ModeManager inherits all of the modes, methods in modes can call change_mode to facilitate mode switching. For example, the quit_task method in ViewMode changes the mode to MenuMode by calling self.change_mode('menu').

    def quit_task(self, task):
        if self.keymap[action.quit]:
            self.keymap[action.quit] = False
            self.change_mode('menu')
            self.close_scene()

        return task.cont

(II) The inheritance diagram

Me from the future, here. pooltool is no longer based on the following inheritance diagram, which as I’ve progressed as a developer, I’ve come to realize is strife with a lack of cohesion and an overabundance of coupling. One day I looked at the code and realized everything was spaghetti. Since this initial design, I’ve learned a lot about python software design, and have started favoring composition over inheritance and making heavy use of some more modern python features like dataclasses. I spent about a month completely refactoring the code, and I’m happy to say it’s in a much better place. All this is to say, don’t model anything after the following inheritance diagram.

So that’s how mode management works. This is how ModeManagement fits into the inheritance diagram for the game:

Red lines show the direction of inheritance (e.g. ViewMode inherits from Mode) and blue lines indicates composition (e.g. Interface makes an instances of Table). On the right hand side are all of the objects like Table, Cue, and Ball. Each of these inherit from a base class Object, and further inherit from respective Render classes that contain all logic relating to rendering procedures. By packaging this rendering responsibility into separate classes, objects remain fully operational for non-rendered circumstances.

The class that glues everything together is Interface, which inherits from ModeManager, and therefore can switch between game modes. It also holds instances of Table, Cue, and Ball objects, which brings these objects into the namespace of tasks setup by the game modes. This is critical because it enables tasks to create, modify, and destroy objects and their rendered states based off of user action.

Taking a shot

With a thoughtful design in place, its time to simulate shots. From the last post, I already know the simulator works, its just a matter of detecting when and how hard the player’s cue strikes the cue ball. The player gains control of the cue whenever they hold down ‘s’, which activates the StrokeMode class.

To detect if the player hit the ball, I setup a task in StrokeMode that checks whether the cue’s position is negative, which implies the cue stick moves through (intersects) the cue ball. If this happens, an impact velocity is determined from a trajectory of the cue stick’s position, and self.change_mode('shot') is called which changes the mode to ShotMode.

ShotMode.enter calls ShotMode.run_simulation in a parallel process that begins the simulation. While this takes place, a translucent overlay is put onto the screen to indicate that the shot is being calculated. Since run_simulation is called in a parallel process, the player retains control over the camera while the shot’s being calculated.

Once done, the simulation animation is played on a loop.

Cushions

It looks kind of ugly how the balls bounce off invisible walls, so I modified TableRender to render the edges of cushion segments:

[Browse commit]

It’s not much better, but at least now I can start to visually sketch out the pockets. So I flipped back to when I drew this diagram for the algorithmic theory post:

Then, I modified Table.__init__ by hardcoding in these cushion segment dimensions, which resulted in something that actually takes the appearance of a pocket billiards table:

[Browse commits: 1 & 2]

It’s awesome to see that I’m now at a point where the game is starting to be actually fun to play. For now I’ll ignore that the balls don’t fall into pockets. There is actually a much more pressing issue. Check this out this bug:

After a little while I realized there was a gap in the detection boundary of the cushion that existed near the intersection of the two cushion segments. This figure explains the situation:

Figure 1. A depiction of the bug. Blue lines represent the edge of the cushion segments, and red lines show where the center of the ball (purple) must pass in order for a collision (the collision segments). The left-hand side shows what has led to the bug, and the right-hand side illustrates the solution: to add a circular cushion segment.

The figure illustrates, in red, where the center of the ball must cross in order for a collision to occur. However, the current implementation (on the left-hand side), has a clear gap that leads to the behavior observed. The solution is to explicitly define a circular cushion segment at the intersection between the linear cushion segments, as shown on the right-hand side.

Ok, so I need to introduce the concept of a circular cushion segment, which I started toying around with here:

[Browse commit]

The circular segment in the middle of the table exists as an attribute of Table, and is properly rendered, but I haven’t written the code to predict ball collisions with circular cushions. I have worked out a similar situation though, during my theoretical treatment of the ball-pocket collision, so I’m going to reuse that math… Essentially, one must solve a 4th order polynomial to determine if/when a ball collides with a circular cushion segment.

Well, this was surprisingly straight forward. I’d encourage you to check out the associated commit:

[Browse commit]

It’s kind of amazing, and easy to forget, that this is not a discretely evolved system. All of these collisions are predicted ahead of time.

[Browse commit]

With circular cushion segments now fully operational, I added circular segments (with 0 radius) to each of the cushion corners via the prescription shown in Figure 1. This patches up the gap in the collision segment depicted in Figure 1. The result is a properly behaving edge case:

[Browse commit]

Pockets

The elephant in the room is that there’s no pockets for the balls to fall into. Let’s change that.

[Browse commit]

Alright pockets exist and are rendered, but they do not yet interact with the balls. For this, I need to implement a collision event I have previously theorized: the ball-pocket collision. Here is the class I came up with:

class BallPocketCollision(Collision):
    event_type = type_ball_pocket

    def __init__(self, ball, pocket, t=None):
        self.state_start = ball.s
        self.state_end = pooltool.pocketed

        Collision.__init__(self, body1=ball, body2=pocket, t=t)


    def resolve(self):
        ball, pocket = self.agents

        # Ball is placed at the pocket center
        rvw = np.array([[pocket.a, pocket.b, -pocket.depth],
                        [0,        0,         0           ],
                        [0,        0,         0           ],
                        [0,        0,         0           ]])

        ball.set(rvw, self.state_end)
        ball.update_next_transition_event()

        pocket.add(ball.id)

BallPocketCollision inherits from Collision, which is a base class I had written for several of the other collision events. Collision doesn’t do much, but it does force all inheriting classes to define a resolve method, which I’ve done above. Resolving this collision is very easy. I update the ball’s rvw state vectors so that the ball has no spin, linear velocity, or angular velocity, and then place the ball at the bottom of the pocket. For good measure, I add the ball’s ID to the pocket, so that I can track which balls are in which pockets.

So now the concept of a ball-pocket collision exists. But I need to be able to predict when they happen, like I do for all of the other events. Fortunately I’ve already developed the math, which amounts to solving a 4th order polynomial. I implemented this algebra in physics.get_ball_pocket_collision_time.

Besides that, I just had to add a routine to evolution.EvolveShotEventBased.get_next_event that considers all ball-pocket pairs and returns the ball-pocket collision that occurs in the least amount of time (this time may in fact be $\infty$, which basically means you suck at pool):

    def get_min_ball_pocket_event_time(self):
        """Returns minimum time until next ball-pocket collision"""

        dtau_E_min = np.inf
        involved_agents = tuple([DummyBall(), NonObject()])

        for ball in self.balls.values():
            if ball.s in pooltool.nontranslating:
                continue

            for pocket in self.table.pockets.values():
                dtau_E = physics.get_ball_pocket_collision_time(
                    rvw=ball.rvw,
                    s=ball.s,
                    a=pocket.a,
                    b=pocket.b,
                    r=pocket.radius,
                    mu=(ball.u_s if ball.s == pooltool.sliding else ball.u_r),
                    m=ball.m,
                    g=ball.g,
                    R=ball.R
                )

                if dtau_E < dtau_E_min:
                    involved_agents = (ball, pocket)
                    dtau_E_min = dtau_E

        dtau_E = dtau_E_min

        return BallPocketCollision(*involved_agents, t=(self.t + dtau_E))

Here is the end result:

[Browse commit]

For now, the ball suddenly appears at the bottom of the pocket rather than falling into the pocket naturally. I’ll have to do something about that in the future. Perhaps a prebaked animation.

Because I already had a strong class framework for events, adding pockets was easy. If you want to check out just how easy it was, check out the pull request.

From interaction to game

At this point, all the major mechanics are laid out. But what’s missing is the extra fluff that turns this interactive experience into an actual game. Currently there is nothing to facilitate games like 8-ball or 9-ball. No rules to tell you whether you scratched the ball, what the win condition is, whose turn it is, and so on.

Well I decided I would end this blog post by filling in all of that fluff, which ended up in this huge PR. Now, pooltool is a primitively functioning game, with a menu, a heads-up display, different game modes, winners and losers, etc. I decided to summarize the current state of affairs in this feature video:

Next up

Next on the agenda is a graphics overhaul. See you then.

Pooltool is now an open source billiards video game rendered in 3D was originally published by Evan Kiefl at Evan Kiefl on April 17, 2021.

Creating a billiards simulator: the transition from equations to code

2021-03-25T00:00:00+00:00

Outline

In the first and second posts of this series, I discussed ad nauseam the physics and algorithmic theory behind pool simulation. With this all now behind me, it’s time to take this theory to the streets.

The skeleton

This project started with 2 main modules: engine.py and physics.py. The rationale for this design was to separate the physics from the objects that the physics acts on (balls, cues, cushions, etc).

With this in mind, engine.py implements the shot evolution algorithm by coordinating when object states should be modified, and physics.py implements the physics that provides the specific rules for how the modification should be carried out. This separation of responsibility allows different physics models to be plugged in or out at will.

Though the codebase has changed dramatically since this original implementation, this central design principle has remained unchanged.

Ball trajectories

Want to follow along?

If you want to follow along, go ahead and clone the repository, and then checkout this branch

git clone https://github.com/ekiefl/pooltool.git
cd pooltool
git checkout f79c801_offshoot

My first goal was to start dead simple: visualize the trajectory of the cue ball that has been struck with a cue stick, assuming there are no cushions, pockets, or other balls. I implemented this in the engine.ShotSimulation class.

Let’s peer in.

In [1]: from psim.engine import ShotSimulation
   ...: shot = ShotSimulation()
   ...: shot.setup_test()

shot instantiates objects you wouldn’t be surprised to find in a pool simulation: Ball, Table, and Cue objects. For example, here are the attributes of the cue ball.

In [2]: vars(shot.balls['cue'])
Out[2]:
{'id': 'cue',
 'm': 0.170097,
 'R': 0.028575,
 'I': 5.555576388825e-05,
 'rvw': array(
     [[0.686, 0.33 , 0.   ],
      [0.   , 0.   , 0.   ],
      [0.   , 0.   , 0.   ]]
 ),
 's': 0,
 'history': {'t': [], 'rvw': [], 's': []}}

Of these attributes, the most important is rvw, which stores the ball state as a $3 \times 3$ numpy array. rvw is named after the 3 state vectors $\vec{r}(t)$, $\vec{v}(t)$, and $\vec{\omega}(t)$.

rvw[0,:] is the displacement vector $\vec{r}(t)$
rvw[1,:] is the velocity vector $\vec{v}(t)$
rvw[2,:] is the angular velocity vector $\vec{\omega}(t)$.

This means the velocity of the cue ball is

In [3]: shot.balls['cue'].rvw[1,:]
Out[3]: array([0., 0., 0.])

In other words, it’s not moving. To simulate something meaningful, energy has to be added to the system. In billiards, that’s done with the Cue:

In [4]: vars(shot.cue)
Out[4]: {'M': 0.567, 'brand': 'Predator'}

This is a 20oz Predator–these things are expensive. Unsurprisingly, shot.cue has a method for striking balls.

In [5]: shot.cue.strike?
Signature: shot.cue.strike(ball, V0, phi, theta, a, b)
Docstring:
"""
Strike a ball
                          , - ~  ,
◎───────────◎         , '          ' ,
│           │       ,             ◎    ,
│      /    │      ,              │     ,
│     /     │     ,               │ b    ,
◎    / phi  ◎     ,           ────┘      ,
│   /___    │     ,            -a        ,
│           │      ,                    ,
│           │       ,                  ,
◎───────────◎         ,               '
  bottom rail           ' - , _ , -
                 ______________________________
                          playing surface
Parameters
==========
ball : engine.Ball
    A ball object
V0 : positive float
    What initial velocity does the cue strike the ball?
phi : float (degrees)
    The direction you strike the ball in relation to the bottom rail
theta : float (degrees)
    How elevated is the cue from the playing surface, in degrees?
a : float
    How much side english should be put on? -1 being rightmost side of ball, +1 being
    leftmost side of ball
b : float
    How much vertical english should be put on? -1 being bottom-most side of ball, +1 being
    topmost side of ball
File:      ~/Software/pooltool_testing/psim/engine.py
Type:      method
"""

This method ultimately calls upon physics.cue_strike, which implements the cue-ball interaction physics described here.

I’m going to strike the cue ball with a center-ball hit $(a=0, b=0)$ straight down the table $(\phi=90)$ with the cue completely level with the table $(\theta = 0)$. I’ll use a relatively slow impact speed of $V_0 = 0.5 \, \text{m/s}$.

In [5]: shot.cue.strike(
   ...:     ball = shot.balls['cue'],
   ...:     V0 = 0.5,
   ...:     phi = 90,
   ...:     theta = 0,
   ...:     a = 0,
   ...:     b = 0,
   ...: )

In [6]: shot.balls['cue'].rvw
Out[6]:
array([[0.686, 0.33 , 0.   ],
       [0.   , 0.877, 0.   ],
       [0.   , 0.   , 0.   ]])

After calling the method, the cue ball now has a $0.88 \, \text{m/s}$ velocity in the $y-$direction. Since it was a center ball hit with no cue elevation, the ball has no spin. i.e. $\vec{\omega} = \langle 0,0,0 \rangle$.

At this point, no time has passed–the ball state has merely been modified according the physics of shot.cue striking self.balls['cue']. So then how does the shot evolve?

Rather than implementing the event-based shot evolution algorithm I wouldn’t shut up about, I implemented some dinky discrete time evolution algorithm just to get the ball rolling. It’s a for loop that increments by $50 \text{ms}$.

q = self.balls['cue']

for t in np.arange(0, 10, 0.05):
    rvw, s = physics.evolve_ball_motion(
        rvw=q.rvw,
        R=q.R,
        m=q.m,
        u_s=self.table.u_s,
        u_sp=self.table.u_sp,
        u_r=self.table.u_r,
        g=psim.g,
        t=t,
    )
    q.store(t, *rvw, s)

The workhorse is evolve_ball_motion, which calculates the new state for each time step. It delegates to evolve_slide_state, evolve_roll_state, and evolve_spin_state, all of which update the ball state according to the appropriate equations of motion.

To test if evolve_ball_motion and its delegates are behaving, I called shot.start, which carries out the discrete time evolution and plots the ball’s trajectory over time.

In [5]: shot.start()

Immediately after being hit, the ball is sliding. Yet after a short amount of time, the relative velocity converges to $\vec{0}$, which defines the transition from sliding to rolling. Once rolling, the ball continues to roll until it reaches the stationary state.

If a picture says a thousand words, a video says a thousand pictures. Before going any further, I needed a way to animate shots because I’m already bored of these static plots. I wasn’t looking for perfection, I just needed something to animate trajectories. For this, I found pygame. It just celebrated its 20th anniversary, which is pretty impressive for a python package.

I implemented the module psim.ani.animate, which animates ball trajectories using pygame. For the sake of demonstration, this functionality already exists in the branch we’re using.

Let’s animate the shot.

In [5]: shot.start(plot=False)
In [6]: shot.animate(flip=True) # Flip orientation to be horizontal

That’s better.

Getting a little more brave, I wanted to try a massé shot, which you can watch the pros do here:

To achieve this effect, you have to strike down on the cue ball $(\theta)$ with a sizable amount of side-spin $(a)$.

In [7]: shot.cue.strike(
   ...:     ball = shot.balls['cue'],
   ...:     V0 = 1,
   ...:     phi = 90,
   ...:     theta = 20,
   ...:     a = -0.5,
   ...:     b = 0.0,
   ...: )
   ...: shot.start(plot=False)
   ...: shot.animate(flip=True)

And voila. Note that all of the curvature takes place in the sliding state. This is because the rolling state by definition has a relative velocity of $\vec{0}$, which is a requirement for curved trajectories.

By the way, all sliding state trajectories under the arbitrary spin model take the form of a parabola. I never proved this but Dr. Dave Billiards did here.

Next, I tried to apply insane levels of massé, like this guy:

Specifically, I tried to tune the parameters to remake the shot at 0:30s. After fumbling around, I ended up with these shot parameters.

In [8]: shot.balls['cue'].rvw[0] = [0.18, 0.37, 0]
   ...: shot.cue.strike(
   ...:     ball = shot.balls['cue'],
   ...:     V0 = 1.15,
   ...:     phi = 335,
   ...:     theta = 55,
   ...:     a = 0.5,
   ...:     b = -0.0,
   ...: )
   ...: shot.start(plot=False)
   ...: shot.animate(flip=False)

This may not be perfect, but it’s close. Yes.

Being able to recapitulate Florian’s shot allows me to answer the following question: what insane levels of spin are required to pull this shot off? Well, In RPMs, the initial rotational speed of the cue ball is

In [21]: np.linalg.norm(shot.balls['cue'].rvw[2])/np.pi*60
Out[21]: 4374.123861245154

$4400$ RPM. That’s… that’s too much, right? Well, the same guy put out this video where he purports his RPM for a different shot to be $3180$. So I’m certainly in the ball park. Maybe he can get up to $4400$ RPM, or maybe my simulated cloth had a higher coefficient of sliding friction, requiring higher RPM.

Overall, these trajectories have me convinced I’m not screwing anything up royally.

Event-based evolution algorithm

So far I’ve been evolving the simulation by incrementing time in small discrete steps (aka a discrete time evolution algorithm). Yet moving forward, I’ve opted to use the event-based evolution algorithm for its superior accuracy and computational efficiency.

The premise of the algorithm is this:

We have beautiful equations of motion for each ball that collectively describe the evolution of the system state. Great.
But events between interfering parties (e.g. a ball-ball collision) disrupt the validity of these equations, since they assume each ball acts in isolation.
Even still, the equations for each ball are valid up until the next event.
So the algorithm works by evolving the system state directly up until the next event, at which time the event must be resolved (e.g. a ball-ball collision event is resolved by applying the ball-ball interaction equations), and then the process repeats itself: the next event is found and the system state is evolved up until the next event.
There’s only one way to calculate when the next event occurs: calculating the time until every single possible next event. By definition of next event, the next event is the event that occurs in the least amount of time.

If you want an in-depth explanation on the event-based evolution algorithm, I may have created the most extensive learning resource on the topic in my last post.

Implementing transitions

All events are either transitions, or they are collisions (details here). Since there are no collisions yet, I decided to implement the algorithm using just transition events to start. Transition events mark the transitioning of a ball from one motion state to another (e.g. from rolling to stationary).

Let’s take a look.

Want to follow along?

For demo purposes, I compiled my progress into a branch. If you want to follow along, go ahead and check(out) it out.

git clone https://github.com/ekiefl/pooltool.git
cd pooltool
git checkout edfc866_offshoot

Like before, I created an instance of ShotSimulation, and then I set up a pre-baked system state specified by the keyword straight_shot

In [1]: import psim.engine as engine
   ...: shot = engine.ShotSimulation()
   ...: shot.setup_test('straight_shot')

Next, I struck down on the cue ball with bottom-left english.

In [2]: shot.cue.strike(
   ...:     ball = shot.balls['cue'],
   ...:     V0 = 1.35,
   ...:     phi = 97,
   ...:     a = 0.3,
   ...:     b = -0.3,
   ...:     theta = 10,
   ...: )

To see what this system state looks like and how it evolves, here is the discrete time evolution.

In [3]: shot.simulate_discrete_time()
   ...: shot.animate(flip=True)

The cue ball starts sliding (red) and then transitions to rolling (green). Offscreen, it transitions to stationary. Because I struck down with side english, there is a slight masse (curve) in the trajectory. In total, there are two transition events: (1) a sliding-rolling transition event and (2) a rolling-stationary transition event.

In comparison, this is what happens when the system state is evolved using the event-based algorithm.

In [4]: shot.setup_test('straight_shot')
   ...: shot.cue.strike(
   ...:     ball = shot.balls['cue'],
   ...:     V0 = 1.35,
   ...:     phi = 97,
   ...:     a = 0.3,
   ...:     b = -0.3,
   ...:     theta = 10,
   ...: )
   ...: shot.simulate_event_based()
   ...: shot.animate(flip=True)

There’s only 3 snapshots of the system state: at the initial time, at the sliding-rolling transition time, and at the rolling-stationary transition time. That is the beauty of the event-based evolution algorithm–rather than evolving in small time steps, the system state is evolved directly to the next event. For this case, this directness has resulted in the shot evolution being compressed into just 3 system state snapshots.

While algorithmically beautiful, it’s an eye-sore to look at. So I wrote a continuize method that calculates all of the intermediate system states.

In [4]: shot.setup_test('straight_shot')
   ...: shot.cue.strike(
   ...:     ball = shot.balls['cue'],
   ...:     V0 = 1.35,
   ...:     phi = 97,
   ...:     a = 0.3,
   ...:     b = -0.3,
   ...:     theta = 10,
   ...: )
   ...: shot.simulate_event_based(continuize=True)
   ...: shot.animate(flip=True)

Now the intermediate system states are calculated ad hoc for the purposes of a nice animation. Nice.

Implementing collisions

The algorithm is a little boring without collisions. Let’s add some.

With discrete time evolution, the only way to know if a ball-ball collision occurs is to use a collision detector to see if 2 balls are overlapping. However in the event-based algorithm, ball collisions are predicted ahead of time by solving a quartic polynomial defined from the balls’ equations of motion.

For everyone’s convenience, ball-ball collisions are already implemented in this demo branch, and can be turned on like so. If you’re interested, here is the code for solving the quartic polynomial.

In [5]: engine.include['ball_ball'] = True

Running the simulation again now yields a more interesting picture.

It’s a shame that the balls fly right off the table, so while we’re at it, let’s include ball-cushion collisions too.

In [5]: engine.include['ball_cushion'] = True

The implementation of the ball-cushion interaction in this branch is non-physical. In fact, it’s not even close to accurate, I just wanted to add another collision event to test the algorithm. In this overly simplistic implementation, ball-cushion collisions are resolved by reversing the velocity component perpendicular to the cushion surface. Of course this is highly unrealistic–for example the cushion has no interaction with the ball’s spin. In the future, I will replace this with the (Han, 2005) physics model discussed previously.

Now, things are really starting to take shape.

I find this to be a pretty illustrative visualization of how the event-based algorithm advances the system state through time.

To me, its incredible to think that each event has been carefully chosen from the entire set of all possible next events. For example, the 3rd event is a sliding-rolling transition of the cue ball after its collision with the 8-ball. The 4th event is determined by considering all of the events in this diagram:

In total 15 possible events were considered, and the time until each of them was explicitly calculated. Based on the system state, it turned out that the one that physically occurs is a collision of the 8-ball (black) with the 3-ball (red), which you can see here:

Comparing the algorithms

How do these results compare to the discrete time algorithm? With discrete time, collisions are detected retrospectively by seeing if there is any overlapping geometry. This leads to an inherent inaccuracy. Sure, it can be reduced by decreasing the time step. But how small does the timestep have to be and what effect does this have on performance?

To compare the two algorithms, I moved the brown 7-ball from the previous simulation just slightly, such that the cue-ball barely grazes it when using the event-based algorithm.

Any simulation evolved with a discrete time algorithm can’t possibly be considered accurate unless it can recapitulate this event.

Let’s see the corresponding discrete time evolution using a timestep of 20ms.

In [6]: import psim.engine as engine
   ...: engine.include['ball_ball'] = True
   ...: engine.include['ball_cushion'] = True
   ...: shot = engine.ShotSimulation()
   ...: shot.setup_test('straight_shot')
   ...: shot.cue.strike(
   ...:     ball = shot.balls['cue'],
   ...:     V0 = 1.35,
   ...:     phi = 97,
   ...:     a = 0.3,
   ...:     b = -0.3,
   ...:     theta = 10,
   ...: )
   ...: shot.simulate_discrete_time(dt=0.020)
   ...: shot.animate(flip=True)

Wow, the cue didn’t even come close to hitting the brown 7-ball after bouncing off the cushion. And the 8-ball is supposed to collide with the red 3-ball, but missed entirely. The problem is that the collision angle between the cue and 8-ball is slightly off due to discrete time error, and the net result is that the overlapping geometry overestimates how thin the cut on the 8-ball is.

So then how small must the timestep be for the sequence of events to match the event-based algorithm? To find out, I ran a series of shots evolved with smaller and smaller timesteps. For each, I considered it to be accurate if the presence and order of events matched what was found with the event-based algorithm. Here is the script:

#! /usr/bin/env python

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

import psim.utils as utils
import psim.engine as engine

engine.include = {
    'motion': True,
    'ball_ball': True,
    'ball_cushion': True,
}

def setup():
    """Return a shot object that is ready for simulation"""
    shot = engine.ShotSimulation()
    shot.setup_test('straight_shot')
    shot.cue.strike(
        ball = shot.balls['cue'],
        V0 = 1.35,
        phi = 97,
        a = 0.3,
        b = -0.3,
        theta = 10,
    )
    return shot

def is_accurate(shot, true_shot):
    """Tests if series of events matches event-based event history"""
    true_history = [(event.event_type, event.agents)
                    for event in true_shot.history['event']
                    if event is not None
                    and event.event_type in ('ball-ball', 'ball-rail')]
    history = [(event.event_type, event.agents)
               for event in shot.history['event']
               if event is not None]

    return True if history == true_history else False

# Simulate using event-based algorithm and store time taken
cts_shot = setup()
with utils.TimeCode() as t:
    cts_shot.simulate_event_based()
cts_time = t.time.total_seconds()

# Init a dict that stores discrete time simulation stats
results = {
    'dt': [],
    'time': [],
    'accurate': [],
}

# Run many discrete time simulation with decreasing timestep
for dt in np.logspace(0, -4.5, 30):
    shot = setup()
    with utils.TimeCode() as t:
        shot.simulate_discrete_time(dt)

    results['dt'].append(dt)
    results['time'].append(t.time.total_seconds())
    results['accurate'].append('accurate' if is_accurate(shot, cts_shot) else 'inaccurate')

results = pd.DataFrame(results)

sns.scatterplot(data=results, x='dt', y='time', hue='accurate')
plt.plot([results['dt'].min(), results['dt'].max()], [cts_time, cts_time], label='event-based', c='green')
plt.xscale('log')
plt.yscale('log')
plt.xlabel('Timestep [s]')
plt.ylabel('Calculation time [s]')
plt.legend(loc='best')
plt.tight_layout()
plt.show()

The results show that accurate simulations (i.e. simulations where the events match the event-based simulation) require timesteps below about $250 \, \mu\text{s}$. Unfortunately, this comes at a monumental speed cost. The fastest calculation time for accurate simulations was about 8s, which is about 80X slower than the compute time for the event-based algorithm (green line).

This isn’t an extensive comparison between the two algorithms. But it illustrates the fundamental difference between them: except for inaccuracies arising from floating point precision, the event-based algorithm is as accurate as the underlying physics models, and performs reasonably fast. In contrast, discrete time algorithms produce an entire spectrum of accuracies, and choosing timesteps that yield sufficient accuracy typically leads to slow performance.

Strengths and weaknesses

The event-based algorithm has some features that may surprise people used to working with discrete time algorithms.

For example, if you want to simulate how a protein folds using molecular dynamics, simulating a 1 microsecond process takes 10X longer than simulating a 100 nanosecond process, since you have to take 10X more discrete steps. In other words, a longer simulated time means a longer compute time.

But for the event-based algorithm, simulated time has no influence on compute time. A ball could roll for 10 seconds or 5 decades.

Let me prove it to you. Let’s take a table that has a length equal to the earth’s circumference (and a width of 2m). Then put a ball on one end, strike it with an incoming cue speed of 10km/s, and let’s see how long the simulation takes to unfold.

In [9]: import psim.utils as utils
   ...: import psim.engine as engine
   ...: import psim.objects as objects
   ...: engine.include['ball_cushion'] = True
   ...:
   ...: shot = engine.ShotSimulation()
   ...: shot.table = objects.Table(l=40_075e3, w=2) # table as long as earth's circumference
   ...:
   ...: # Set up cue ball
   ...: shot.cue = objects.Cue()
   ...: ball = objects.Ball('cue')
   ...: ball.rvw[0] = [1, 1, 0]
   ...: shot.balls = {'cue': ball}
   ...: shot.cue.strike(
   ...:     ball = shot.balls['cue'],
   ...:     V0 = 10000,
   ...:     phi = 89.9999, # not quite straight down table, so it bangs into side cushions during its journey
   ...:     a = 0.0,
   ...:     b = 0,
   ...:     theta = 0,
   ...: )
   ...: shot.touch_history()
   ...: with utils.TimeCode(): shot.simulate_event_based()
✓ Code finished after 0:00:00.736432

It took 0.73s to simulate a shot that journeyed for…

In [17]: shot.history['time'][-2]/3600
Out[17]: 8.923913780057543

9 hours. During which time the cue ball traveled the length of the earth and back. The reason it took so little compute time is because there were only 54 events.

In [44]: len(shot.history['event'])
Out[44]: 54

So what determines the compute time is the number of events, since that’s roughly equal to the number of computational tasks.

I wanted to find out what else influences or doesn’t influence compute time. So I calculated the compute time with respect to two variables of interest: (1) size of table and (2) number of balls. To do this, I initialized random system states with varying table sizes and number of balls. In each case, the initial starting positions and velocities of each ball were randomized. Then, I evolved the system using event-based algorithm, taking note of the compute time.

Here is the script.

#! /usr/bin/env python

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

import psim
import psim.utils as utils
import psim.engine as engine
import psim.objects as objects

from matplotlib.colors import LogNorm

engine.include = {
    'motion': True,
    'ball_ball': True,
    'ball_cushion': True,
}

def setup(num_balls=10, scale=1.0, v=1.0):
    """Return a shot object that is ready for simulation

    Generates a random system state where each ball has initial speed `v`

    Parameters
    ==========
    num_balls : int, 10
        How many balls should be in the simulation? They are randomly placed.
    scale : float, 1.0
        Scale factor for how big the table should be. 1.0 Corresponds to a 9-foot table
    v : float, 1.0
        The speed each ball starts with. The direction is randomly assigned
    """

    shot = engine.ShotSimulation()
    shot.table = objects.Table(w=psim.table_width*scale, l=psim.table_length*scale)
    shot.cue = objects.Cue(brand='Predator')
    for i in range(num_balls):
        new_ball = objects.Ball(i)
        # Ensure no balls overlap
        while True:
            pos = [
                new_ball.R + (shot.table.w-2*new_ball.R)*np.random.rand(),
                new_ball.R + (shot.table.l-2*new_ball.R)*np.random.rand(),
                0,
            ]
            for ball in shot.balls.values():
                if np.linalg.norm(pos - ball.rvw[0]) > 2*ball.R:
                    continue
                else:
                    break
            else:
                break

        shot.balls[i] = new_ball
        # Set position
        shot.balls[i].rvw[0] = pos
        # Set velocity
        vel_angle = 360*np.random.rand()
        shot.cue.strike(shot.balls[i], V0=v, phi=vel_angle, sweet_spot=True)

    shot.touch_history()

    return shot

event_results = {
    'scale': [],
    'balls': [],
    'time': [],
}

ball_nums = range(2, 16, 2)
scales = np.linspace(1, 5, 10)

for ball_num in ball_nums:
    for scale in scales:
        print(ball_num, scale)

        # Measure time for event-based simulation
        shot = setup(ball_num, scale, v=2.0)
        with utils.TimeCode() as t:
            shot.simulate_event_based()

        event_results['scale'].append(scale)
        event_results['balls'].append(ball_num)
        event_results['time'].append(t.time.total_seconds())

# Plot the results

event_results = pd.DataFrame(event_results)
event_results['time'] = np.round(event_results['time'], 2)

sns.heatmap(
    event_results.pivot(index='scale', columns='balls', values='time'),
    annot=True,
    fmt='g',
    cmap='mako',
    norm=LogNorm(vmin=event_results['time'].min(), vmax=event_results['time'].max())
)
plt.title('Calculation time for event-based simulation [s]')
plt.show()
plt.close()

This script produces a heatmap.

Looking at the heatmap, it’s clear that increasing the number of balls $(n)$ increases computation time. This is because (1) the state of every ball must be updated every time step ($\mathcal{O}(n)$ dependence on $n$), and (2) collision prediction of each ball with every other ball must be carried out every time step ($\mathcal{O}(n^2)$ dependence on $n$).

You might expect that increasing the size of the table would increase computation time because there is more distance to travel. Yet keep in mind that in the event-based algorithm, collisions are predicted by solving the roots of polynomial equations. Whether the balls are far apart, close together, traveling fast, or traveling slow does not increase or decrease the computational complexity of solving the roots. The only effect these parameters have is changing the values of the coefficients. This is the power of the event-based algorithm.

The real effect that table size has on computation is the number of events. More events means more computation time. And when the table size increases, the balls become more spread out, which decreases the likelihood of events. So in fact, we see a decrease in compute time with respect to table size.

Conclusion

At this point in the project, I’m happy where things are. The ball trajectories match the trick shots of Florian Kohler, the event-based algorithm is working like a charm for transition events, ball-ball collisions, and ball-cushion collisions, and I have some primitive physics implementations of the ball-ball and ball-cushion interactions.

But I’m sick of looking at 2D animations, and I’m sick of setting up simulations by writing code. In the next post, I’m turning pooltool into an interactive game with 3D visualization using panda3d.

See you then.

Creating a billiards simulator: the transition from equations to code was originally published by Evan Kiefl at Evan Kiefl on March 25, 2021.

Classifying dog barks for my dog-sitting software

2021-03-14T00:00:00+00:00

Intro

Overview

This post details the step-by-step process of creating an audio classifier that tracks and responds to the behavior of my girlfriend’s dogs while we are gone. The resulting data is distilled into an interactive interface such as the one shown here.

[Click for interactive plot]

Where I left off

In the first post of this series, I made a virtual dog-sitter that detects audio events, responds to them, and stores them in a database that can be interactively visualized.

Using this dog-sitter (github here), I’ve been tracking the behavior of my girlfriend’s dog (Maple) when we leave her alone in the apartment.

I ended that post saying I would report back after acquiring a lot more data and there are a few logistical updates on that front.

First, it is no longer just Maple we are tracking. We decided that Kourtney’s other dog, Dexter (pictured above), should also be in the room. Originally we separated them based on anecdotal evidence that they barked more when together than when separated. As the data in this post will tell us, keeping them together is a very good idea.

Second, we don’t lock Maple in the crate anymore. This way she can interact and play with Dexter, and pick up new hobbies like destroying our bedroom door (more on that later).

Finally, there were some complications, which means I don’t have as much data as I wanted. Highlights include: I accidentally deleted about a month of data, the volume knob on my microphone changed and went unnoticed for a month, and my calibration system for event detection kept event rates inconsistent across sessions. All of this is sorted out now.

So there isn’t enough data to do a comprehensive analysis on the dogs’ improvement over time, and whether or not praising/scolding has an effect on their behavior. Not yet anyways.

Creating an audio classifier

While I wait another couple months for the data to pour in, I decided that being able to classify audio events could greatly increase the accuracy and capabilities of the dog-sitter.

In the last post I developed flowcharts to determine when the dog-sitter should intervene with pre-recorded audio that either praises or scolds the dog. The logic relied on the assumption that loud is bad and quiet is good. Yet if I could classify each audio event, I could step away from this one dimensional paradigm and begin developing more nuanced responses that understand the sentiment of the audio.

For example, whining could trigger a console response. Playing could trigger an encouragement response. Implementing this with the old paradigm would be impossible, because whining and playing produce similar levels of noise.

So that’s one thing an audio classifier would enable: nuanced response.

Another thing it would enable is protection against non-dog audio. The dog-sitter may detect undesirable noise from nondescript sources, like a nearby lawnmower, a person talking in the alley, a dump truck beeping. If I train a good classifier, I could filter out non-dog audio from ever entering my databases. This would filter bad data out of downstream analyses, and prevent any mishaps like scolding a dog because of a lawnmower.

And finally, creating a classifier would greatly improve the ability to understand what happens when I leave. After returning home, I would be able to immediately assess in a highly-resolved manner how well the dogs behaved while I was gone.

Deciding on a classifier algorithm

Don't make anything more complicated than it needs to be.

In my opinion, this quote should be attributed to the random forest algorithm. There is really no point starting with any other classifying algorithm. If the classifier is bad, I’ll try something more complicated.

Moving on.

Section I: Data transformation

Unfortunately, I can’t just throw the audio data into a random forest and be done with it. The data needs to be wrangled and transformed into a suitable format that’s consistent across samples and pronounces distinguishable features.

Moving forward, I’ll use these two events as examples.

The first is a door scratch:

Your browser does not support the audio element.

[click for raw text data]

The second is a whine:

Your browser does not support the audio element.

[click for raw text data]

Using the maple API

I found these events using an interactive plot generated with

./main.py analyze --session 2021_02_13_19_19_33

Once I found these two events, I plotted and exported the audio/text files using the SessionAnalysis class in the maple.data module:

from maple.data import SessionAnalysis
session = SessionAnalysis(path='data/sessions/2021_02_13_19_19_33/events.db')

session.plot_audio_event(160)
session.save_event_as_wav(event_id=160, path='door_scratch.wav')
session.save_event_as_txt(event_id=160, path='door_scratch.txt')

session.plot_audio_event(160, xlim=(0, 4.53))
session.save_event_as_wav(event_id=689, path='whine.wav')
session.save_event_as_txt(event_id=689, path='whine.txt')
session.disconnect()

For anyone who doesn’t work with audio files, the data is simple. Audio signal is just a 1D array, where the value of each point is proportional to the pressure/strength of the sound. As you move through the array, you’re moving through time at a specified sampling rate. In my recordings I chose a standard sampling rate of 44100 Hz.

Why 44100 Hz? The human ear can perceive sound waves up to around 20,000 Hz. But resolving a sinusoidal wave requires sampling frequency that is at least double the wave’s frequency. Hence we have 44100 Hz.

Denoising

There are some sources of noise that I have to contend with. Sometimes I leave a (non-rotating) fan on in the room that the mic picks up. Second, there exists some inherent hiss from the microphone. Since both of these noise sources are static (non-dynamic), applying some simple spectral gating can significantly reduce noise. I did exactly this using the python module noisereduce.

Here’s a demonstration of how effective the denoising is. First is raw, second is denoised.

Your browser does not support the audio element. Your browser does not support the audio element.

This denoising procedure requires a sample of the background audio, so at the start of a session recording, a 2-second audio clip of the background noise is recorded during calibration. Event audio is then denoised upon detection using this background audio sample. This means the audio used for training and classification has already been denoised.

Data chunking

Each audio event (sample) has a unique length (number of features), yet most classifiers demand that every sample must have the same number of features.

To deal with this, I decided each audio event would be split into equal sized chunks and the classifier would classify chunk-by-chunk.

What length of time should each chunk be? I didn’t want to exclude any events that were shorter than the chunk size, so I took a look at the distribution of event lengths for some random session.

import matplotlib.pyplot as plt
from maple.data import SessionAnalysis
session = SessionAnalysis(path='data/sessions/2021_02_13_19_19_33/events.db')
plt.hist(session.dog['t_len'], bins=300)
plt.yscale('log')
plt.xlabel('event length [s]')
plt.show()

It seems most event times are less than 2.5 seconds, and events are most frequently seen between 0.3s and 0.5s. To make sure events are composed of at least one chunk, I opted to use a chunk size of 0.25s. Alternatively I could have picked a larger chunk size and zero-padded shorter events such that they contained at least one chunk.

The minimum event length time

If you’re wondering why there are no events lower than 0.33s, it has to do with the event detection heuristics. If I used different values, by editing the config file, the minimum event length would have been different. Here are the settings I used for this session (and all sessions):

[detector]
# Standard deviations above background noise to consider start an event
event_start_threshold = 4
# The number of chunks in a row that must exceed event_start_threshold in order to start an event
num_consecutive = 4
# Standard deviations above background noise to end an event
event_end_threshold = 4
# The number of seconds after a chunk dips below event_end_threshold that must pass for the event to
# end. If during this period a chunk exceeds event_start_threshold, the event is sustained
seconds = 0.25
# If an event lasts longer than this many seconds, everything is recalibrated
hang_time = 180

Frequency or time? Answer: both

I want to represent the data so that distinguishable features are accentuated for the classifier. Thinking in these terms, audio has two really important features: time and frequency.

Above is the 1D audio signal for the door scratching event. This plot basically illustrates how energy is spread along the axis of time. And there is clearly distinguishing information in this view. For example, we can see that my door is being destroyed in discrete pulses that are roughly equally spaced through time.

But frequency-space also holds critical information. We can visualize how energy is spread across the axis of frequency by calculating a Fourier transform of both events:

from maple.data import SessionAnalysis
session = SessionAnalysis(path='data/sessions/2021_02_13_19_19_33/events.db')
session.plot_audio_event_freq(160)
session.plot_audio_event_freq(689)

Your browser does not support the audio element.

You can really see how my door is being destroyed over a wide range of frequencies. If you listen carefully to the audio, you can hear the bassline and the high hat. In comparison, Maple’s whine is much more concentrated in frequency space, localized to a range of about 500-2500 Hz. This is reflected in the relative pureness of the audio, giving that whistley sort of sound.

Clearly, time and frequency are giving complementary information that the classifier should use both of. To accomplish this, I decided to transform the data into spectrograms, where the $x-$axis is time and the $y-$axis is frequency. This shows how energy is spread across time and frequency space.

from maple.data import SessionAnalysis
session = SessionAnalysis(path='data/sessions/2021_02_13_19_19_33/events.db')
session.plot_audio_event_spectrogram(160, log=True)
session.plot_audio_event_spectrogram(689, log=True)

Your browser does not support the audio element.

Note that I log-transformed the signal with log=True so that low values are visualizable.

This new view simultaneously illustrates distinguishing features in both time- and frequency-space, and the results are very promising. Look at how different the spectrograms look! And they are very informative. In the whine event, you can really see Maple’s pitch starts high and ends low, which perfectly matches the perceived audio.

Keep in mind that I’ve shown the spectrograms for the entire audio event. But the plan is to slice each audio event into 0.25 second samples, and then calculate spectrograms for each. This ensures each sample has the same number of features (length).

As seen, a spectrogram is a 2D matrix of data, but a classifier expects samples that are 1D. I solve this by flattening each spectrogram by joining all the rows of data into one long 1D array. For example, here is what the first chunk of the whine event ultimately looks like to the classifier:

import matplotlib.pyplot as plt
from maple.data import SessionAnalysis
from maple.audio import get_spectrogram

session = SessionAnalysis(path='data/sessions/2021_02_13_19_19_33/events.db')
subevent_audio = session.get_subevent_audio(event_id=689, subevent_id=0, subevent_time=0.25)
_, _, S = get_spectrogram(subevent_audio, log=False, flatten=True)

plt.plot(S, c='#165BAA')
plt.ylabel('Signal amplitude [16 bit]')
plt.xlabel('Flattened axis')
plt.show()

This may not look as informative as the 2D heatmaps shown above, but it contains exactly the same information.

Transformation summary

A brief summary is in order.

Audio clips are denoised with spectral gating to remove static noise
Audio clips are segmented into 0.25 second chunks and will be classified chunk-by-chunk
Each chunk is transformed into a spectrogram to accentuate the most important qualities of sound: amplitude with respect to time, and amplitude with respect to pitch.
The spectrograms are flattened into 1D arrays for the sake of the classifier

Section II: Labeling

The random forest algorithm requires training data. I considered two options for obtaining training data:

Use a database of labelled dog audio, assuming such a dataset exists freely on the internet somewhere.
Manually label a subset of the audio events I have recorded over the last several months.

Ultimately, I decided on option 2, but option 1 is not without merits.

First of all, option 1 doesn’t require manual labeling, which is a huge investment of time. Second, such a broad database would presumably have data recorded under a multitude of audio settings and for a multitude of dogs, which is good and bad. It’s good because it makes the classifier more suitable for productionalization for when different mic settings, room acoustics, and dogs would in theory be used. But it also is bad because accuracy will suffer in comparison to a dataset trained specifically on Maple and Dexter with consistent mic settings and acoustic environments.

Ultimately this is what swayed me to option 2. Additionally, manually labelled data reduces constraints because I can create whichever labels I want. For example, a pre-labeled database would be unlikely to have a scratch_door label, and if it did, it would be unlikely to sound like my particular door.

Picking labels

Before going through the process of manually labelling data, I needed to decide on the set of labels to use. After reviewing the audio, Kourtney and I decided on these 6 labels.

Whine

At this point, you’re familiar with the whine. This is primarily a Maple special, although Dexter is also well-versed in the art.

Your browser does not support the audio element.

Howl

The howl is a distinct escalation of the whine, exhibited exclusively by Maple.

Your browser does not support the audio element.

Bark

The bark space is dominated by Dexter. It’s characterized by its high pitch, and being miserable to listen to.

You should turn your volume down.

Your browser does not support the audio element.

Play

Maple and Dexter often play together. It may surprise you that it sounds like this:

Your browser does not support the audio element.

Usually playing means tug of war with a toy, biting each other’s ears, or chasing each other.

Door scratch

This one you’ve already heard. I haven’t collected any direct evidence for who is responsible for ruining my door, but I suspect it is Maple.

Your browser does not support the audio element.

None

This class is a catch-all for events that are not made by the dogs, or don’t fit into the other classes. For example, here is me greeting Dexter after returning home:

Your browser does not support the audio element.

The more audio events I can label with this class, the more protection I will have against false-positives.

With a set of labels, it’s time to label a boat-load of data.

Labeling

Manual labelling is so laborious that I wanted to streamline the process as best I could. Basically, I needed something that would subsample events from a collection of session databases, play them to me, splice them into chunks, and then store the label I attribute into a table. This is all handled by the class LabelAudio in the maple.classifier module.

I extended the command line utility to use the class with the following command:

./main.py label --label-data label_data.txt --session-paths training_session_paths

Here’s a demo.

With the labeler in hand, my initial goal was to label 10,000 audio chunks. But of the 50,621 audio chunks at my disposable, Kourtney and I were able to withstand labelling just 2,745 audio chunks before giving up. Hopefully it’s enough.

Section III: Training

The next step was to actually create and validate a random forest with this labelled data.

Class description

Basically, I needed to load in the label data, transform the audio into spectrograms, split the data into training and validation sets, build the classifier using the training set, validate the classifier with the validation set, and save the model for further use. To do this, I wrote a class called Train which lives in the maple.classifier module. You can peruse it here at your leisure if you want the full story. Here I’ll summarize some things.

This class starts in the __init__ method by loading in the label data as a dataframe.

...
self.label_data_path = Path(args.label_data)
self.label_data = pd.read_csv(self.label_data_path, sep='\t')
...

The label data looks like this:

session_id	event_id	subevent_id	t_start	t_end	label	date_labeled
2021_03_05_20_10_07	430	0	0.0	0.25	bark	2021-03-15 22:41:29.135011
2021_03_05_20_10_07	430	1	0.25	0.5	none	2021-03-15 22:41:44.855221
2021_01_19_13_59_02	39	0	0.0	0.25	howl	2021-03-15 22:42:07.451854
2021_01_19_13_59_02	39	1	0.25	0.5	howl	2021-03-15 22:42:08.434805
2021_01_19_13_59_02	39	2	0.5	0.75	howl	2021-03-15 22:42:09.842651
2021_01_19_13_59_02	39	3	0.75	1.0	howl	2021-03-15 22:42:10.959118
2021_01_19_13_59_02	39	4	1.0	1.25	howl	2021-03-15 22:42:12.299899
2021_01_19_13_59_02	39	5	1.25	1.5	howl	2021-03-15 22:42:13.635911
2021_01_19_13_59_02	39	6	1.5	1.75	howl	2021-03-15 22:42:44.608062
2021_02_11_16_56_39	7	0	0.0	0.25	whine	2021-03-15 22:42:55.567570
…	…	…	…	…	…	…

Notice that the actual audio data isn’t in here, it’s merely referenced by session_id, event_id, and subevent_id. In theory, I could have added the audio as another column like this:

audio
216,224,222,224,223,209,187,135,82,4…
-10,-3,13,33,37,16,8,22,31,52,73,8…
76,136,175,185,156,105,44,-40,-112,-…
192,196,200,209,213,228,242,244,251,…
39,41,39,59,59,55,52,67,103,130,137…
30,49,51,40,18,-27,-2,24,-39,-119,-…
-222,62,144,-71,-212,-45,152,65,-135,…
9,45,88,155,88,-47,-78,-37,0,-52,-1…
6,9,4,-20,-31,-34,-30,-27,-25,-18,-…

However this is pretty messy and a waste of space. Why duplicate the audio when it’s already stored in the session databases? Furthermore, I’ve written an API that makes accessing the data easy, so why not use it?. To associate these labels to their underlying audio data, I created a dictionary of SessionAnalysis instances, one for each session:

...
self.dbs = {}
session_ids = self.label_data['session_id'].unique()
for session_id in session_ids:
    self.dbs[session_id] = data.SessionAnalysis(name=session_id)
...

Then, I wrote methods to query the underlying audio data of any audio chunk, which I call a subevent:

def get_event_audio(self, session_id, event_id):
    return self.dbs[session_id].get_event_audio(event_id)


def get_subevent_audio(self, session_id, event_id, subevent_id):
    event_audio = self.get_event_audio(session_id, event_id)

    subevent_len = int(self.subevent_time * maple.RATE)
    subevent_audio = event_audio[subevent_id * subevent_len: (subevent_id + 1) * subevent_len]

    return subevent_audio

Now I can easily access the audio for any subevent. For example, the first label data is

session_id	event_id	subevent_id	t_start	t_end	label	date_labeled
2021_03_05_20_10_07	430	0	0.0	0.25	bark	2021-03-15 22:41:29.135011

The audio can be accessed with

self.get_subevent_audio('2021_03_05_20_10_07', 430, 0)

By the way, if you go down the rabbit hole, this audio data is ultimately obtained by an SQL query using the sqlite3 python API.

Preparing the data is done with self.prep_data.

def prep_data(self, transformation='spectrogram'):
    """Establishes the training and validation datasets

    This method sets the attributes `self.X`, and `self.y`

    Parameters
    ==========
    transformation : str, 'spectrogram'
        Pick any of {'spectrogram', 'none', 'fourier'}.
    """

    a = self.label_data.shape[0]

    if transformation == 'spectrogram':
        transformation_fn = self.get_subevent_spectrogram
        b = self.get_spectrogram_length()
    elif transformation == 'none':
        transformation_fn = self.get_subevent_audio
        b = self.get_audio_length()
    elif transformation == 'fourier':
        transformation_fn = self.get_subevent_fourier
        b = self.get_fourier_length()
    else:
        raise Exception(f"transformation '{transformation}' not implemented.")

    self.X = np.zeros((a, b))
    self.y = np.zeros(a).astype(int)

    for i, subevent in self.label_data.iterrows():
        label = self.label_dict[subevent['label']]
        self.y[i] = label

        self.X[i, :] = transformation_fn(
            session_id = subevent['session_id'],
            event_id = subevent['event_id'],
            subevent_id = subevent['subevent_id'],
        )

    if self.log:
        self.X = np.log2(self.X)

This method sets the attributes self.X and self.y. Each row of self.X is the data for each audio chunk, and each element of self.y is a numerical label corresponding to each of the 6 classes. transformation will define the contents of each row. If transformation='spectrogram', each row is the flattened spectrogram of an audio chunk. If transformation='fourier', each row is the Fourier amplitude spectrum of an audio chunk. And finally, if transformation='none', each row is the raw audio signal. I find the ability to train using any of these transformations very exciting because I can directly compare performance depending on which transformation is used.

With self.X and self.y defined, the model can be trained. This is done with self.fit_data which is a sklearn.ensemble.RandomForestClassifier wrapper.

def fit_data(self, *args, **kwargs):
    """Trains a random forest classifier and calculates an OOB model score.

    This method trains a model that is stored as `self.model`. `self.model` is a
    `sklearn.ensemble.RandomForestClassifier` object. The model score (fraction of correctly
    predicted validation samples) is stored as `self.model.xval_score_`

    Parameters
    ==========
    *args, **kwargs
        Uses any and all parameters accepted by `sklearn.ensemble.RandomForestClassifier`
        https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
    """

    self.model = RandomForestClassifier(*args, **kwargs)
    self.model.fit(self.X, self.y)

The model is trained and stored as self.model and a quality score using out-of-bag samples (a unique quality of the random forest classifier) is stored as self.model.oob_score_.

Once the model has been created, it can be saved for later use with self.save_model, which makes use of joblib.

    def save_model(self, filepath):
        """Saves `self.model` as a file with using `joblib.dump`

        Saves `self.model`, which is a `sklearn.ensemble.RandomForestClassifier` object, to
        `filepath`.  Before saving, some the `sample_rate`, `subevent_time`, `subevent_len`, and
        whether the spectrogram was log-transformed (`log`) are stored as additional attributes of
        `self.model`.

        Parameters
        ==========
        filepath : str, Path-like
            Stores the model with `joblib.dump`
        """

        self.model.log = self.log
        self.model.subevent_time = self.subevent_time
        self.model.sample_rate = maple.RATE
        self.model.subevent_len = int(self.model.subevent_time * self.model.sample_rate)
        joblib.dump(self.model, filepath)

This provides convenient access to the model so that at any time, the model can be loaded with

import joblib
model = joblib.load(path)

As a matter of convenience, I also store training and data parameters as attributes of self.model. For example, the sampling rate of the audio, the size of each data chunk, and the transformation type are all stored directly in self.model which makes this information available when the model is loaded at a later date.

The totality of the model-building procedure is glued together with self.run.

def run(self):
    """Run the training procedure

    This method glues the procedure together.
    """

    self.prep_data(spectrogram=True)

    self.fit_data(
        oob_score = True,
    )

    self.save_model(self.model_dir / 'model.dat')
    self.disconnect_dbs()

Performance

This is the fun part. How good is the model? I wanted to address this question from several standpoints.

Hyperparameter tuning: What are the optimal parameters for the model fit?
Data transformation comparisons: Does the spectrogram data outperform the raw audio signal? How about the Fourier signal?
Quantity of training data: Would labeling more data significantly increase accuracy or have I reached the point of diminishing returns?

But first, let’s just get a rough idea of how good the model is using out-of-the-box parameters, aka sklearn’s defaults. I extended the command line to include a train mode, so I can train the data like so.

./main.py train --label-data label_data.txt --model-dir model

This created a model file under model/model.dat. Loading up the model and printing out the out-of-bag validation score yields

import joblib
model = joblib.load('model/model.dat')
print(model.oob_score_)
>>> 0.8506375227686703

It correctly labelled 85% of the validation dataset. In my opinion, that’s really good given such a small training dataset.

Hyperparameter tuning

The default model parameters did a pretty good job, but let’s see if tuning the hyperparameters can improve performance.

The plan is to do a hyperparameter grid search to see what kind of model performance improvements can be achieved. However, before doing that there is one parameter of especial interest, since it greatly affects the speed of training: the number of trees.

Random forest classifiers create a forest of decision trees, and for achieving the highest accuracy, more trees is better. Let’s call the number of trees $n$. Increasing $n$ will statistically speaking increase model accuracy, however there is a law of diminishing returns and this comes at the cost of fit speed.

Choosing $n=10,000$ would be a bad idea for the hyperparameter grid search, since it would take minutes to generate each model. So to increase my search space, I wanted to pick the lowest number of trees possible so that model generation is fast, yet the results are good enough to make reliable decisions from. Then, the plan is to hold the number of trees constant for the grid search and vary all of the other hyperparameters.

So how many trees should I pick? I wrote a quick script that prepares the data using the Train class and then generates models for varying $n$. For each $n$ value I used a 3-fold cross validation to reduce the effect of overfitting to any particular data subset. Then, I plotted the mean and standard error of the cross-validation scores at each $n$ along with the mean time that it took to fit each model.

#! /usr/bin/env python
import argparse
import numpy as np
import matplotlib.pyplot as plt

from maple.classifier import Train
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

args = argparse.Namespace(label_data = 'label_data.txt', model_dir = 'none')
train = Train(args)
train.prep_data()

param_grid = {
    'n_estimators': np.logspace(start=0, stop=3, num=30, dtype=int),
}

model_search = GridSearchCV(
    estimator = RandomForestClassifier(),
    param_grid = param_grid,
    cv = 3,
    verbose = 2,
    n_jobs = -1,
)
model_search.fit(train.X, train.y)

# Plotting

model_df = pd.DataFrame(model_search.cv_results_)
n_trees = model_df['param_n_estimators'].values
mean_scores = model_df['mean_test_score'].values
std_err_scores = model_df['std_test_score'].values/np.sqrt(len(n_trees))
fit_times = model_df['mean_fit_time'].values
std_err_times = model_df['std_fit_time'].values/np.sqrt(len(n_trees))

fig, ax1 = plt.subplots()

color = 'tab:red'
ax1.set_title('Tradeoff of fit time vs accuracy')
ax1.set_xlabel('Number of trees')
ax1.set_ylabel('Cross-validation score', color=color)
ax1.set_xscale('log')
ax1.errorbar(n_trees, mean_scores, yerr=std_err_scores, capsize=4.0, fmt='--', color=color)
ax1.tick_params(axis='y', labelcolor=color)

ax2 = ax1.twinx()

color = 'tab:blue'
ax2.set_ylabel('Fit time [s]', color=color)
ax2.errorbar(n_trees, fit_times, yerr=std_err_times, capsize=4.0, fmt='--', color=color)
ax2.tick_params(axis='y', labelcolor=color)

fig.tight_layout()
plt.show()

Running the script produces a plot that illustrates the increase in accuracy (shown in red) as a function of $n$, the number of trees, at the expense of fit time (shown in blue).

So as expected, the more trees the better, but there is definitely a point of diminishing returns past around 20 trees. And this comes at substantial time cost, where in blue we can see the model fit time increases exponentially as a function of $n$.

Overall, these data are showing me it wouldn’t be a bad idea to use $n=30$ for the hyperparameter grid search.

To do the grid search, I took a look at the sklearn.RandomForestClassifier docs as well as this blog and ultimately decided on the following parameter grid:

param_grid = {
    'n_estimators': [30],
    'max_features': ['sqrt', 'log2'],
    'max_depth': list(np.arange(5, 100, 5).astype(int)) + [None],
    'min_samples_split': [2, 5, 10, 20],
    'min_samples_leaf': [1, 2, 4, 8],
}

That’s 640 different parameter combinations, and with a 5-fold cross-validation per parameter set, I needed to generate 3200 different models. That’s why picking the smallest $n$ possible is so important. I ran the grid search with the following script, which took about an hour to run (I estimate it would have taken 3 hours for $n=200$ and 9 hours for $n=500$).

#! /usr/bin/env python
import argparse
import numpy as np

from maple.classifier import Train
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV

args = argparse.Namespace(label_data = 'label_data.txt')
train = Train(args)
train.prep_data()

param_grid = {
    'n_estimators': [30],
    'max_features': ['sqrt', 'log2'],
    'max_depth': list(np.arange(5, 100, 5).astype(int)) + [None],
    'min_samples_split': [2, 5, 10, 20],
    'min_samples_leaf': [1, 2, 4, 8],
}

model_search = GridSearchCV(
    estimator = RandomForestClassifier(),
    param_grid = param_grid,
    cv = 5,
    verbose = 2,
    n_jobs = -1,
)
model_search.fit(train.X, train.y)

df = pd.DataFrame(model_search.cv_results_).sort_values(by='rank_test_score')
cols = ['mean_fit_time', 'mean_score_time', 'mean_test_score'] + [x for x in df.columns if x.startswith('param') and not x.endswith('param')]
df = df[cols]
df.to_csv('hyperparameter_tuning_results.txt', sep='\t', index=False)

The output of this script is a dataframe. First, I looked at the model performances by ordering them according to their rank, and noticed there appears to be 3 apparent regimes of model quality, that I’ve colored below.

In [1]: plt.plot(df['rank_test_score'], df['mean_test_score']); plt.xlabel('Rank'); plt.ylabel('Accuracy'); plt.show()

I wondered if there were any keystone parameters that partition the models into these three regimes. So I looked at the most common parameter value chosen for each of the three regimes.

In [2]: df.loc[df['rank_test_score'] < 20, 'regime'] = 'top'
In [3]: df.loc[(df['rank_test_score'] >= 20) & (df['rank_test_score'] < 600), 'regime'] = 'middle'
In [4]: df.loc[df['rank_test_score'] >= 600, 'regime'] = 'bottom'
In [5]: counts = df.groupby('regime')[['param_max_features', 'param_max_depth', 'param_min_samples_leaf', 'param_min_samples_split']].describe()
In [6]: counts = counts.iloc[:, counts.columns.get_level_values(1).isin(['count', 'freq', 'top'])]
In [7]: most_common = counts.iloc[:, counts.columns.get_level_values(1) == 'top'].droplevel(1, axis=1).transpose()
In [8]: most_common
Out[8]:
                        bottom middle   top
param_max_features        log2   log2  sqrt
param_max_depth              5     30    40
param_min_samples_leaf       8      4     2
param_min_samples_split     20     20     5

Surprisingly, this partitioning according to rank is 100% interpretable.

Take a look at max_features.

In [9]: most_common.loc['param_max_features']
Out[9]:
bottom    log2
middle    log2
top       sqrt
Name: param_max_features, dtype: object

The most commonly seen max_features in the top regime is sqrt, whereas the most common in the middle and bottom regimes both have max_features equal to log2. In fact, the best ranking model with max_features equal to log2 ranks 101st. This indicates that sqrt is strongly favored for the spectrogram transformation, and that max_features is a very important distinguishing hyperparameter.

Moving on to the other 3 parameters, here I’ve plotted the most common value seen for each regime.

In [10]: most_common.iloc[1:].plot.bar(rot=10); plt.tight_layout(); plt.ylabel('value'); plt.show()

On the far left is max_depth, where we see a clear increase in max_depth as the models get better and better. In fact, around 75% of models in the bottom regime have a max_depth of 5. Similarly for min_samples_leaf and min_samples_split, we see that the middle and bottom regimes consistently have high values, and the top regime has low values.

The consistent theme in the trends of max_depth, min_samples_leaf, and min_samples_split is that top-performing models do not want to be constrained in the number of leaves in the decision tree. This makes intuitive sense, but it’s nice to see it in the data.

Based on these findings, I’ve decided to update Train.fit_data to perform a small parameter scan with 10-fold cross validation. The new method looks like this:

def fit_data(self):
    """Trains a random forest classifier and calculates a model score.

    This method trains a bunch of models over a small subset of hyperparameter space based on an
    ad-hoc analysis described here:
    ekiefl.github.io/2021/03/14/maple-classifier/#-hyperparameter-tuning

    For each model setting, a 5-fold cross validation is used. When the most accurate model is
    determined, it is stored as `self.model` and the 5-fold cross validation accuracy is stored
    as self.model.xval_score_.
    """

    model_search = GridSearchCV(
        estimator = RandomForestClassifier(),
        param_grid = {
            'n_estimators': [200],
            'max_features': ['sqrt', 'log2'],
            'criterion': ['gini', 'entropy'],
        },
        cv = 10,
        verbose = 2,
        n_jobs = -1,
    )
    model_search.fit(self.X, self.y)

    self.model = model_search.best_estimator_
    self.model.xval_score_ = model_search.best_score_

Data transformation comparisons

In the data transformation section, I kept harping about how spectrograms would outperform time series audio (time-space) and Fourier spectra (frequency-space) because they resolve both time and frequency components simultaneously. Let’s see if that’s actually true.

Thanks to how general Train.prep_data is, this is really easy to test. To do so, I wrote this little script that calculates models with varying transformations.

#! /usr/bin/env python
import argparse
import pandas as pd

from pathlib import Path
from maple.classifier import Train

trans_settings = [
    ('spectrogram', False),
    ('spectrogram', True),
    ('fourier', False),
    ('fourier', True),
    ('none', False),
]

args = argparse.Namespace(label_data = 'label_data.txt')
train = Train(args)

scores = []
for trans, log in trans_settings:
    train.trans, train.log = trans, log
    train.model_dir = Path(f'model_{trans}' + ('_log' if log else ''))
    train.run(disconnect_dbs = False)
    scores.append(train.model.xval_score_)
train.disconnect_dbs()

results = pd.DataFrame({
    'transformation': list(zip(*trans_settings))[0],
    'log': list(zip(*trans_settings))[1],
    'score': scores
})
print(results.to_markdown())

I was also curious whether or not log-transforming the data could increase accuracy, so that’s in the script too.

The output is a table summarizing the prediction quality for various types of transformations on the audio data:

	transformation	log	score
0	spectrogram	False	0.852823
1	spectrogram	True	0.850638
2	fourier	False	0.825865
3	fourier	True	0.826958
4	none	False	0.632423

It’s great to verify what I’ve been advertising: the spectrogram outperforms the time- and frequency-space transformations. It’s interesting to see how poor the classifier performs when trained on the time series audio signal. As a final observation, log-transforming the data seems to have little effect on performance.

Quantity of training data

So hyperparameter tuning and investigating alternative data transformations did not yield significant increases in classification performance. The only remaining thing to investigate is: should I label more data?

A bigger training dataset is definitely better, but there comes a point where the reward does not match the effort. So before committing to labeling more data, which sucks, I wanted to know how much room there is for improvement. Basically, if $N$ is the size of my training dataset, and $S$ is the model score, I wanted to know where I lie on this learning curve:

To get an idea of where I am on the learning curve, I trained a series of models with a subset of my 2,745 label data and visualized the results. At each fraction of the full data, I’m doing a 5-fold cross validation for 5 trials, where the training data is resampled for each trial.

#! /usr/bin/env python
import argparse
import pandas as pd
import matplotlib.pyplot as plt

from maple.classifier import Train

args = argparse.Namespace(label_data = 'label_data.txt')
train = Train(args)
full_data = train.label_data.copy()

param_grid = {
    'n_estimators': [30],
},

fracs = np.linspace(0.1, 1, 50)
scores = np.zeros(len(fracs))

trials = 5

for i, frac in enumerate(fracs):
    s = np.zeros(trials)
    for t in range(trials):
        train.label_data = full_data.sample(frac=frac).reset_index(drop=True)
        train.prep_data(transformation='spectrogram')
        train.fit_data(param_grid=param_grid, cv=5)
        s[t] = train.model.xval_score_
    scores[i] = np.mean(s)

color = '#9E70B8'
plt.title('Should I keep labeling?')
plt.xlabel('Size of training dataset (N)')
plt.ylabel('Model score (S)', color=color)
plt.plot((full_data.shape[0]*fracs).astype(int), scores, c=color)
plt.yticks(np.round(np.arange(min(scores), max(scores)+0.01, 0.01), 2))
plt.show()

To my eye, it seems like there isn’t much to gain from labeling more data, but I really wanted to see if I could break 86% accuracy, so I decided that I would label up to $N=5000$.

To start labeling more data, I ran the command

./main.py label --label-data label_data.txt --session-paths training_session_paths

75 minutes later, there are now 5,031 labelled audio chunks. Repeating the analysis I ended up with this new learning curve:

The results did not change significantly. Certainly not enough to have justified 75 minutes. It seems like the increase in accuracy was about 0.5-1.0%, bringing the total accuracy just above 85%.

These plots were generated using 30 trees to cut down on run-time, so the final model with 200 trees will produce a slightly higher score.

Final model accuracy

After considering hyperparameter space, alternative data transformations, and doubling the amount of training data, I’ve exhausted all my options. It is time to run one last model: the one I’ll use moving forward:

./main.py train --label-data label_data.txt --model-dir model

This model is produced from a cross-validation grid search of potential models, which has no constraints on tree depth or leaf number, considers 2 alternatives for the max number of features each tree is built from (sqrt and log2), 2 alternatives for splitting criterion (gini and entropy), and a fixed tree number of 200. Each model’s score is determined from a 20-fold cross-validation scheme and the top-ranking model becomes the chosen model.

Loading up the resultant model and printing its score yields a final accuracy of 86.1%

import joblib
model = joblib.load('model/model.dat')
print(model.xval_score_)
>>> 0.8610715866691964

Section IV: Classifying

Training the model is one thing. Using it is another. In this section I will use the model to classify and visualize past audio events from the last several months.

The workhorse for all classification in maple is this simple class (source code):

class Classifier(object):
    def __init__(self, path):
        path = Path(path)
        if not path.exists():
            raise Exception(f'{path} does not exist')

        self.model = joblib.load(path)

        if self.model.scale:
            self.scaler = joblib.load(path.parent / 'scaler.dat')


    def predict(self, event_audio, as_label=False):
        """Given an arbitrary audio length, predict the class"""

        num_chunks = int(len(event_audio) / self.model.subevent_len)
        if not num_chunks:
            return 'none' if as_label else 0

        data = np.zeros((num_chunks, self.model.n_features_))
        for i in range(num_chunks):
            audio_chunk = event_audio[i * self.model.subevent_len: (i + 1) * self.model.subevent_len]
            data[i, :] = self.transform(audio_chunk)

        chunk_predictions = self.model.predict(data)

        # most common
        prediction = np.bincount(chunk_predictions).argmax()
        return labels[prediction] if as_label else prediction


    def transform(self, audio_chunk):
        if self.model.trans == 'spectrogram':
            data = audio.get_spectrogram(audio_chunk, fs=self.model.sample_rate, flatten=True)[2]
        elif self.model.trans == 'fourier':
            data = audio.get_fourier(audio_chunk, fs=self.model.sample_rate)[0]
        elif self.model.trans == 'none':
            data = np.copy(audio_chunk)
        else:
            raise Exception(f"Transformation '{selfmodel.trans}' not implemented for Classifier")

        if self.model.log:
            data = np.log2(data)

        if self.model.scale:
            data = self.scaler.transform(data.reshape(1, -1)).flatten()

        if self.model.norm:
            data = (data - data.mean()) / data.std()

        if np.isnan(data).all():
            # In diabolical cases an audio chunk may have all zeros, which result in nan's that break
            # the predict method.
            data = np.zeros(len(data))

        return data

Classifier is initialized by providing the path to a model. Because Train stored all of the transformation metadata as attributes of the model object (such as whether it was trained on spectrograms, Fourier spectra, or raw audio trans; whether it was log-transformed log; whether it was feature scaled scale; whether the samples were normalized norm), upon loading the model, Classifier knows exactly how it must transform incoming target data.

This means predicting the class of an audio event is as simple as passing audio data to Classifier.predict. When this happens, it slices the audio into chunks, calculates the appropriate transformation, and then classifies each chunk. The overall prediction for the entire event is chosen as the most frequently observed chunk classification. So if 3 chunks are predicted to be bark and 1 is predicted to be a whine, the prediction for the event is bark.

To classify events from past sessions, I extended the command line utility to include a classify mode which makes use of Classifier. It accepts a model directory as input, and optionally a list of session dbs to run the classifier on. It goes through each event of each session db, classifies the event, and stores the results in the class column of the events table. I ran it on all the session DBs with

./main.py classify --model-dir model

This command provides convenient means to classify or re-classify events with a single command, which is super convenient if one day I train a superior classifier and want to apply it to past datasets.

Verdict: pretty accurate

From the training validation I know the accuracy is around 86%, but I wanted to see the classifier in action.

So I loaded up a session database, which now houses fresh labels for each event

./main.py analyze -s 2021_02_13_19_19_33

and used SessionAnalysis.play_many to start comparing the predictions to the audio. Here is a spattering of what I found that included a diverse range of classes.

It’s clearly working pretty well.

Real-time classification

Now I have a way to classify events from past sessions. But moving forward, I would like to classify events as they occur so that the dog-sitter can make real-time decisions on whether to respond to the dog that are based on the class predictions. With the Classifier class in place, this is all I had to change.

Upon this change, all future audio events will be classified on-the-fly.

Speed

One concern about on-the-fly classification is speed. If classification is too slow, I may have to make some sacrifices. For example, tree traversal would be much faster if I decrease the number of trees in the random forest from 200 down to something more light-weight–say 30.

To measure the classification speed, I borrowed a timing class I had written for anvi’o a few years back and put it in maple.utils.

class TimeCode(object):
    """Time a block of code.

    This context manager times blocks of code.

    Examples
    ========
    >>> import time
    >>> import maple.utils as utils
    >>> # EXAMPLE 1
    >>> with utils.TimeCode() as t:
    >>>     time.sleep(5)
    ✓ Code finished successfully after 05s

    >>> # EXAMPLE 2
    >>> with terminal.TimeCode() as t:
    >>>     time.sleep(5)
    >>>     print(asdf) # undefined variable
    ✖ Code encountered error after 05s

    >>> # EXAMPLE 3
    >>> with terminal.TimeCode(quiet=True) as t:
    >>>     time.sleep(5)
    >>> print(t.time)
    0:00:05.000477
    """

    def __init__(self, quiet=False):
        self.s_msg = '✓ Code finished after'
        self.f_msg = '✖ Code encountered error after'
        self.quiet = quiet


    def __enter__(self):
        self.timer = Timer()
        return self


    def __exit__(self, exception_type, exception_value, traceback):
        self.time = self.timer.timedelta_to_checkpoint(self.timer.timestamp())
        return_code = 0 if exception_type is None else 1

        if not self.quiet:
            if return_code == 0:
                print(f"{self.s_msg} {self.time}")
            else:
                print(f"{self.f_msg} {self.time}")

Then, I wrote a script that loads up the audio events of an example session database, and measures the time it takes to classify each event for the classifier. Since I suspected that the number of trees would have a major influence on speed, I went ahead and did these measurements for a spattering of models generated with different tree counts.

#! /usr/bin/env python
import argparse

from maple.data import SessionAnalysis
from maple.utils import TimeCode
from maple.classifier import Train, Classifier

tree_nums = [10, 20, 40, 80, 160, 320, 640]
timing_df = {n: [] for n in tree_nums}

train = Train(argparse.Namespace(label_data = 'label_data.txt'))
train.prep_data()

session = SessionAnalysis(path='data/sessions/2021_02_13_19_19_33/events.db')

for n in tree_nums:
    # Train a model
    train.model_dir = f"model_n_{n}"
    train.setup_dir()
    train.fit_data(param_grid={'n_estimators': [n]}, cv=2)
    train.save()

    # Load the model
    classifier = Classifier(path=f"model_n_{n}/model.dat")

    for _, event in session.dog.iterrows():
        event_audio = event['audio']

        with TimeCode(quiet=True) as t:
            classifier.predict(event_audio)

        timing_df[n].append(t.time)

train.disconnect_dbs()

import seaborn as sns
import matplotlib.pyplot as plt

timing_df = pd.DataFrame(timing_df).melt(var_name='num trees', value_name='time [s]')
timing_df['time [s]'] = timing_df['time [s]'].dt.total_seconds()
timing_df.\
    groupby('num trees').\
    describe().\
    droplevel(0, axis=1).\
    assign(stderr = lambda x: x['std'] /x['count']**(1/2)).\
    plot(kind="bar", y="mean", yerr="stderr")

plt.title('Time-to-classify dependence on tree number')
plt.xlabel('Number of trees [n]')
plt.ylabel('Time [s]')
plt.tight_layout()
plt.show()

For a model with 200 trees, the cost of classifying during runtime is about 25ms of downtime following each event. Because my application isn’t multi-threaded, this time eats into time that should be dedicated to listening for the next event. Fortunately for me, 25ms is barely anything for my application. On the other hand, a tree count of 640 yields a 60ms downtime, which is starting to become problematic.

Based on these results, I am very happy to keep my model at a tree count of 200.

Section V: Visualizing

One utility of creating an audio classifier is being able to distill the entirety of a session into an intuitive visualization that illustrates how the dogs behaved. Last time I created an interactive interface that can be opened for any session from the command line.

./main.py analyze --session 2021_02_13_19_19_33

Before, it looked like this.

[Click for interactive plot]

Pretty cool. It shows every event and how loud the dogs were over time. It also shows owner responses, where green is praise and red is scold. But further decomposing the events into their classes would really paint a much more illustrative picture of what’s going on.

I decided that a good way to visualize the data would be to bin the events into minute timeframes and show the proportion of events belonging to each class in a stacked line chart. Ultimately, I decided on the following visual aesthetic:

[Click for interactive plot]

I create these plots with plotly and the source code can be found here.

With this visualization you can see at a glance how the dog’s behaved. For example, destroying my door seemed to be a passionate side project for Maple, where her progress is apparently driven by repeated bursts of inspiration.

Section VI: Updating response logic

Door scratching is the most undesirable behavior exhibited by the dogs. But in the above session, it went under the radar because it’s not very loud. This is the fundamental problem with the old decision-logic. Yet with real-time classification, now I can buff the decision logic with a more sentiment-based understanding of the events.

The possibilities for buffing the response logic are now endless. I could create different pools of owner responses like praise, scold, warn, console, encourage, etc. But in the interest of me graduating in a timely manner, I decided to keep things pragmatic: I am going to stick to just praise and scold, but I am going to update the decision logic that will take note of the event classes.

Praise

The old logic (above) took note of how many events were in the considered praise window, and enforced (1) that the total number of events should be below some threshold and (2) that none of the events should pass a certain pressure threshold.

This is pretty sound logic, but it fails under circumstances when the dogs produce noise that isn’t inherently bad. Currently, play is the only class that would fall under this category. To accommodate for the fact that noise is not always bad, the new logic is now the same, except all play events in the praise window are removed prior to the logic shown in the above flowchart. This way, the dogs can play and still be praised.

Scold

The old logic (above) took note of the total level of noise, whether the most recent event was particularly loud, and if both of these values pass their respective thresholds, the dog-sitter scolds. Again, this is an oversimplification because noise is not always bad.

After re-analyzing sessions, I came to the following new workflow.

Instead of measuring the total amount of sound pressure, I now count the number of barks. This is first thing that must be triggered to consider a scold. If the number of barks in the scold window is met, then I further demand that that the last $n$ events were all barks. This is just giving them the benefit of the doubt. And then, as a final gift of generosity, I demand that the last event passes some noise threshold. If all these conditions are met, the dog-sitter scolds.

So that’s the workflow for barking. I also wanted to handle door scratching since it is so destructive, so there is another flowchart for scratching.

More simply, the dog-sitter fires off a scold if a certain number of door scratching events are found within the scold window.

Today we’re leaving the house, and these poor doggies are going to be left alone for about 5 hours. A perfect opportunity to test the new decision logic.

Trial run

I nervously set up the system, locked the dogs in the bedroom, and quietly shut the door.

And 5 hours later the results are in. This is a big milestone: first session that implemented real-time audio classification and sentiment-based decision logic.

[Click for interactive plot]

Considering it was the longest we had left them alone in months, they behaved so well. I’m proud of them. And to be honest, I’m proud of the virtual dog-sitter too, because I think it acted quite rationally.

So I see some door scratching. Were any of the 4 scold responses triggered by door scratching? Here is a zoom-in of the second scold that was properly triggered in response to a series of door scratching events.

Great! And as cherry on top, the scold seems to have deterred the behavior, at least for a brief minute.

As another example, here is a zoom-in of the first 10 minutes that we were gone, where things got crazy.

Upon being alone, they are initially struck with worry. Worry that they can’t see their mommy, worry that they heard the front door close, worry that they are no longer in control. This manifests in their vocalizations as whining, and I think it corresponds to them fearing the worst, but trying to keep it together because they want to be good dogs.

Yet as their reality sets in, worry and disbelief turn to panic and desperation. This spiral in mindset is clearly seen as the sound landscape transitions from whining-dominant to barking-dominant. Now unhinged, they continue barking until the dog-sitter decides that enough is enough and scolds them. This seems to snap the dogs back to reality, as evidenced by the abrupt halt in barking.

So the new decision logic for scolding is triggering at the right time, and appears to have a real influence on their behavior.

While the last example shows some very interesting data from a technical standpoint, this is a rather depressing illustration of separation anxiety, something very common in dogs that’s driven by an unhealthy emotional dependence on their owners.

Separation anxiety is essentially the fear of being alone. Currently the dog-sitter addresses separation anxiety by creating the illusion that they are not actually alone. Instead, auditory responses of Kourtney and I give a convincing facade that big brother is watching. This lets them hear our voices and promotes the concept that the same behavorial rules apply even when we are not physically present–which provides them with a sense of normalcy.

So currently separation anxiety is being addressed mostly in an illusory manner. Yet moving forward, I would like the dog-sitter to countercondition the fear associated with being alone, with a newfound excitement of being alone. Currently, all they get for being good is an auditory praise, but what if I created something that dispenses treats instead? A surplus of treats (contingent on good behavior) is certainly a silver lining for being left alone, and would help countercondition a lot of the separation anxiety. If I take this project further, this is the direction I will head.

Conclusion

In summary, this work builds upon the maple software by implementing a random forest classifier that can identify common sounds produced by Maple and Dexter (whine, bark, etc). By classifying events in real time, I was able to develop more robust decisions on how the dog-sitter should respond to the dogs.

All of this new functionality has been implemented in the code base, and the totality of changes made can be viewed in this git compare.

See you next time.

Classifying dog barks for my dog-sitting software was originally published by Evan Kiefl at Evan Kiefl on March 14, 2021.

The algorithmic theory behind pool/billiards simulation

2020-12-20T00:00:00+00:00

Intro

What is a pool simulator?

In the last post I discussed the physics for all the different phenomena in pool and outlined equations of motion for each scenario. Yet there is still a critical missing piece: how and when should these equations be applied? For instance, I have equations to resolve the collision between two balls, but how do I know which two balls collided and when?

A pool simulator is more than just a sum of physics equations. Critically, a pool simulator requires an algorithm that coordinates the proper usage of these equations, and glues them together to evolve a shot from the moment the cue ball is struck to the moment the last ball stops moving. This algorithm is what I call the evolution algorithm. The evolution algorithm advances the system state from some initial time $t_i$, to some final time $t_f$.

In this post I define the system state, and then discuss two evolution algorithms: (1) discrete time evolution, and (2) continuous event-based evolution.

What is the system state?

In the last post I defined the ball state by 3 vectors: the ball’s displacement $\vec{r}(t)$, velocity $\vec{v}(t)$, and angular velocity $\vec{\omega}(t)$. Together, these three vectors fully characterize the state of a ball at any given time $t$. The system state for a given time $t$ is just the collection of individual ball states at that time.

Definition: system state

Mathematically, the ball state for the $i\text{th}$ ball is

\[s_i(t) = \{ \vec{r}_i(t) \, , \vec{v}_i(t), \, \vec{\omega}_i(t) \} \notag\]

where $\vec{r}_i(t)$ is the ball’s position, $\vec{v}_i(t)$ is the ball’s velocity, and $\vec{\omega}_i(t)$ is the ball’s angular velocity. The system state for a table with $N$ balls at an arbitrary time $t$ is then

\[S(t) = \{ s_i(t) \, \, \forall i \in \{1, ..., N\} \} \notag\]

The evolution algorithm calculates the system for a desired $t$ based on the system state at an earlier time. With this definition formalized, let’s move on to discuss options for a shot evolution algorithm.

Two options: discrete or continuous

Discrete time evolution

The premise of discrete time evolution is to advance the system state in very small discrete steps of time. After each timestep events can be detected and resolved. Did a ball collide with another ball, or with a cushion? If so, resolve the ball states, then advance to the next system state.

This is a really straight-forward evolution algorithm that is very convenient, and simple to understand. Unfortunately, discrete time evolution introduces intrinsic error that reduces accuracy of when events occur. An example is outlined in Figure 1.

Figure 1. Collision detection using discrete time evolution. The collision is detected at the timestep when the 2 balls overlap slightly (marked as x).

In the above scenario, the blue ball gets incrementally closer to the red ball with each timestep. Upon the third time step, the blue and red balls overlap slightly, triggering the detection of a collision. Herein lies the fundamental problem with discrete time evolution: collision events are never predicted, but rather detected after they were already supposed to have happened. So in the above example, the calculated collision time is $3 \Delta t$, even though the actual collision time is less than that–more like $2.8 \Delta t$.

The inaccuracy introduced by discrete time evolution can always be ameliorated by making $\Delta t$ smaller and smaller, yet there will always exist a margin of error on the order of $\Delta t$.

Furthermore, decreasing $\Delta t$ comes at massive computational cost. For example if you decrease the timestep 100-fold—sure, you increase the accuracy 100-fold—but you also increase the computational complexity 100-fold, since now you’ve gotta step through 100 times more time steps.

This can become brutal when dealing with pool simulations. Ball speeds commonly reach up to $10$ m/s, and a reasonable requirement for realism is that 2 balls should never intersect more than $1/100$th of a ball radius ($0.3$ mm). The required timestep for this level of realism is then $30$ microseconds. If the time from the cue strike to the last ball rolling is 10 seconds, that is $30,000$ time steps in total for one shot. If the level of realism is $1/1000$th a ball radius, that is $300,000$ time steps. Yikes.

The problem with this is all of the wasted computation… I don’t need $30$ microsecond time steps when all of the balls are far apart and barely moving. It is only really in select scenarios, such as a pool break, that realism demands such miniscule time steps.

To save computation, a smart discrete time evolution scheme would cut down on the number of time steps by making them adaptive depending on the state of the system. There are an infinite number of ways you could develop heuristics for an adaptive time stepper, that may be based on the distances between balls (if they are far apart, increase the time step), or based on velocities (if they are moving fast, decrease the time step). I’m not even going to go there because the possibilities are endless.

Aside: sometimes, you have to use discrete time evolution

Discrete time evolution is sometimes a necessary evil in many-body systems when equations of motion cannot be solved analytically. For example, the three-body problem (3 planets exhibiting gravitational forces on one another) has no analytical formulae that can express the positions of the planets as a function of time. That’s because the system is just so complex:

Trajectories for three planetary bodies exhibiting gravitational forces on one another must be found through discrete time evolution, since no analytical solutions exist. Source

Fortunately, for any given state of the system, the forces governing the equations are calculable (inverse square law), and this is all that is needed for discrete time evolution. If $\Delta t$ is chosen to be small enough, one can consider the force between the $i\text{th}$ and $(i+1)$th time step to be constant and since $\vec{F}=m\vec{a}$, that means acceleration is constant. We can discretely integrate this equation once and then again to yield how the velocity and position should be updated for the $(i+1)$th timestep:

\[\vec{a}_{i+1} = \vec{F}/m \notag\] \[\vec{v}_{i+1} = (\vec{F}/m) t + \vec{v}_i \notag\] \[\vec{r}_{i+1} = \frac{(\vec{F}/m)}{2}t^2 + \vec{v}_{i+1} t + \vec{r}_i \notag\]

Voila, we have a discrete time evolution algorithm for advancing the system state.

Continuous event-based evolution

The continuous event-based evolution algorithm was first developed by Leckie and Greenspan in a seminal paper entitled An Event-Based Pool Physics Simulator. If you would like to hear it straight from the horse’s mouth, a free pre-print of this publication is available here.

In the last post I presented a bunch of analytical equations of motion for ball trajectories, depending on whether the ball is stationary, spinning, rolling, sliding, or airborne.

These equations are perfect, because we can use them to evolve the ball states to any arbitrary time. The only (vital) problem is that events such as the collision in Figure 1 disrupt their validity, since they are developed assuming the ball acts in isolation.

For example, a ball rolling with a very high velocity may be destined to travel 50m in an isolated environment, yet collides with a cushion after just a few meters. Thus, the ball can safely be evolved via the rolling equations of motion up until the moment of collision, at which point the event must be resolved via the ball-cushion interaction equations. After the collision is resolved, the ball can be safely evolved up until the next event.

At this point, my mind was made up. I was going to use the continuous event-based algorithm for its speed and accuracy.

The algorithm, in its entirety

The rest of this post is concerned with detailing each and every aspect of this algorithm.

(1) Bird’s eye view

The prescription of evolving the equations of motion up until the next event forms the basis of the continuous event-based evolution algorithm. Simply stated, the algorithm goes like this:

Continuous event-based evolution algorithm

Set $t \rightarrow 0$.
Calculate the time $\Delta t_E$ until the next event. If $\Delta t_E = \infty$, stop.
Evolve the system state $S(t) \rightarrow S(t + \Delta t_E)$.
Resolve the event.
Set $t \rightarrow t + \Delta t_E$.
Repeat steps 2-5.

Let’s walk through a loop of the algorithm. Assume the time is currently $t$, such that the system state is $S(t)$.

The first step is determining when the next event occurs. By next event, I mean that no other event precedes it. Assume this occurs an amount of time $\Delta t_E$ from the current time. Calculating $\Delta t_E$ is non-trivial, and my guess is that around 95% of the algorithm’s computational complexity is allocated to this specific task. The intricacies of this process will be described ad nauseam in the next section, but until then, assume $\Delta t_E$ is calculable.

Once calculated, the next step is to advance the system state from $S(t)$ to $S(t + \Delta t_E)$. This means updating each ball’s state to $s_i(t + \Delta t_E)$, where the equations of motion are determined depending on whether the ball is stationary, spinning, rolling, sliding, or airborne. Since we know that no other event occurs before $\Delta t_E$, we can safely evolve their equations to $t + \Delta t_E$, but no further.

Then, the states of any balls involved in the event must be resolved. Some examples: If the event is two balls colliding, their states are resolved via the ball-ball interaction equations. If the event is a ball contacting a cushion, its state is resolved via the ball-cushion interaction equations. If the event is a ball transitioning from sliding to rolling, its state is resolved by updating its equations of motions from sliding to rolling.

Once the states of the balls involved in the event have been solved, every ball now has equations of motion that will be valid until the next event occurs. Thus, the process repeats itself.

So that’s how the algorithm works, and it is superior to the discrete time evolution algorithm because one can advance directly to the next event. A typical shot may only have as little as 5 events, and calculating when they occur is orders of magnitude less computation than the hundreds of thousands of time steps used in discrete time evolution for a typical shot. Not to mention the only error in the continuous event-based evolution algorithm is due to floating point precision.

This is a beautifully simple algorithm, but its utterly useless if we can’t calculate $\Delta t_E$, the time until the next event–and that’s no easy task. That’s what I tackle in the next sections. First, I formally define what an event is and outline all possible events. Then, I define the strategy and in excruciating detail outline how to calculate the time until the next event.

(2) What are events?

In the context of the continuous event-based evolution algorithm, an event is defined as anything that invalidates the equations of motion for 1 or more balls, due to something that is unaccounted for in the equations of motion, such as a collision with another ball. In total, there are 8 events, 4 of which are collisions, 4 of which are transitions. I discuss each below.

ball-ball collision

Unlike every other event, the ball-ball collision involves 2 balls, one of which must be in a translating motion state: rolling, sliding, or airborne. The other ball may be in any motion state: stationary, spinning, rolling, sliding or airborne. The collision is governed by the ball-ball interaction equations.

Figure 2. The possible inputs and outputs of the ball-ball collision event.

If the incoming motion state isn’t airborne, then neither will be the outgoing state. Conversely, if the incoming motion state is airborne, so will be the outgoing motion state. In other words, a table-bound ball cannot become airborne due to a ball-ball collision. I admit this might be confusing if you consider what happens when an airborne ball collides with a ball on the table:

They both end up airborne. Since the line of centers between the two balls is directed into the table, the outgoing velocity of the table-bound ball has a component into the table, which initiates a ball-slate collision. Consequently, the table-bound ball (the 3-ball) becomes airborne. In the continuous event-based algorithm I’ve developed, the ball becomes airborne due to the ball-slate collision, not the ball-ball collision, which are separated by an infinitesimally small amount of time. As a network, the totality of this process looks like this:

Figure 3. Network representation of events and ball motion states for the ball-ball collision slow-mo video.

ball-cushion collision

The ball-cushion collision involves just one ball, which must be in a translating motion state: rolling, sliding, or airborne. The output state is either sliding or airborne. The collision is governed by the ball-cushion interaction equations.

Figure 4. The possible inputs and outputs of the ball-cushion collision event.

The only way to have an outgoing airborne motion state is if the incoming motion state is also airborne. Just like in the ball-ball collision, you may find this confusing, since after a collision with a cushion, you often see it airborne:

Yet just like as described in the ball-ball collision, the ball becomes airborne due to the ball-slate collision, which occurs an infinitesimally small amount of time after the ball-cushion collision. You can see the above shot represented as an interaction:

Figure 5. Network representation of events and ball motion states for the ball-cushion slow-mo video

ball-slate collision

The ball-slate collision occurs when a ball contacts the table with a velocity component into the table ($-z-$direction). The collision is governed by the ball-slate interaction equations.

Stationary, spinning, and rolling balls necessarily have 0 speed along the $z-$axis, so the only available input motion states are sliding and airborne.

So when do we see ball-slate collisions? An airborne ball undergoes a ball-slate collision whenever it contacts the table. Additionally, we’ve seen in Figures 3 and 5 that sliding balls can also have non-zero velocities in the $z-$direction (caused by ball-ball and ball-cushion collisions that immediately precede the ball-slate collision), and also undergo ball-slate collisions.

The first possible outgoing motion state of the ball-slate interaction is being airborne. The second possible outgoing motion state is sliding, which occurs when the ball doesn’t have enough velocity to become airborne.

Figure 6. The possible inputs and outputs of the ball-slate collision event.

ball-pocket collision

Of all the events, this is definitely the most fabricated. You may hear ball-pocket collision and think about all the complexities of the ball leaving the slate and falling into the pocket, bouncing off the sides of the pocket, etc. but that’s not what I mean by ball-pocket collision. In fact, I don’t really plan to model that.

Instead, the ball-pocket collision is merely a way to determine if a ball is pocketed and if it is, removing it from the simulation. Basically, if a ball undergoes a ball-pocket collision, the ball is considered pocketed. It went in the hole. Good job.

To resolve the collision, the ball is simply removed from the simulation.

transition events

In Figures 2-6, I’ve shown that collisions often induce motion state transitions, however in each of those cases the transition is caused by the collision. Yet motion state transitions also occur naturally:

In the above short clip, the ball transitions from spinning to stationary. This is considered an event, because upon becoming stationary, the ball’s equations of motion (spinning) become invalid and need to be updated to the stationary equations of motion.

As a matter of interest, if the equation was not updated, it dictates that the ball would begin spinning in the reverse direction, gaining more and more rotational kinetic energy indefinitely. This is obviously non-physical, and illustrates the need to properly handle motion state transition events.

Definition: transition event

A transition event is when a ball’s motion state naturally transitions to another motion state, i.e. without being catalyzed by a collision.

Figure 7. A transition event network is very simple. X goes in and Y goes out.

Mathematically, there are $5^4 = 1024$ transitions, but only 4 occur naturally. Thus there are only 4 transition events:

spinning-stationary
rolling-stationary
rolling-spinning
sliding-rolling

Fundamentally, these transitions happen due to a loss of energy due to friction. So these transitions are always from high energy to low energy.

I’ve actually omitted 2 very rare transition events. In theory, a ball can transition from sliding to stationary if the perfect amount of backspin is applied. Same goes for sliding to spinning. So technically, there are 2 more transition events:

5. sliding-stationary (very rare)

6. sliding-spinning (very rare)

Yet, there is 0 margin for error for these scenarios. In virtually all simulated cases, a ball will exit the sliding state with non-zero speed (using 64-bit floats with SI units yields femtometer $(10^{-15})$ per second precisions). For this reason, the sliding-stationary and sliding-spinning transition events can in all practical senses be decomposed into 2 transition events separated by some very small amount of time. For example, the sliding-stationary transition can be decomposed into the sliding-rolling transition followed by a rolling-stationary transition.

Figure 8. The sliding-stationary transition event can be decomposed into the sliding-rolling followed by the rolling-stationary transition events.

This wraps up my description of the possible event types.

(3) The strategy

We now know all the possible types of events. So what’s the strategy for calculating when the next event occurs? As it turns out, the most feasible strategy is to explicitly calculate the time until every possible next event. If you calculate the time until literally every possible next event, then by definition the one that occurs in the shortest amount of time is the one that physically occurs next.

The strategy for calculating $\Delta t_E$

To calculate $\Delta t_E$, we calculate all possible events:

Calculate all ball-ball collision event times
Calculate all ball-cushion collision event times
Calculate all ball-slate collision event times
Calculate all ball-pocket collision event times
Calculate all ball transition event times
$\Delta t_E$ is the smallest of all calculated event times

If $\Delta t_E$ is infinite, there is no next event, which happens in the unique scenario that the system is in its lowest energy state (all balls are stationary).

Toy example

Consider the example pictured in Figure 9. Ball A is sliding, balls B & C are stationary, and there are no other balls on the table. For simplicity, assume there are no cushions or pockets and that the table surface extends indefinitely. To calculate the next event, we need to consider all possible events. So let’s do that.

Figure 9. A system with 1 sliding ball (A) and 2 stationary balls (B & C). The solid-filled balls indicate where the balls currently are. There are 3 next events: (1) Ball A can transition from sliding to rolling $(\Delta t^{(1)})$, (2) ball A can collide with ball B $(\Delta t^{(2)})$, or (3) ball A can collide with ball C $(\Delta t^{(3)})$.

ball-cushion collision events. Since there are no cushions, there are no ball-cushion collision events to consider.
ball-pocket collision events. Let’s also assume there are no pockets, so no ball-pocket collision events to consider either.
ball-slate collision events. Balls B and C are stationary, so cannot undergo a ball-slate collision event. If ball A has a velocity component into the table, then the next event would be a ball-slate collision, since it would occur at $t=0$. But for the sake of example, let’s assume ball A has no velocity component into the table. Then there are no ball-slate collision events to consider.
ball-ball collision events. There are 3 potential ball-ball collision events: between balls A & B, A & C, and B & C. However, since 2 stationary balls cannot collide, the B & C collision is an impossible event, and only A & B, and A & C collisions need to be considered.
transition events. Since balls B and C are stationary, there are no available transition events–they are already in their lowest energy states. On the other hand, Ball A is sliding, so a sliding-rolling transition event is a possibility for Ball A.

In summary, there are 3 events to consider: (1) a sliding-rolling transition for ball A, (2) a ball-ball collision with balls A and B, and (3) a ball-ball collision with ball A and C.

Based on how I’ve drawn Figure 9, it’s obvious that the next event is the sliding-rolling transition, yet in general the time for each of these events must be calculated explicitly. If ball A was spinning a lot, ball A could collide with ball B whilst spinning, which would make the collision with ball B the next event. In fact, we can’t even rule out that a collision with C is not the next event, which could happen if ball A has a curved trajectory that goes around ball B.

So the point is this: given a system state $S(t)$, the only way to determine the next event $\Delta t_E$, is to explicitly calculate the time until all possible next events. The next event is the one that occurs in the smallest amount of time.

This means we must develop means to calculate the time until every possible next event, which is the subject of the next section. Buckle your seatbelts.

(4) Calculating possible event times

Transition times

Transition times are the easiest to calculate, so I’ll start with them.

The spinning-stationary transition is defined by the moment at which a spinning ball reaches 0 angular velocity in the $z-$direction. From the spinning equations of motion, the angular velocity as a function of time is given by

\[\omega_z(t) = \omega_{0z} - \frac{5\mu_{sp}g}{2R}t \notag\]

Setting this to 0 and solving for time yields the spinning-stationary transition time:

Time until spinning-stationary transition event

\[\Delta t_E = \frac{2R}{5 \mu_{sp} g} \omega_{0z} \label{spinning_stationary_time}\]

where $\omega _{0z}$ is the current angular velocity in the $z-$direction.

Reminder, this equation is significant because for a given time $t$, the spinning-stationary event for every spinning ball must be considered as a potential next event.

On to the next. Both the rolling-stationary events and the rolling-spinning events are defined by the moment at which a ball’s center of mass velocity reaches 0. From the rolling equations of motion, the velocity of a rolling ball as a function of time is given by

\[\vec{v}(t) = \vec{v}_0 - \mu_r g t \hat{v}_0 \label{rolling_velocity} \notag\]

Taking the magnitude of both sides, setting the LHS to 0, and solving for time yields the rolling-stationary and rolling-spinning transition times:

Time until rolling-(stationary/spinning) transition events

\[\Delta t_E = \frac{v_0}{\mu_{r} g} \label{rolling_spinning_stationary_time}\]

where $v_0$ is the current ball speed.

If $\omega _{z}(\Delta t_E) = 0$, the event is a rolling-stationary transition event. Otherwise, it is a rolling-spinning transition event.

Reminder, this equation is significant because for a given time $t$, the rolling-spinning and rolling-stationary transition events for every rolling ball must be considered as a potential next event.

Finally, we got the sliding-rolling transition, which is defined by the moment at which the relative velocity $\vec{u}$ becomes $\vec{0}$. From the sliding equations of motion, the relative velocity as a function of time is given by

\[\vec{u}(t) = (u_0 - \frac{7}{2} \mu_s g t ) \, \hat{u}_0 \notag\]

Taking the magnitude of both sides, setting the LHS to 0, and solving for time yields the sliding-rolling transition time:

Time until sliding-rolling transition event

\[\Delta t_E = \frac{2}{7} \frac{u_0}{\mu_s g} \label{sliding_rolling_time}\]

where $u_0$ is the current magnitude of the relative velocity.

Reminder, this equation is significant because for a given time $t$, the sliding-rolling transition event for every sliding ball must be considered as a potential next event.

That’s all of the event transition times. Short and sweet.

Ball-ball collision times

We are on the search for calculating all possible next events.

A very large subcategory of next possible events is all of the ball-ball collisions that may occur. In fact, if there are 15 balls on the table, there are $_ {15}\text{C} _ {2} = 105$ different ball-ball collision events that may be the next event, and the time until each of them must be explicitly calculated. As we will see, many may take an infinite or non-real amount of time.

First, an important assumption needs to be discussed. Suppose the current time is $t$, and you want to calculate the time until collision, $\Delta t_E$, between two balls with states $s^{(i)}(t)$ and $s^{(j)}(t)$. I’ll call this the $i-j$ collision event. Within the time range $[t, t + \Delta t_E]$, I will assume neither of the balls are involved in any other event. This means neither of the 2 balls engage in any transitions or collisions. If either did, then the intervening event necessarily precedes the $i-j$ collision event, which means it doesn’t need to be considered as a candidate for the next event.

To predict when two balls colide, we need a collision condition. The collision occurs when the distance between the center of masses of the two balls is $2R$, where $R$ is the radii of the balls.

Figure 10. On overhead view of a collision trajectory of the $i\text{th}$ and $j\text{th}$ balls. The magnitude of the distance vector is $d _{ij}(t)$, and the collision occurs when $d _{ij}$ is $2R$.

Mathematically, we can track the distance between the two balls as a function of time by defining a distance vector $\vec{d}_{ij}(t)$:

\[\vec{d}_{ij}(t) = \vec{r}^{(j)}(t) - \vec{r}^{(i)}(t) \label{dist_vec_defn}\]

where $\vec{r}^{(i)}(t)$ is the position of the $i\text{th}$ ball and $\vec{r}^{(j)}(t)$ is the position of the $j\text{th}$ ball. The collision condition is thus

\[\lvert \vec{d}_{ij}(\Delta t_E) \rvert = 2R \label{collision}\]

where $\Delta t_E$ is the collision time. The forms of $\vec{r}^{(i)}(t)$ and $\vec{r}^{(j)}(t)$ will depend on the whether the balls are stationary, spinning, rolling, spinning, or airborne. Fortunately, we have equations of motion for each of these scenarios, each of which can be cast in the following form:

\[\vec{r}^{(i)}(t) = \begin{bmatrix} a_x^{(i)} t^2 + b_x^{(i)} t + c_x^{(i)} \\ a_y^{(i)} t^2 + b_y^{(i)} t + c_y^{(i)} \\ a_z^{(i)} t^2 + b_z^{(i)} t + c_z^{(i)} \end{bmatrix} \label{quad_r}\]

As you can see, each component can be expressed as a quadratic equation with respect to time (though in many cases the $a$, $b$, and $c$ coefficients are 0). The coefficients depend on which motion state the ball is in. For example, Eq. $\eqref{quad_r}$ for a rolling ball has the following coefficients:

\[a_x^{(i)} = - \frac{1}{2} \mu_r g \cos(\phi^{(i)}) \notag\] \[a_y^{(i)} = - \frac{1}{2} \mu_r g \sin(\phi^{(i)}) \notag\] \[a_z^{(i)} = 0 \notag\] \[b_x^{(i)} = v_0^{(i)} \cos(\phi^{(i)}) \notag\] \[b_y^{(i)} = v_0^{(i)} \sin(\phi^{(i)}) \notag\] \[b_z^{(i)} = 0 \notag\] \[c_x^{(i)} = r_{0x}^{(i)} \notag\] \[c_y^{(i)} = r_{0y}^{(i)} \notag\] \[c_z^{(i)} = 0 \notag\]

which was determined by looking at the the rolling equations of motion. As another example, looking at the stationary equations of motion reveals that a stationary ball has the following coefficients in Eq. $\eqref{quad_r}$:

\[a_x^{(j)} = a_y^{(j)} = a_z^{(j)} = 0 \notag\] \[b_x^{(j)} = b_y^{(j)} = b_z^{(j)} = 0 \notag\] \[c_x^{(j)} = r_{0x}^{(j)} \notag\] \[c_y^{(j)} = r_{0y}^{(j)} \notag\] \[c_z^{(j)} = 0 \notag\]

Regardless of the particulars, the critical point is that a ball’s trajectory is defined by these 9 coefficients. To determine if two balls collide, we can formulate the distance vector $\vec{d}(t)$ from Eq. $\eqref{dist_vec_defn}$ in terms of these 18 coefficients (2 balls, 9 coefficients each):

\[\vec{d}_{ij}(t) = \begin{bmatrix} A_x t^2 + B_x t + C_x \\ A_y t^2 + B_y t + C_y \\ A_z t^2 + B_z t + C_z \\ \end{bmatrix} \label{diff_vec}\]

where

\[A_x = a_x^{(j)} - a_x^{(i)} \label{A_x_ball}\] \[A_y = a_y^{(j)} - a_y^{(i)} \label{A_y_ball}\] \[A_z = a_z^{(j)} - a_z^{(i)} \label{A_z_ball}\] \[B_x = b_x^{(j)} - b_x^{(i)} \label{B_x_ball}\] \[B_y = b_y^{(j)} - b_y^{(i)} \label{B_y_ball}\] \[B_z = b_z^{(j)} - b_z^{(i)} \label{B_z_ball}\] \[C_x = c_x^{(j)} - c_x^{(i)} \label{C_x_ball}\] \[C_y = c_y^{(j)} - c_y^{(i)} \label{C_y_ball}\] \[C_z = c_z^{(j)} - c_z^{(i)} \label{C_z_ball}\]

Plugging Eq. $\eqref{diff_vec}$ into the collision condition Eq. $\eqref{collision}$ yields an equation whose roots describe the time until two arbitrarily balls collide:

Time until ball-ball collision event

Let $\Delta t_E$ be the time until collision between the $i\text{th}$ and $j\text{th}$ balls. $\Delta t_E$ is the smallest real and positive root to this polynomial equation:

\[(A_x^2 + A_y^2 + A_z^2) \, \Delta t_E^4 + \\ (2 A_x B_x + 2 A_y B_y + 2 A_z B_z) \, \Delta t_E^3 + \\ (B_x^2 + B_y^2 + B_z^2 + 2 A_x C_x + 2 A_y C_y + 2 A_z C_z) \, \Delta t_E^2 + \\ (2 B_x C_x + 2 B_y C_y + 2 B_z C_z) \, \Delta t_E + \\ C_x^2 + C_y^2 + C_z^2 - 4 R^2 = 0 \label{ball_poly}\]

where the coefficients are defined by Eqs. $\eqref{A_x_ball}$ - $\eqref{C_z_ball}$.

If there exists no real and positive root, then the balls do not collide.

Eq. $\eqref{ball_poly}$ can be solved analytically or numerically to reveal if/when the $i-j$ collision event occurs. This prescription can be applied for each ball-ball pair.

Ball-cushion collision times

Another subset of possible events are all of the ball-cushion collisions—and there are a lot of collisions to account for.

In my model, I deconstruct the entire cushion contact surface into a collection of straight-line cushion segments, and potential collisions for each ball must be assessed for every cushion segment. On a standard table with 6 pockets, there are 18 cushion segments (Figure 11). With 15 balls on the table, that is 270 potential ball-cushion collision events that must be explicitly considered as the next event.

Figure 11. On a standard table with 6 pockets, there are 18 straight-line segments that define the cushion contact surfaces.

Overall, the treatment is very similar to the ball-ball collision time calculation: develop an expression for the distance between the ball and the cushion as a function of time–based off the quadratic form of the ball’s trajectory–and solve for the collision condition. I do this two ways, one which is simple, and one which is very, very, very ugly but more accurate.

The first is simpler, and defines the ball-cushion collision as the moment the ball contacts a plane that extends vertically from the cushion segment’s edge. I call this the collision plane.

Figure 12. Visualization of collision planes (orange) that extend vertically from the cushion edges. The planes extend infinitely in the vertical direction (despite how they are pictured). A ball-cushion collision is assumed to occur whenever a ball contacts a collision plane.

This assumption is convenient and mostly accurate when balls remain on or nearly on the table, but creates false-positive collision events when balls in airborne motion states intersect the collision plane. Even if balls remain on or nearly on the table, there is a slight inaccuracy introduced that depends on how airborne the ball is, which is depicted in Figure 13. Nevertheless, convenience often outweighs accuracy.

Figure 13. A side view of the collision plane (orange). Two mock balls (red & blue) are pictured at the moment they contact the cushion edge. As seen, there is a discrepancy between when balls contact the collision plane versus the cushion edge.

The collision condition for a collision of the $i\text{th}$ ball with the $j\text{th}$ cushion segment occurs when the $i\text{th}$ ball contacts the $j\text{th}$ cushion segment’s collision plane. The most convenient way to define the collision plane is with two points on the $xy-$plane, which can be expressed as these two vectors:

\[\vec{p}_1^{(j)} = \langle p_{1x}^{(j)}, \, p_{1y}^{(j)}, \, 0 \rangle \label{p1_plane}\] \[\vec{p}_2^{(j)} = \langle p_{2x}^{(j)}, \, p_{2y}^{(j)}, \, 0 \rangle \label{p2_plane}\]

The collision plane is the plane that extends vertically from the line connecting $\vec{p}_1^{(j)}$ and $\vec{p}_2^{(j)}$. We can also define the collision plane as a classic $y = m x + b$ scenario by using these points:

\[y = m x + b \notag\]

Channel your high school math education to express $m$ and $b$ in terms of the points:

\[y = \frac{p_{2y}^{(j)} - p_{1y}^{(j)}}{p_{2x}^{(j)} - p_{1x}^{(j)}} x + p_{1y}^{(j)} - \frac{p_{2y}^{(j)} - p_{1y}^{(j)}}{p_{2x}^{(j)} - p_{1x}^{(j)}} p_{1x}^{(j)} \notag\]

Finally, cast everything onto the left-hand side:

\[l_x^{(j)} x + l_y^{(j)} y + l_0^{(j)} = 0 \notag\]

where

\[l_x^{(j)} = -\frac{p_{2y}^{(j)} - p_{1y}^{(j)}}{p_{2x}^{(j)} - p_{1x}^{(j)}} \\ l_y^{(j)} = 1 \\ l_0^{(j)} = \frac{p_{2y}^{(j)} - p_{1y}^{(j)}}{p_{2x}^{(j)} - p_{1x}^{(j)}} p_{1x}^{(j)} - p_{1y}^{(j)} \label{ls}\]

Alright, the shortest distance between a point $(p_x, p_y, p_z)$ and the collision plane is defined by

\[d = \frac{\lvert l_x^{(j)} p_x + l_y^{(j)} p_y + l_0^{(j)} \rvert}{\sqrt{l_x^{(j)\, 2} + l_y^{(j)\, 2}}} \label{dist_to_plane}\]

(Source)

This describes the distance between the point $(p_x, p_y, p_z)$ and a plane that extends infinitely in the $xy-$plane, rather than a plane that terminates at the points $\vec{p}_1^{(j)}$ and $\vec{p}_2^{(j)}$. We will have to deal with this oversight in a minute.

Rather than use an arbitrary point $(p_x, p_y, p_z)$, here I throw in the trajectory of the $i\text{th}$ ball from Eq. $\eqref{quad_r}$:

\[d_{ij}(t) = \frac{\lvert l_x^{(j)} r_x^{(i)}(t) + l_y^{(j)} r_y^{(i)}(t) + l_0^{(j)} \rvert}{\sqrt{l_x^{(j)\, 2} + l_y^{(j)\, 2}}} \notag\] \[d_{ij}(t) = \frac{\lvert l_x^{(j)} (a_x^{(i)} t^2 + b_x^{(i)} t + c_x^{(i)}) + l_y^{(j)} (a_y^{(i)} t^2 + b_y^{(i)} t + c_y^{(i)}) + l_0^{(j)} \rvert}{\sqrt{l_x^{(j)\, 2} + l_y^{(j)\, 2}}} \notag\]

where $d_{ij}$ is the distance between the $i\text{th}$ ball and the $j\text{th}$ cushion segment.

Figure 14. A top down view of a collision trajectory of the $i\text{th}$ ball with the $j\text{th}$ cushion segment. The magnitude of the distance vector is $d _{ij}(t)$, and the collision occurs when $d _{ij}$ is $R$–in other words when the ball contacts the collision plane (green line).

With an expression that describes the distance between the $i\text{th}$ ball and the $j\text{th}$ cushion segment, I can mathematically express the collision condition by setting $d_{ij}$ to $R$, which by definition happens at a time $\Delta t_E$:

\[\lvert l_x^{(j)} (a_x^{(i)} \Delta t_E^2 + b_x^{(i)} \Delta t_E + c_x^{(i)}) + l_y^{(j)} (a_y^{(i)} \Delta t_E^2 + b_y^{(i)} \Delta t_E + c_y^{(i)}) + l_0^{(j)} \rvert - \\ R\sqrt{l_x^{(j)\, 2} + l_y^{(j)\, 2}} = 0 \notag\]

So this is the collision condition, which is quadratic with respect to time. Calculating the roots of this quadratic will reveal if and when a collision occurs. Specifically, if there are any real, positive roots to the equation, a collision between the $i\text{th}$ ball and $j\text{th}$ cushion is a potential next event.

There is just one critical problem. This collision condition assumes the collision plane extends infinitely in the $xy-$plane, rather than terminating at the points $\vec{p}_1^{(j)}$ and $\vec{p}_2^{(j)}$, each which define the cushion segment’s physical boundaries. This is bad news, because there will be many false-positive collision events for a given cushion segment if this is not dealt with. Check out Figure 15.

Figure 15. A ball transits the playing surface and encounters two potential cushion segments. The first cushion segment is defined by the points $(p _{1x}^{(1)}, p _{1y}^{(1)})$ and $(p _{2x}^{(1)}, p _{2y}^{(1)})$ and represents the long rail. The second cushion segment is defined by the points $(p _{1x}^{(2)}, p _{1y}^{(2)})$ and $(p _{2x}^{(2)}, p _{2y}^{(2)})$ and represents the left jaw of the side pocket. The ball’s collision with the first cushion segment is legitimate because $0 \le s^{(1)} \le 1$, whereas the collision with the second cushion segment is illegitimate because $s^{(2)} > 1$.

The collision plane of cushion segment 2 defines the left jaw of the side pocket. It begins at $(p _{1x}^{(2)}, p _{1y}^{(2)})$ and ends at $(p _{2x}^{(2)}, p _{2y}^{(2)})$. But because of Eq. $\eqref{dist_to_plane}$’s flawed assumption, the collision plane mathematically extends into the playing surface, and triggers false collision events with unsuspecting passersby, e.g. the 11-ball.

Obviously this won’t do. Fortunately, the solution is self-apparent: if the alleged collision occurs outside the boundary defined by the physical portion of the collision plane, disregard it. I needed to express this mathematically, and I did it by parameterizing a cushion segment’s collision plane with the following representation:

\[\vec{p}^{(j)} = \begin{bmatrix} p_{1x}^{(j)} + (p_{2x}^{(j)} - p_{1x}^{(j)}) \, s \\ p_{1y}^{(j)} + (p_{2y}^{(j)} - p_{1y}^{(j)}) \, s \\ z \end{bmatrix} \label{parameterized_plane}\]

Since it is a vertical plane, $z$ is a free parameter

$s$ is the parameter that defines where on the plane you lie. It’s basically a slider scale that goes from $-\infty$ to $+\infty$. But very importantly, this parameterization is defined such that $s=0$ corresponds to the coordinate $\langle p_{1x}^{(j)}, p_{1y}^{(j)}, z \rangle$ and $s=1$ corresponds to the coordinate $\langle p_{2x}^{(j)}, p_{2y}^{(j)}, z \rangle$. In other words, the range $s \in [0, 1]$ defines the physical portion of the collision plane!

So to see whether or not any collision predicted by the collision condition is legitimate, we need only find the $s$ value at the time of collision. If $0 \le s \le 1$, the collision is legitimate, and otherwise it is not and can be ignored.

According to Eq. 3 of this source, the value of $s$ at the collision time can be calculated via the following equation:

\[s = - \frac{ (\vec{p}_1^{(j)} - \vec{r}^{(i)}(t + \Delta t_E)) \bullet (\vec{p}_2^{(j)} - \vec{p}_1^{(j)}) }{ (\vec{p}_2^{(j)} - \vec{p}_1^{(j)}) \bullet (\vec{p}_2^{(j)} - \vec{p}_1^{(j)}) } \label{s_at_plane}\]

Now we are fully equipped to detect ball-cushion collisions.

Time until ball-cushion collision event (I) Let $\Delta t_E$ be the time until collision between the $i\text{th}$ ball and $j\text{th}$ cushion segment, where the collision is detected via the collision plane. $\Delta t_E$ can be obtained by solving the roots of the following quadratic polynomial

\[A \Delta t_E^2 + B \Delta t_E + C = 0 \label{cushion_poly_1}\]

where

\[A = l_x^{(j)} a_x^{(i)} + l_y^{(j)} a_y^{(i)} \notag\] \[B = l_x^{(j)} b_x^{(i)} + l_y^{(j)} b_y^{(i)} \notag\] \[C = l_0^{(j)} + l_x^{(j)} c_x^{(i)} + l_y^{(j)} c_y^{(i)} \pm R \sqrt{l_x^{(j)\, 2} + l_y^{(j)\, 2}} \notag\]

where $l_x^{(j)}$, $l_y^{(j)}$, and $l_0^{(j)}$ are defined by Eq. $\eqref{ls}$. For each real, positive root, it should be determined if $0 \le s \le 1$, where $s$ is given by Eq. $\eqref{s_at_plane}$. In Eq. $\eqref{s_at_plane}$, $\vec{p} _{1}^{(j)}$ and $\vec{p} _{2}^{(j)}$ are given by Eqs. $\eqref{p1_plane}$ and $\eqref{p2_plane}$ and each of their $z$ components should be set to $\vec{r} _z^{(i)}(t + \Delta t_E)$ so that there is no $z$ contribution to $s$. When calculating Eq. $\eqref{s_at_plane}$, the candidate root should be substituted in for $\Delta t_E$.

The smallest, real, positive root where $0 \le s \le 1$ is the time until collision. If no real, positive root satisfies this requirement, the $i\text{th}$ ball and $j\text{th}$ cushion segment do not collide.

As a more advanced treatment, I now consider a more physically accurate case in which the cushion segment is treated as a line that coincides with the cushion segment’s edge, depicted in orange in Figure 16.

Figure 16. A ball in contact with a cushion. Mathematically, the ball-cushion collision is detected by defining the cushion’s edge, highlighted in orange.

Defining a collision line rather than a collision plane allows complete accuracy in determining collision times and permits balls to fly off the table, thereby avoiding all false-positive ball-cushion collision events that occur with a collision plane.

The most convenient way to define the collision line of a cushion segment is to specify two points, where the line connecting them is the collision line. The points for the collision line of the $j\text{th}$ cushion segment can be defined with the following vectors

\[\vec{p}_1^{(j)} = \langle p_{1x}^{(j)}, \, p_{1y}^{(j)}, \, h \rangle \label{p1_line}\] \[\vec{p}_2^{(j)} = \langle p_{2x}^{(j)}, \, p_{2y}^{(j)}, \, h \rangle \label{p2_line}\]

This time the $z-$component is not free, but rather fixed at the height of the cushion, $h$. Here I assume $h$ is constant, but it may be fun exploring a relaxation of this assumption later. With these 2 vectors defined, the line can be parameterized via the following vector:

\[\vec{p}^{(j)} = \begin{bmatrix} p_{1x}^{(j)} + (p_{2x}^{(j)} - p_{1x}^{(j)}) \, s\\ p_{1y}^{(j)} + (p_{2y}^{(j)} - p_{1y}^{(j)}) \, s\\ h \end{bmatrix} \label{p1_vec}\]

where $s$ determines where on the line you are.

Just as the distance of a point to the collision plane is given by Eq. $\eqref{dist_to_plane}$, we need an equation for the distance of a point to the collision line.

If such a point is described by the vector $\vec{p}_0 = \langle p _{0x}, p _{0y}, p _{0z} \rangle$, then the minimum distance to the collision line is

\[d = \frac{\lvert (\vec{p}_0 - \vec{p}_1^{(j)}) \times (\vec{p}_0 - \vec{p}_2^{(j)}) \rvert} {\lvert \vec{p}_2^{(j)} - \vec{p}_1^{(j)} \vert} \notag\]

(Source) where $\times$ denotes the cross product. By replacing $\vec{p}_0$ with the trajectory of the $i\text{th}$ ball, we get the distance of the $i\text{th}$ ball to the $j\text{th}$ cushion segment as a function of time.

\[d_{ij}(t) = \frac{\lvert (\vec{r}^{(i)}(t) - \vec{p}_1^{(j)}) \times (\vec{r}^{(i)}(t) - \vec{p}_2^{(j)}) \rvert} {\lvert \vec{p}_2^{(j)} - \vec{p}_1^{(j)} \vert} \label{dist_to_line}\]

Like before, the collision condition is defined by setting $d_{ij}$ to $R$, which by definition happens at time $\Delta t_E$. After substituting Eq. $\eqref{quad_r}$ into Eq. $\eqref{dist_to_line}$, the algebra blows up.

This page experiences a lot of lag if I try to render this math as an html object, so I took the above screenshot instead. Click here for the html version.

It’s beauty no doubt challenges that of Euler’s $e^{i\pi} + 1 = 0$. Whether or not you agree doesn’t take away from the fact that this is a legitimate quartic polynomial with respect to time, the roots of which describe the time until collision of ball $i$ and cushion segment $j$.

The only thing that needs to be done is grouping the different orders together so the solution can be presented.

Time until ball-cushion collision event (II)

Let $\Delta t_E$ be the time until collision between the $i\text{th}$ ball and $j\text{th}$ cushion segment, where the collision is detected via the collision line. $\Delta t_E$ can be obtained by solving the roots of the following quartic polynomial:

\[A \Delta t_E^4 + B \Delta t_E^3 + C \Delta t_E^2 + D \Delta t_E + E = 0 \label{cushion_poly_2}\]

where $A$ is given by this equation, $B$ is given by this equation, $C$ is given by this equation, $D$ is given by this equation, and $E$ is given by this equation.

For each real, positive root, it should be determined if $0 \le s \le 1$, where $s$ is given by Eq. $\eqref{s_at_plane}$. In Eq. $\eqref{s_at_plane}$, $\vec{p} _{1}^{(j)}$ and $\vec{p} _{2}^{(j)}$ are given by Eqs. $\eqref{p1_line}$ and $\eqref{p2_line}$. When calculating Eq. $\eqref{s_at_plane}$, the candidate root should be substituted in for $\Delta t_E$.

Wow.

Ball-slate collision times

In comparison to the ball-cushion collision, this is a breeze.

The ball-slate collision occurs either when a sliding ball has a $-z-$velocity component, or an airborne ball hits the slate. The first case is somewhat pendantic, because if it has a $-z-$velocity component, the time until the ball-slate collision is by definition 0. So the real calculation is determining when an airborne ball hits the slate.

An airborne ball hits the slate when $r_z = R$, where $R$ is the ball’s radius. According to the airborne equations of motion, $r_z(t)$ is given by

\[r_z(t) = r_{0z} + v_{0z} t - \frac{1}{2} g t^2 \notag\]

Setting this to $R$ and solving for time yields the ball-slate collision time:

Time until ball-slate collision event

Let $\Delta t_E$ be the time until the ball-slate collision event for a given ball.

If the ball is sliding,

\[\Delta t_E = \begin{cases} 0, & \text{if $v_{0z} < 0$} \\ \infty, & \text{otherwise} \end{cases} \notag\]

If the ball is airborne,

\[\Delta t_E = \frac{ v_{0z} + \sqrt{v_{0z}^2 + 2 g (r_{0z} - R)} }{ g } \notag\]

Reminder, this equation is significant because for a given time $t$, the ball-slate collision event for every sliding and airborne ball must be considered as a potential next event.

Ball-pocket collision times

So as mentioned, the ball-pocket collision exists as a means to determine when a ball is pocketed, and to remove the ball from the simulation when it is pocketed. Defining the ball-pocket collision condition requires a suitable geometry for pockets, which I am going to be very rudimentary about.

Basically, the $j\text{th}$ pocket is going to be a circle with center $(a^{(j)}, b^{(j)})$ and radius $R^{(j)}$. Here’s an illustration because I spoil you:

Figure 17. A depiction of the pocket geometry (shown in orange) for the $j\text{th}$ pocket. The center is $(a^{(j)}, b^{(j)})$ and the radius is $R^{(j)}$.

First of all, we are able to exclude any balls that are stationary, spinning, or airborne. So we are only including balls that are rolling or sliding. Excluding airborne balls simplifies things considerably, since we can assume that $r_z^{(i)}(t)$ has a constant value of $R$, and therefore we can treat the collision condition as a 2D problem.

Actually, I’d like to modify the above statement slightly. If you absolutely slam a ball into a pocket, it is likely to be slightly airborne, if only by a millimeter. Yet it still falls into the pocket because the back lip of the pocket is at a height higher than the ball’s radius, thus redirecting the ball into the pocket. Let’s call this lip height $h_M$ and assume that any ball with $r_z(t)$ less than $h_M$ is pocketed, and any ball greater than $h_M$ isn’t. An obviously crude means of modelling what in reality is a complex rigid body dynamics problem that doesn’t easily lend itself to continuous event-based evolution.

Based on my own table, it seems like a reasonable value for $h_M$ is $7R/5$.

To muster up a mathematical collision condition between the $i\text{th}$ ball and the $j\text{th}$ pocket, I define the distance between the ball and the center of the pocket:

\[d_{ij}(t) = \sqrt{(r_x^{(i)}(t) - a^{(j)})^2 + (r_y^{(i)}(t) - b^{(j)})^2} \notag\]

The collision is assumed to occur at the moment the ball rolls over the table’s edge:

Figure 18. Side profile of a pocket and ball during the moment of the ball-pocket collision.

At this moment, $d_{ij}(\Delta t_E) = R^{(j)}$, where $R^{(j)}$ is the radius of the pocket, not the ball. Like we’ve been doing for the ball-ball and ball-cushion collisions, we can plug this into the left-hand-side of the distance equation and solve for $\Delta t_E$:

Time until ball-pocket collision event

Let $\Delta t_E$ be the time until collision between the $i\text{th}$ ball and the $j\text{th}$ pocket.

If the ball is spinning or stationary, $\Delta t_E = \infty$.

Otherwise, $\Delta t_E$ can be obtained by solving the roots of the following quartic polynomial:

\[A \Delta t_E^4 + B \Delta t_E^3 + C \Delta t_E^2 + D \Delta t_E + E = 0 \label{pocket_poly}\]

where

\[A = \frac{1}{2} ( {a_x^{(i)}}^2 + {a_y^{(i)}}^2 ) \notag\] \[B = a_x^{(i)} b_x^{(i)} + a_y^{(i)} b_y^{(i)} \notag\] \[C = a_x^{(i)} (c_x^{(i)} - a^{(j)}) + a_y^{(i)} (c_y^{(i)} - b^{(j)}) + \frac{1}{2} ({b_x^{(i)}}^2 + {b_y^{(i)}}^2) \notag\] \[D = b_x^{(i)} (c_x^{(i)} - a^{(j)}) + b_y^{(i)} (c_y^{(i)} - b^{(j)}) \notag\] \[E = \frac{1}{2} ({a^{(j)}}^2 + {b^{(j)}}^2 + {c_x^{(i)}}^2 + {c_y^{(i)}}^2 - R^2) - (c_x^{(i)} a^{(j)} + c_y^{(i)} b^{(j)}) \notag\]

Scrap paper of the calculation

I’m sorry I used $a$ and $b$ both for ball trajectory coefficients $(^{(i)})$ and for the pocket coordinates $(^{(j)})$. Writing out these equations, I now realize how brutal this is.

For each positive, real root, the height of the ball at the time $t + \Delta t_E$ should be calculated, i.e. $r_z(t + \Delta t_E)$.

The smallest real, positive root that satisfied $r_z(t + \Delta t_E) \le h_M$ is the time until collision. If no real, positive roots satisfy this requirement, the $i\text{th}$ ball and $j\text{th}$ pocket do not collide.

(5) Summary

That was a lot of work. I think now would be a good time to consolidate all of this information about the continuous event-based evolution algorithm into a brief summary.

The train of thought can be boiled down into just 5 statements.

We have equations of motion for every ball
But events disrupt the validity of these equations
We can safely evolve all balls up until the next event occurs
To find the next event, we calculate all possible event times
The one that occurs in the least amount of time is the next event

Essentially, we have developed these analytic equations of motion that are defined for all of the ball motion states, which assume the ball acts in complete isolation. Of course, there are events such as collisions that are not accounted for in these equations of motion that must be detected and resolved.

To make sure events don’t screw up our equations of motion, we can only evolve the balls’ states up until the next event that occurs on the table. It is worth clarifying that even if a collision acts between balls 1 and 2, ball 3 can only be evolved up until the time at which balls 1 and 2 collide.

Once evolved up to the event, the event must be resolved via the appropriate physics equations. Then, its rinse and repeat and the balls’ states can be evolved up until the next event.

Calculating when the next event occurs involves calculating all possible events between all possible objects. By definition, the event that occurs in the smallest amount of time is chosen as the next event, and all other candidate events are discarded.

(6) Extras

This section you can find some additional topics you may find of interest.

Number of candidate events?

How many candidate events are there to solve for at each step of the algorithm?

This depends on the number of pockets, cushions, balls, as well as the states the balls are in. But for the sake of example, let’s consider a standard table with 6 pockets, and a standard set of 15 balls + the cue ball. As you will see later on, I decompose the cushion surface into straight line segments, each of which must be tested for ball-cushion collisions independently. For a standard table with 6 pockets, there are 18 such segments (Figure 11). So the makeup of agents that are potentially involved in events is as follows:

16 balls
18 cushion segments
6 pockets

With these numbers, let’s calculate just how many potential event times we have to solve for at each step of the continuous event-based evolution algorithm.

(1) How many potential transition events?

In general, this will depend on each ball’s state. For example, there are no transition events available for a stationary ball, but there are two transition events available for a rolling ball (rolling-spinning, rolling-stationary). For simplicity, let’s assume that each ball has 1 transition state available to it. This might be the case immediately after the break, when many balls are in motion. In that case, we have ~16 potential transition events

(2) How many potential ball-ball events?

If there are $N$ balls on the table, collisions between each combination must be checked. That means there are $N (N-1) / 2$ potential ball-ball collisions to check. For 16 balls that’s 120 potential events.

Of course, solving for the roots of the associated quartic polynomial for each of these collision cases is not necessary, because when both balls are either stationary or spinning, the collision can immediately be ruled impossible.

(3) How many potential ball-cushion events?

With 18 cushion segments and 16 balls, each ball must be tested against each cushion segment explicitly. That’s $18 \times 16=288$ potential ball-cushion events. This one really hurts, but decomposing the cushion surface into linear segments is a necessary evil.

Just like in the ball-ball collision, solving for the roots of the associated quadratic or quartic polynomial is only necessary if the ball is translating.

(4) How many potential ball-slate events?

Ball-slate collision events are relatively rare. They are only relevant when a ball is airborne and/or has a velocity component into the table. So calculating a ball-slate collision is only necessary under these circumstances, which is going to be a pretty rare occurrence, requiring explicit calculation perhaps just a handful of times throughout the entire evolution of a shot. Just to associate a number, let’s estimate that we have ~1 potential transition event.

(5) How many potential ball-pocket events?

With 16 balls and 6 pockets, each ball must be tested against each pocket explicitly. That means $16 \times 6=96$ potential ball-pocket events.

Just like the ball-cushion events, calculating the associated quartic polynomial can be avoided for any stationary or spinning balls.

(6) How many in total?

Summing these numbers up yields an upper bound for the number of potential event times which must be explicitly computed during each loop of the evolution algorithm (for a standard table and ball count). In total, we are looking at ~521 events. A typical shot may have anywhere from 10 to 50 events, meaning that in total, we are looking within the range of 5,000-25,000 event time calculations per shot.

Closing remarks

When I first read the Leckie and Greenspan’s paper, understanding it required hours of reading with pencil and paper in hand. It’s not really their fault–it was a well written paper. The problem is that it was written in an academic format, which favors correctness and succinctness over understandability. I hope that this post along with the other serve as a more approachable and more fleshed out perspective on pool simulation theory.

This marks the last theory post in this blog series, and I gotta say, I am really thankful to be done writing about pool simulation theory. These posts are brutal to write due to their technical nature, so I’m glad that the next posts will be centered around actual implementation of a pool simulator, which will include a lot more code and visualization.

The algorithmic theory behind pool/billiards simulation was originally published by Evan Kiefl at Evan Kiefl on December 20, 2020.

Creating a virtual dog sitter with live audio processing

2020-07-20T00:00:00+00:00

I recently moved in with my girlfriend (Kourtney) who has a 6-month old puppy named Maple. She’s a sweet girl (the puppy), however she was born in the COVID era, and that comes with some behavioral challenges. The biggest problem is that she’s developed an unrealistic assumption that she can be with us 100% of the time. And when that unrealistic expectation is challenged by us leaving the apartment, she barks. Loudly. I wanted to create an application written in Python that monitors and responds to her barks.

By the end of this post, the program (which you have full, open-access to) will detect dog barks using PyAudio and make decisions on whether to praise or scold the dog based on its behavior. This is all done in real time. At this point, both praising and scolding means playing a pre-recorded voice of the owner that is either of positive or negative sentiment. The audio, statistics, and time of each bark, as well as statistics of owner responses are stored in a SQLite database. Finally, I’ll show some interactive plots of the results using Plotly.

Simple demo

It’s always good to start simple so I just searched for python live audio processing and found this blog post by Scott W Harden. In it, he posts a simple script that monitors in real-time the amplitude of the audio signal. I turned it into a class structure:

#! /usr/bin/env python

import numpy as np
import pyaudio
import argparse

CHUNK = 2**11
RATE = 44100

class LiveStream(object):
    def __init__(self, args = argparse.Namespace()):
        self.args = args

        self.p = pyaudio.PyAudio()
        self.stream = None


    def start(self):
        while True:
            data = np.fromstring(self.stream.read(CHUNK), dtype=np.int16)
            self.process_data(data)


    def __enter__(self):
        self.stream = self.p.open(
            format = pyaudio.paInt16,
            channels = 1,
            rate = RATE,
            input = True,
            frames_per_buffer = CHUNK,
        )

        return self


    def __exit__(self, exception_type, exception_value, traceback):
        self.stream.stop_stream()
        self.stream.close()
        self.p.terminate()


    def process_data(self, data):
        peak=np.average(np.abs(data))*2
        bars="#"*int(2000*peak/2**16)
        print("%05d %s"%(peak,bars))

        # uncomment to push highest amplitude frequency to stdout
        #x = data
        #w = np.fft.fft(x)
        #freqs = np.fft.fftfreq(len(x))
        #max_freq = abs(freqs[np.argmax(w)] * RATE)
        #peak = max_freq
        #bars="o"*int(4000*peak/2**16)
        #print("%05d %s"%(peak,bars))


if __name__ == '__main__':
    with LiveStream() as s:
        s.start()

Here’s a demo:

(Browse code)

Event detection

With a bare-bones script that demos a live output of audio amplitude and/or frequency, it was time to get serious: detecting events. I wanted to be able to detect events since in order to rationally respond to your dog, you need to be able to detect when it is barking.

Events are basically anomalies in the background noise, and so to detect events, we need to properly distinguish background from signal. To do this, I wrote a calibration method that runs at the start of the program. The premise is to wait until the audio signal reaches an equilibrium, and then measures the mean ($ \mu $) and standard deviation ($ \sigma $) of the signal strength. I consider equilibrium to be established by demanding that the coefficient of variation ($ \sigma/\mu $) is less than some threshold value, since a low coefficient of variation means a stable signal. The background mean and standard deviation that satisfied this constraint for equilibrium can then be used to distinguish signal from noise.

We can do some back of the envelope calculations to show that if the signal is drawn from a Normal distribution (a bell-shaped curve), there is a ~16% chance that any given datapoint in the signal will exceed 1 standard deviation about the mean ($\mu + \sigma$). That probability becomes ~2.5% that it will exceed 2 standard deviations ($\mu + 2\sigma$) and ~0.5% that it exceeds 3 standard deviations ($\mu + 3\sigma$). As a first step, I went ahead and wrote a detector that detects the start of an event whenever the signal exceeds 3 standard deviations, and the end of the event whenever it dips below 2 standard deviations. Here is a demo showing the efficacy of this approach:

From here on in, I started developing within a multi-file environment (the whole repo can be found here). To make everything accessible as possible, each code snippet and demo video will be followed by a hyperlink that brings you to the stage in the codebase from where the snippet or demo was taken.

(Browse code)

Better event detection

As mentioned in the video, there are a lot of false-positives for the ends of events. In other words, many of my spoken sentences were being split up into multiple events, even when there was little or no break in my speaking rhythm. This was happening because the criterion for events ending was too simple, and based on a single audio frame (reminder: the event ends if the mean audio signal of an audio frame drops below $\mu + 2\sigma$, where $ \mu $ and $ \sigma $ are the mean and standard deviation of the background noise). This is problematic because each frame is only a couple of milliseconds. To more accurately depict the start and stop of events, I wanted to create criteria that spanned multiple frames.

In the above video, the program has just 1 state variable called in_event, and its possible values are either True or False. Here is a flowchart of the transitions possible:

Soon we’ll see that making a more robust event detector inevitably complicates this simple picture.

Event start criterion

In the video, transitioning from in_event == False to in_event == True occurred whenever a frame had a mean signal 3 standard deviations above the mean. Let’s call this threshold value $X$ for convenience. My new criterion is that there must be $N$ consecutive frames that are all above $X$. If $N$ consecutive frames all meet this threshold, then the event start is attributed to the first frame in this frame sequence. Requiring consecutive frames to pass the threshold effectively guards against false-positives when loud but short (~ millisecond) sounds are made, which trigger an event. The stringency is thus controlled by $N$: the lower $N$ is, the more false-positives in event starts you allow.

Programmatically, whenever a frame’s mean signal exceeds $X$ while in_event == False, a second state variable in_on_transition is set to True. Whenever in_on_transition == True, it basically means, “Ok, we’re not for sure in an event, but things are starting to get loud, and if we stay in this state for long enough, we’ll for sure know we are in an event”. If in_on_transition == True for $N$ consecutive frames, then the program is convinced it is actually an event, so in_event is set to True and in_on_transition is returned to False. On the other hand, if any frame fails to exceed $X$, in_on_transition is set to False and the potential event is deemed not an event. To avoid clipping the start of the event because the detector is busy making its mind up, whenever in_on_transition == True, the audio frames are stored in a buffer and retroactively added to the event.

Event end criterion

Before, an event ended whenever a frame had a mean signal that dipped below 2 standard deviations above the mean ($\mu + 2\sigma$). Call this threshold $Y$ for convenience. My new method uses the same criterion, but rather than ending the event when this occurs, a countdown of $T$ seconds starts. If $T$ reaches 0, the event ends. But it is possible to “save” the event, if any frame during the countdown has a mean signal exceeding $X$ (the event start threshold). In this case, the event will continue until the next time the mean signal dips below $Y$. The countdown safeguards against against false-positives that end events because there was a momentary lapse in sound amplitude. The stringency is thus controlled by $T$: the lower $T$ is, the more false-positives in event ends you allow.

Programmatically, whenever a frame’s mean signal dips below $Y$ while in_event == True, a second state variable in_off_transition is set to True. This starts a countdown. If any frame during the countdown exceeds $X$, in_off_transition is set to False, and the event is given life anew. But if no frames exceed $X$ during the countdown, the event is deemed to have finished, so in_event is set to False and in_off_transition is returned to False.

With the implementation of these new criteria, here is the updated state logic visualized as a flow chart:

It’s a little more complicated, but there is a pleasant symmetry.

Results

Here is a demo of the new event detector.

(Browse code)

Overall, I am happy with how it works and ready to move on.

A generic audio detector

I noticed at this point that nothing I have done so far has anything to do with dogs and barking. Being noncommittal is a great quality in a codebase because it creates flexibility. For the purpose of making this useful to anyone with their own applications, I created a well-polished branch of the repository that can be used for generic audio event detection. I reorganized everything into a single file, so after installing numpy and pyaudio, you are ready to rumble:

#! /usr/bin/env python

import numpy as np
import pyaudio

CHUNK = 2**11
RATE = 44100


def calc_power(data):
    """Calculate the power of a discrete time signal"""

    return np.mean(np.abs(data))*2


class Stream(object):
    def __init__(self):
        self.p = pyaudio.PyAudio()

        self._stream = self.p.open(
            format = pyaudio.paInt16,
            channels = 1,
            rate = RATE,
            input = True,
            frames_per_buffer = CHUNK,
            start = False, # To read from stream, self.stream.start_stream must be called
        )


    def __enter__(self):
        if not self._stream.is_active():
            self._stream.start_stream()

        return self


    def __exit__(self, exc_type, exc_val, traceback):
        self._stream.stop_stream()


    def close(self):
        """Close the stream gracefully"""

        if self._stream.is_active():
            self._stream.stop_stream()
        self._stream.close()
        self.p.terminate()


class Monitor(object):
    def __init__(self, args = argparse.Namespace()):
        self.args = args

        self.dt = CHUNK/RATE # Time between each sample

        # Calibration parameters
        self.calibration_time = 3 # How many seconds is calibration window
        self.calibration_threshold = 0.50 # Required ratio of std power to mean power
        self.calibration_tries = 1 # Number of running windows tried until threshold is doubled

        # Event detection parameters
        self.event_start_threshold = 3 # standard deviations above background noise to start an event
        self.event_end_threshold = 2 # standard deviations above background noise to end an event
        self.seconds = 0.5
        self.num_consecutive = 4

        self.stream = None
        self.background = None
        self.background_std = None

        self.detector = None
        self.event_recs = {}
        self.num_events = 0


    def read_chunk(self):
        """Read a chunk from the stream and cast as a numpy array"""

        return np.fromstring(self.stream._stream.read(CHUNK), dtype=np.int16)


    def calibrate_background_noise(self):
        """Establish a background noise

        Calculates moving windows of power. If the ratio between standard deviation and mean is less
        than a threshold, signifying a constant level of noise, the mean power is chosen as the
        background. Otherwise, it is tried again. If it fails too many times, the threshold is
        increased and the process is repeated.
        """

        stable = False
        power_vals = []

        # Number of chunks in running window based on self.calibration time
        running_avg_domain = int(self.calibration_time / self.dt)

        with self.stream:
            tries = 0
            while True:
                for i in range(running_avg_domain):
                    power_vals.append(calc_power(self.read_chunk()))

                # Test if threshold met
                power_vals = np.array(power_vals)
                if np.std(power_vals)/np.mean(power_vals) < self.calibration_threshold:
                    self.background = np.mean(power_vals)
                    self.background_std = np.std(power_vals)
                    return

                # Threshold not met, try again
                power_vals = []
                tries += 1

                if tries == self.calibration_tries:
                    # Max tries met--doubling calibration threshold
                    print(f"Calibration threshold not met after {tries} tries. Increasing threshold "
                          f"({self.calibration_threshold:.2f} --> {1.5*self.calibration_threshold:.2f})")
                    tries = 0
                    self.calibration_threshold *= 1.5


    def setup(self):
        self.stream = Stream()

        self.calibrate_background_noise()

        self.detector = Detector(
            background_std = self.background_std,
            background = self.background,
            start_thresh = self.event_start_threshold,
            end_thresh = self.event_end_threshold,
            seconds = self.seconds,
            num_consecutive = self.num_consecutive,
            dt = self.dt,
        )

        self.wait_for_events()


    def wait_for_events(self):
        while True:
            self.wait_for_event()

            # Do anything you want here


    def wait_for_event(self, log=True):
        """Waits for an event, records the event, and returns the event audio as numpy array"""

        self.detector.reset()

        with self.stream:
            while True:
                self.detector.process(self.read_chunk())

                if self.detector.event_finished:
                    break

        return self.detector.get_event_data()


    def stream_power_and_pitch_to_stdout(self, data):
        """Call for every chunk to create a primitive stream plot of power and pitch to stdout

        Pitch is indicated with 'o' bars, amplitude is indicated with '-'
        """

        power = calc_power(data)
        bars = "-"*int(1000*power/2**16)

        print("%05d %s" % (power, bars))

        w = np.fft.fft(data)
        freqs = np.fft.fftfreq(len(data))
        peak = abs(freqs[np.argmax(w)] * RATE)
        bars="o"*int(3000*peak/2**16)

        print("%05d %s" % (peak, bars))


class Detector(object):
    def __init__(self, background_std, background, start_thresh, end_thresh, num_consecutive,
                 seconds, dt, quiet=False):
        """Manages the detection of events

        Parameters
        ==========
        background_std : float
            The standard deviation of the background noise.

        background : float
            The mean of the background noise.

        start_thresh : float
            The number of standard deviations above the background noise that the power must exceed
            for a data point to be considered as the start of an event.

        end_thresh : float
            The number of standard deviations above the background noise that the power dip below
            for a data point to be considered as the end of an event.

        num_consecutive : int
            The number of frames needed that must consecutively be above the threshold to be
            considered the start of an event.

        seconds : float
            The number of seconds that must pass after the `end_thresh` condition is met in order
            for the event to end. If, during this time, the `start_thresh` condition is met, the
            ending of the event will be cancelled.

        dt : float
            The inverse sampling frequency, i.e the time captured by each frame.

        quiet : bool
            If True, nothing is sent to stdout
        """

        self.quiet = quiet

        self.dt = dt
        self.bg_std = background_std
        self.bg_mean = background
        self.seconds = seconds
        self.num_consecutive = num_consecutive

        # Recast the start and end thresholds in terms of power values
        self.start_thresh = self.bg_mean + start_thresh*self.bg_std
        self.end_thresh = self.bg_mean + end_thresh*self.bg_std

        self.reset()


    def update_event_states(self, power):
        """Update event states based on their current states plus the power of the current frame"""

        if self.event_started:
            self.event_started = False

        if self.in_event:
            if self.in_off_transition:
                if self.off_time > self.seconds:
                    self.in_event = False
                    self.in_off_transition = False
                    self.event_finished = True
                elif power > self.start_thresh:
                    self.in_off_transition = False
                else:
                    self.off_time += self.dt
            else:
                if power < self.end_thresh:
                    self.in_off_transition = True
                    self.off_time = 0
                else:
                    pass
        else:
            if self.in_on_transition:
                # Not in event
                if self.on_counter >= self.num_consecutive:
                    self.in_event = True
                    self.in_on_transition = False
                    self.event_started = True
                elif power > self.start_thresh:
                    self.on_counter += 1
                else:
                    self.in_on_transition = False
                    self.frames = []
            else:
                if power > self.start_thresh:
                    self.in_on_transition = True
                    self.on_counter = 0
                else:
                    # Silence
                    pass


    def print_to_stdout(self):
        """Prints to standard out to create a text-based stream of event detection"""

        if self.quiet:
            return

        if self.in_event:
            if self.in_off_transition:
                msg = '         o '
            else:
                msg = '        |||'
        else:
            if self.in_on_transition:
                msg = '         | '
            else:
                msg = ''

        if self.event_started:
            msg = '####### EVENT START #########'
        elif self.event_finished:
            msg = '####### EVENT END #########'
        print(msg)


    def reset(self):
        """Reset event states and storage buffer"""

        self.in_event = False
        self.in_off_transition = False
        self.in_on_transition = False
        self.event_finished = False
        self.event_started = False

        self.frames = []


    def append_to_buffer(self, data):
        if self.in_event or self.in_on_transition:
            self.frames.append(data)


    def process(self, data):
        """Takes in data and updates event transition variables if need be"""

        # Calculate power of frame
        power = calc_power(data)

        self.update_event_states(power)

        # Write to stdout if not self.quiet
        self.print_to_stdout()

        # Append to buffer
        self.append_to_buffer(data)


    def get_event_data(self):
        return np.concatenate(self.frames)


if __name__ == '__main__':
    s = Monitor()
    s.setup()

(Browse code)

Using this code is as easy as saving this file to a script and running it. It works out of the box. You can process the audio any way you want by changing the contents of wait_for_events.

Storing the data

Detecting events is merely one side of the coin. Equally important is storing the data for downstream analyses. When deciding on a storage structure, always start simple. So first I considered storing everything in a simple tab-delimited file, which would have sufficed if I only wanted to store basic knowledge like when the event happened, how long it lasted, and how loud it was. Maybe even for some more sophisticated stuff like what frequency ranges it dominated, what class it belonged to, etc. But instead, I uncompromisingly wanted to retain as much of the raw data as possible, so it was important for me to store the audio itself. This way I’m guaranteed not to be bottlenecked by an incomplete data storage when I inevitably come up with interesting ways to analyze data that would require going back to the raw audio. So to me, it made sense to store everything in a database. I have familiarity with SQLite, so I wrote a bare-bones Python class to interface with SQLite that’s based off of the db module of another repo I contribute to, anvio. It supports basic reading and writing of data, as well as playing back stored audio. You can check it out here.

Here is a demo of the database features:

(Browse code)

In summary,

A database is created for each session.
Each database has a self table and an events table
The self table that contains administrative info like when the session started, which microphone was used, and all free parameters like the event threshold parameters $X$, $Y$, etc.
Each row of the events table contains all of the info pertaining to an individual event.

Here is an example events table:

event_id	t_start	t_end	t_len	energy	power	pressure_mean	pressure_sum	class	audio
0	2020-08-14 19:33:52.590223	2020-08-14 19:33:53.518187	0.927964	0.23233255845044	0.250368072953735	0.0013950959289966	0.0012945987986554		�
1	2020-08-14 19:33:59.424781	2020-08-14 19:33:59.797081	0.3723	0.0124660974593919	0.0334840114407519	0.000334789915565585	0.000124642285565067		�
2	2020-08-14 19:34:16.613249	2020-08-14 19:34:17.313352	0.700103	0.0726653914778307	0.103792429796517	0.000341944134424603	0.000239396114343068		�
3	2020-08-14 19:34:17.788128	2020-08-14 19:34:18.161044	0.372916	0.0286456558738386	0.0768153039125127	0.000415586707439386	0.000154978932591466		�
4	2020-08-14 19:34:34.419520	2020-08-14 19:34:34.792764	0.373244	0.0736813904700202	0.197408104269647	0.000704391482153759	0.000262909894364998		�
5	2020-08-14 19:34:40.413646	2020-08-14 19:34:40.970223	0.556577	0.147239283014793	0.264544318243106	0.000939155610160731	0.000522712412036429		�
6	2020-08-14 19:34:41.438835	2020-08-14 19:34:42.090197	0.651362	0.341081074757945	0.523642881773799	0.00178524750772019	0.00116284238712364		�
7	2020-08-14 19:34:59.914203	2020-08-14 19:35:02.610606	2.696403	8.75341461582365	3.24633024656316	0.00410196267867266	0.011060544472661		�
8	2020-08-14 19:35:09.355492	2020-08-14 19:35:11.909793	2.554301	2670.53404983006	1045.50483667746	0.0609668444851899	0.155727671835365		�
9	2020-08-14 19:35:15.349941	2020-08-14 19:35:15.723554	0.373613	0.0930470076768425	0.24904649376987	0.000803286108113117	0.000300118132710466		�
10	2020-08-14 19:35:22.231554	2020-08-14 19:35:24.185004	1.95345	4224.9895692335	2162.83476374286	0.108183158167411	0.21133039032213		�
11	2020-08-14 19:35:25.261638	2020-08-14 19:35:27.487578	2.22594	6242.85853608116	2804.59425504783	0.139811369221431	0.311211719204751		�
12	2020-08-14 19:35:29.078094	2020-08-14 19:35:30.798650	1.720556	4640.45106447108	2697.06482350535	0.13138258673385	0.226051097900447		�
13	2020-08-14 19:35:34.001482	2020-08-14 19:35:42.686516	8.685034	4513.27087836138	519.660703499996	0.0301028059471401	0.261443893146314		�
14	2020-08-14 19:35:52.047493	2020-08-14 19:35:52.419169	0.371676	0.0484807744715422	0.130438270083466	0.000595550670187947	0.000221351890892775		�
15	2020-08-14 19:36:08.757406	2020-08-14 19:36:10.665634	1.908228	4241.5389432906	2222.76318306335	0.11528165374789	0.219983679568028		�

event_id is the unique ID of the event, and increases chronologically.
t_start is the datetime of the event’s start.
t_end is the datetime of the event’s end.
t_len is the duration of the event in seconds.
energy is the signal energy, i.e. the sum of squared signal $ \sum_{i}^{n} S_i(t)^2 $, where $ S(t) $ is the audio signal. This value increases the louder the event is, or the longer the event lasts. Through my experience it does not seem to scale proportionally with human-perceived “loudness”. (Side note: I wanted to be able to say how many decibels each event was, but this is extremely difficult to pin down without detailed knowledge of the microphone’s physics and circuitry).
power is the energy divided by the t_len, i.e. power is the time derivative of energy.
pressure_mean is a quantity I feel scales proportionally better with human-perceived loudness. It is defined as $ \sum_{i}^{n} |S_i(t)|/n $.
pressure_sum is just like pressure_mean, except it is not normalized by the length of the event. It is defined as $ \sum_{i}^{n} | S_i(t) | $.
class merely symbolizes my aspirations of one day classifying the events. For example as bark, “whine”, “dig”, “howl”, etc. For now, it is blank.
audio is a gzipped binary object of the numpy array that represents the audio. The compression and decompression is carried out by these two functions:

import gzip
import numpy as np

def convert_array_to_blob(array):
    return gzip.compress(memoryview(array), compresslevel=1)


def convert_blob_to_array(blob, dtype=maple.ARRAY_DTYPE):
    return np.frombuffer(gzip.decompress(blob), dtype=dtype)

(Browse code)

Dog monitoring trial #1

With both event detection and event storage solved, it’s time for the first trial: leaving Maple in the confines of her cage and monitoring what happens. Here she is in the crate just moments before her worst fear is realized: abandonment (for 30 whole minutes).

About 10-feet away lies a Fifine K669B USB microphone I picked up for this project. After double checking everything, I start the application:

./main.py run

(Browse code)

… and then quietly leave the apartment.

Results

Fast forward 30 minutes, I’m back with at the apartment with 39MB of data. At first I just wanted to visualize on a line where all the events took place with something I’m familiar with, like matplotlib or seaborn, but after a quick Google search, before I knew it I was learning the basics of Plotly, which can be used to make interactive plots. I’m super glad I spent the time because now I can view an interactive plot for each session. The plot shows up in your browser after typing

./main.py analyze

By default it will pick the last session, but you can pick any session by first listing saved sessions:

./main.py analyze --list

and then choosing a specific session, like:

./main.py analyze --session 2020_08_19_16_45_18

Doing so, yields this:

[Click for interactive plot]

The top plot shows how loud Maple was each minute. And the bottom shows each individual event as a vertical line, where the line’s length reflects how loud the event was (pressure_sum). The most striking thing is that Maple has outbursts followed by periods of silence. In this trial it seems like she has 3 main outbursts:

Each outburst is composed of whines, howls, and barks. Most common is her whine:

Audio files are shrill–turn your volume down.

Your browser does not support the audio element.

“i not liking this” - Maple

Then, of course, her coveted howl:

Your browser does not support the audio element.

“they forgot me and i need to call them back” - Maple

And finally, her ear-piercing bark (trademarked):

Your browser does not support the audio element.

“i frantic and need to get out of this cage so I can find them” - Maple

Uncaptured in this trial, it is known that Maple is capable of yet another form of self-expression, in which she gnaws at the bars of her cage while grunting and foaming at the mouth. This is her most anxiety-ridden behavior and I was happy to see she didn’t do it–hopefully it is a sign she’s already progressed even before I had a chance to acquire data.

Adding owner responses

Missing from the first trial is any mechanism in which the program can “intervene”, either by praising or scolding based on Maple’s behavior. Since Maple, like most other dogs, is very treat motivated, I think it would be excellent to create a treat dispenser that gives Maple treats when she’s quiet. Figuring out the details of that is currently in the works. Until then, I decided that it would be interesting to try and influence her behavior by playing pre-recorded audio of Kourtney and I either praising or scolding her based on the current state of her behavior.

This venture was broken up into two parts: (1) creating something that can record the dog owner’s voice and (2) deciding if and when to intervene.

Recording voices

At first I thought, “I’ll just record a stream of audio in QuickTime player where I praise and scold Maple, chop it up into individual audio clips using Logic Pro X, and then move them into a folder that the application expects to find such audio”. But remember when I was talking about non-committal code being flexible? I realized my codebase already does what I want: it records and clips audio events. This specifically happens in a class called Monitor. The class is big, so I won’t paste it here, but one can interact with the class in the following way.

monitor = Monitor()
event_audio = monitor.wait_for_event()

Calling the wait_for_event method of Monitor will wait patiently until the start of an event is detected. Then it will record the audio until the event ends according to the “event detection criteria” discussed already. After the event ends, it returns the audio, which in the above code is captured by the variable event_audio. So to harness this pre-existing functionality, I simply created another class called RecordOwnerVoice, which inherits Monitor. RecordOwnerVoice, you guessed it, records the owner’s voice, and it does it by asking a series of questions through a very simple command line interface (CLI). By inheriting Monitor, I didn’t have to write a single line of code related to recording audio. In fact, almost all the code in RecordOwnerVoice pertains to management of the CLI menu logic:

class RecordOwnerVoice(events.Monitor):
    """Record and store audio clips to yell at your dog"""

    def __init__(self, args=argparse.Namespace(quiet=True)):
        events.Monitor.__init__(self, args)

        self.menu = {
            'home': {
                'msg': 'Press [r] to record a new sound, Press [q] to quit. Response: ',
                'function': self.menu_handle,
            },
            'review': {
                'msg': 'Recording finished. [l] to listen, [r] to retry, [k] to keep. Press [q] to quit. Response: ',
                'function': self.review_handle
            },
            'name': {
                'msg': 'Great! type a name for your audio file (just a name, no extension). Response: ',
                'function': self.name_handle
            },
            'sentiment': {
                'msg': 'Final question. Choose the sentiment: [g] for good, [b] for bad, [w] for warn. Press [q] to quit. Response: ',
                'function': self.sentiment_handle
            },
        }

        self.state = 'home'
        self.recording = None

        self.recs = OwnerRecordings()
        print(f"You have {self.recs.num} recordings.")


    def run(self):
        self.setup()

        while True:
            self.menu[self.state]['function'](input(self.menu[self.state]['msg']))
            print()

            if self.state == 'done':
                print('Bye.')
                break


    def menu_handle(self, response):
        if response == 'r':
            print('Listening for voice input...')
            self.recording = self.wait_for_event()
            self.state = 'review'
        elif response == 'q':
            self.state = 'done'
        else:
            print('invalid input')


    def review_handle(self, response):
        if response == 'l':
            print('Played recording...')
            print(self.recording.dtype)
            self.recording = audio.denoise(self.recording, self.background_audio)
            self.recording = audio.bandpass(self.recording, 150, 20000)
            sd.play(self.recording, blocking=True)
        elif response == 'r':
            print('Listening for voice input...')
            self.recording = self.wait_for_event()
        elif response == 'k':
            self.state = 'name'
        elif response == 'q':
            self.state = 'done'
        else:
            print('invalid input')


    def name_handle(self, response):
        if response == '':
            print('Try again.')
        elif ' ' in response:
            print('No spaces are allowed. Try again.')
        elif response == 'q':
            self.state = 'done'
        else:
            self.name = response
            self.state = 'sentiment'


    def sentiment_handle(self, response):
        if response == 'w':
            sentiment = 'warn'
        elif response == 'g':
            sentiment = 'good'
        elif response == 'b':
            sentiment = 'bad'
        elif response == 'q':
            self.state = 'done'
        else:
            print('invalid input')
            return

        self.recs.write(self.name, self.recording, maple.RATE, sentiment)
        print('Stored voice input...')
        print(f"You now have {self.recs.num} recordings.")
        self.state = 'home'

Here is a demo of it in action:

(Browse code)

Decision logic

Now I’ve got 30 recordings of Kourtney and I either praising or scolding Maple. Here is an example of a praise:

Your browser does not support the audio element.

And here is a scold:

Your browser does not support the audio element.

The next thing to do is figure out when these should be sprinkled in based on Maple’s behavior. This is an endless hole, and in the interest of keeping things pragmatic, I wanted to develop the most simple heuristics possible that got the job done.

Praise

First up, when to praise… To get a sense of things, I studied the interactive plots and decided upon 4 parameters for praising:

praise_response_window is how big of a time window should be looked at when considering to praise. I chose a default of 2 minutes.
praise_max_events is the maximum number of events that can be within praise_response_window in order to consider praising. In other words, too many events equals no praise. I chose a default of 10.
praise_max_pressure_sum sets a threshold for the loudest that any given event in praise_response_window can be in order to consider praising. If any event has a pressure_sum (read: loudness) above this value, no praise for you. I chose a default of 0.15, but the magnitude is arbitrary and depends on the microphone, its placement, and its settings. To keep it consistent, I place the microphone in the same place each time and keep all settings consistent.
praise_cooldown is how much time has passed since the last praise to in order consider praising. I chose a default of 2 minutes. This prevents rapid-fire praising.

Together, these criteria create the following flowchart, which is processed after each event, or every 10 seconds (which happens first):

Scold

Next, we have scolding. I decided to use these 4 parameters for scolding:

scold_response_window is how big of a time window should be looked at when considering to scold. I chose a default of 1 minute.
scold_threshold sets a threshold for whether or not enough noise has been made within scold_response_window to consider scolding. If the sum of pressure_sum values for all events within scold_response_window exceeds this value, the threshold is passed and scolding is considered. I chose a default of 1.8. For context, in the first trial you can see this was satisfied 4 times.
scold_trigger defines how loud the triggering event has to be in order to be in order to scold. If the pressure_sum of the last event exceeds this value, and the scold_threshold is also met, Maple will be scolded. This is to ensure that she is scolded immediately after an especially loud sound, rather than after a wimper. I chose a default value of 0.15.
scold_cooldown is how much time has passed since the last scold to in order consider scolding. I chose a default of 3 minutes. This prevents rapid-fire scolding.

Together, these criteria create the following flowchart, which is processed after each event:

Implementation

I implemented the decision logic for praising and scolding. If the program decides to respond, a clip with the appropriate sentiment is randomly chosen and played. To store these “owner” events, I added an additional table to the session databases that includes all of the owner response data:

Save the headache, consolidate your tunables

By this point in the project, I have a lot of tunable variables, and so I wanted to consolidate them in one place. To do this, I created a YAML file called config that contains all of my tunable parameters. It looks like this:

[general]
# microphone
microphone = Built-in Microphone
# Recalibrate after this many minutes has passed
recalibration_rate = 10000
# Store events into DB whenever this many dog events occur. No reason to set this lower than 100
max_buffer_size = 100

[respond]
# Should there be owner responses? If 0, all other parameters in this section are irrelevant
should_respond = 1
# Should the owner praise?
praise = 1
# The timeframe when considering if to praise (minutes)
praise_response_window = 2
# The max number of events that should be within timeframe to consider praising
praise_max_events = 10
# The maximum pressure sum of any individual event in the timeframe to consider praising
praise_max_pressure_sum = 0.01
# After praising, wait this many minutes to consider praising again.
praise_cooldown = 2
# Should the owner scold?
scold = 0
# The timeframe when considering if to scold (minutes)
scold_response_window = 1.0
# The sum of pressure sums in timeframe to consider scolding
scold_threshold = 1.8
# The pressure sum of the event required to trigger scolding
scold_trigger = 0.15
# After scolding, wait this many minutes to consider scolding again.
scold_cooldown = 3

[calibration]
# The length of the audio in seconds that is used to calibrate
calibration_time = 3
# The standard deviation divided by the mean of the audio signal must be less than this value to be considered calibrated
calibration_threshold = 0.3
# After this many failed attemps, the calibration_threshold will be increased by 0.1 and the process is repeated.
calibration_tries = 4

[detector]
# Standard deviations above background noise to consider start an event
event_start_threshold = 4
# The number of chunks in a row that must exceed event_start_threshold in order to start an event
num_consecutive = 4
# Standard deviations above background noise to end an event
event_end_threshold = 4
# The number of seconds after a chunk dips below event_end_threshold that must pass for the event to
# end. If during this period a chunk exceeds event_start_threshold, the event is sustained
seconds = 0.25
# If an event lasts longer than this many seconds, everything is recalibrated
hang_time = 20

[analysis]
# how many seconds should each bin be
bin_size = 60

To keep things organized, they are placed under section headings like [general], [analysis], etc. To make these parameters readily accessible in my multi-file project, I added the following to the module’s __init__.py:

import configparser
import ast

# Load up the configuration file, store as nested dictionary `config`
config_path = Path(__file__).parent.parent / 'config'
config_obj = configparser.ConfigParser()
config_obj.read(config_path)
config = {}
for section in config_obj.sections():
    config[section] = {}
    for k, v in config_obj[section].items():
        try:
            config[section][k] = ast.literal_eval(v)
        except:
            config[section][k] = v

(Browse code)

configparser is in the built-in library and parses config, and ast tries its best to interpret the parameter values as either strings, integers, or floats. By putting this in the __init__.py file, whenever maple is imported, parameters in config are immediately accessible:

>>> import maple
>>> maple.config
{'general': {'microphone': 'Built-in Microphone', 'recalibration_rate': 10000, 'max_buffer_size':
100}, 'respond': {'should_respond': 1, 'praise': 1, 'praise_response_window': 2,
'praise_max_events': 10, 'praise_max_pressure_sum': 0.01, 'praise_cooldown': 2, 'scold': 0,
'scold_response_window': 1.0, 'scold_threshold': 1.8, 'scold_trigger': 0.15, 'scold_cooldown': 3,
'warn': 0, 'warn_response_window': 0.25, 'warn_cooldown': 1}, 'calibration': {'calibration_time': 3,
'calibration_threshold': 0.3, 'calibration_tries': 4}, 'detector': {'event_start_threshold': 4,
'num_consecutive': 4, 'event_end_threshold': 4, 'seconds': 0.25, 'hang_time': 20}, 'analysis':
{'bin_size': 60}}

Consolidating your variables into an easy-to-access location is essential.

Trial #2

With what is hopefully a properly implemented praise/scold framework, I decided it was time for trial #2. The intention of this trial is to do as I did previously, except this time Maple will be scolded and/or praised based on the decision logic outlined above.

Will praising help quiet her down, or will it initiate outbreaks? Will scolding curb an outbreak or aggravate her? I was so excited to find out.

I set up the speaker and placed it near her. I made sure the praises were soft and soothing while the scold clips were loud and assertive while not being scary loud. When I was happy, I started the run and left the apartment.

One hour after Maple’s least favorite pastime, here are the results:

[Click for interactive plot]

The plot is just as before, except scolds (red) and praises (green) have been overlaid in the bottom plot. Within the first 15 minutes, Maple clearly has two very loud outbursts. To my delight as a programmer, Maple was indeed scolded during these two outbursts. But unfortunately, it seems like scolding if anything only aggravated her. I am also happy to see that praises occur during periods of silence.

Although this data is too anecdotal to say whether or not scolding/praising really makes a difference, it at least verifies that the decision logic for praising and scolding makes sense, and I’m not totally off base. But to really do some interesting stuff, I’m going to need a lot more data…

A longitudinal study is underway

So far, I’ve developed a considerable framework for data acquisition and I’m very happy with the infrastructure. Yet at this point there is almost no data to work with, so over the next few months, I plan to measure Maple’s activity level and see how she progresses over time. Once I feel there are enough data, my plan is to analyze how she has (hopefully) improved, and whether or not verbal praising and/or scolding is effective. In the meantime I may begin working on crafting a DIY treat dispenser. Regardless of what happens next, I’ll detail it all in the next blog post of this series, which I’ll link here.

Bye.

Creating a virtual dog sitter with live audio processing was originally published by Evan Kiefl at Evan Kiefl on July 20, 2020.

The physics of pool/billiards

2020-04-24T00:00:00+00:00

This post is the first of many in my journey to make a realistic pool/billiards simulator called pooltool.

Before jumping into code, I have to trudge through the theory of both (a) the physics of pool, and (b) the algorithms for evolving a pool shot. The physics of pool is what is covered in this post, and algorithms for evolving a pool shot covered in the second post. Both of these will contain a lot of equations, and little if any code. If this sounds uninteresting to you, skip ahead to the third post in this series. With that said, let’s get started.

A lot has happened since I initially wrote this post, and pooltool is now a fully fledged 3D game/tool. The equations in this post form the underlying physics of pooltool.

The physics of billiards

The thing about pool is that it’s pretty old. This is one of the first historical depictions of billiards, which dates back to 1674.

Two blokes shootin’ the shit over a game of billiards. Source.

While these jolly chaps were smacking balls together, at the the same time in history Isaac Newton was busy inventing calculus, his self-titled Newtonian physics, and a universal theory of gravitation that wholly explained the previously disparate phenomena of tides, why things fall, and the motions of celestial bodies. His contributions to science would spark a revolution in the physical sciences more illustrious than anyone before him, and arguably ever since him.

But most importantly by far, Newton’s work would enable the theoretical treatment of the game of billiards, and as such, the game has been studied for at least 200 years.

As a result of its long history, the theoretical treatment of billiards has been sufficiently solved for most scenarios: (1) ball-cloth interactions, aka how the balls slide, spin, roll, and lie on the table; (2) ball-ball interactions aka how balls collide with one another; (3) ball-cushion interactions, aka how balls bounce off rails; (4) ball-air interactions, aka how the ball behaves when it becomes airborne due to jump shots, etc.; and (5) ball-slate interactions, aka how the ball bounces on the table.

I’m pretty sure the above list covers every phenomenon that happens on (and off) the table. The jury is still out on the finer details of these interactions, but each of these individual scenarios have analytical solutions that are accurate to “sufficient” degree. What I mean by sufficient is that all qualitative effects a pro-player may be expecting to observe manifest directly from the equations.

To build a pool simulator, I’m gonna have to get my hands dirty with these equations. I decided having a reference source was necessary, so I bought the non-exhaustive but heavily referenced modern day treatment of billiards, “The Physics of Pocket Billiards” by Wayland C. Marlow. The physics used in this post comes partly from this book, and partly from random sources on the internet.

In what follows, I am going to lay out all of the physics I’m going to include in the simulation.

Section I: ball-cloth interactions

In this section I review the ball-cloth interaction, aka how pool balls interact with their playing surface.

It is somewhat obvious that the cloth provides a frictional surface that slows the ball’s motion. Yet, depending on the ball’s spin state, this same friction also leads to curved trajectories due to the application of forces orthogonal to the ball’s motion. So modelling the ball-cloth interaction is essential for realism, and quickly gets complicated.

Let’s go over some models from least to most realistic.

(1) No friction

This is the null model. The friction coefficients at the point of contact (PoC) between ball and cloth are 0. This means no energy dissipates from the ball, and it rolls indefinitely. It also spins indefinitely.

(2) Spinless ball

In this model, a frictional force exists between the cloth and ball that opposes the ball’s motion. This means the ball dissipates energy over time and eventually come to a halt. Sounds like pool to me.

However, what’s unrealistic about this model is that the ball has no spin, which is impossible for a ball moving on a cloth with friction. To see why, let’s look at the force contributions that act on the ball:

Figure 1. Force contributions acting on a ball that has o spin. Here, $ \vec{v} $ is the velocity of the ball, $ m $ its mass, and $ R $ its radius. Additionally, we got good old $ \vec{g} $, the gravitational constant, and the normal force $ \vec{N} $.

In this example, pretend the ball initially has no spin (e.g. it’s not rolling) but is moving in the $ +x $-direction with a speed $ |\vec{v}| $.

In the $ y $-axis, there is a gravitational force (mass $ \times $ gravitational constant $ = m\vec{g} $) pulling the ball into the table. Since the table is supporting the ball, it exerts an equal and opposite force onto the ball. This is called the normal force, $ \vec{N} $. Without it, the ball would fall through the table.

So even while the ball remains perfectly still on the table, there’s a perpetual tug-of-war between the ball wanting to accelerate towards the center of the earth, and the table stopping it from doing so. This contention results in friction whenever the ball moves along the table. The ball and cloth essentially rub each other the wrong way as the ball moves, and so a frictional force is exerted on the ball in a direction opposite the ball’s motion, that is denoted here as $ \vec{F}_f $.

So what makes the ball spin? Well, since $ \vec{F}_f $ is applied at the point of contact (PoC) between ball and cloth, this creates a torque on the ball that causes it to rotate. Intuitively, the bottom of the ball is slowing down, but the top of the ball isn’t, so it ends up going head over heels.

To demonstrate this, I took a slow-mo shot of an object ball being struck head on with the cue ball.

Directly after impact, the object ball has a non-zero velocity and no spin. But quickly over time, the ball transitions from sliding across the cloth (no spin) to rolling across the cloth (yes spin). I hope it’s convincing footage.

Wrapping things up for this model, where spin is ignored, you can imagine that instead of the frictional force being applied at the PoC, it’s applied at the ball’s center. Then, there is no torque on the ball, and therefore no spin. This provides a mechanisms that slows the balls down, so it checks that box for realism.

Overall, this is the kind of model you can expect from a pool game that offers a primitive “overhead” perspective, since it provides a passable playing experience for beginners and is simple to code.

(3) Ball with arbitrary spin

From this point on, I’ll refer to a ball with spin as a ball with angular velocity.

In this example, I take on the general case of the ball-cloth interaction. This is the most realistic model I came across that can be solved analytically, and has the following assumptions:

The ball can be in an arbitrary state (but must be on the table)
There is a single point of contact (PoC) between ball and cloth

By the ball can be in an arbitrary state, what I mean is that it can have an arbitrary velocity $(\vec{v})$, angular velocity $(\vec{\omega})$, and displacement relative to some origin $(\vec{r})$. These 3 vectors fully characterize the state of the ball.

The goal is to find equations of motion that can evolve these 3 vectors through time. Essentially, these equations are functions that, when given an initial state $(\vec{v}_0$, $\vec{\omega}_0$, $\vec{r}_0)$, can give you an updated state $(\vec{v}$, $\vec{\omega}$, $\vec{r})$ some time $t$ later.

As for the second assumption, a single point of contact is a fairly accurate assumption, but technically the weight of the ball bunches up the cloth as it moves. The amount of bunching up depends on how loosely the cloth is stretched over the slate. Additionally, the cloth itself can be compressed, and cloth fibers and other non-idealities can contact the ball at multiple points. And so in actuality there does not exist a point of contact, but rather, an area of contact.

Figure 2. The cloth is a compressible surface, and so in actuality there does not exist a “point of contact”, but rather, an “area of contact”.

The most important thing to realize is that throughout a ball’s trajectory, it will always be in any of these 4 different modes: sliding, rolling, spinning,stationary, or airborne (all of these states occur during the ball-cloth interaction, with the exception of the airborne state, which is covered later on). The physics is different for each of these cases, so I tackle them piecewise from least to most complicated.

Case 1: Stationary

If the ball is stationary, the ball stays where it is and there is no angular or linear momentum. In other words:

Stationary equations of motion

Displacement:

\[\vec{r}(t) = \vec{r}_0 \label{stationary_r}\]

Velocity:

\[\vec{v}(t) = \vec{0} \label{stationary_p}\]

Angular velocity:

\[\vec{\omega}(t) = \vec{0} \label{stationary_o}\]

Validity:

$0 \le t < \infty$.

It’s important to keep in mind each of Eqs. $\eqref{stationary_r}$, $\eqref{stationary_p}$, and $\eqref{stationary_o}$ are vector equations that can be broken down into 3 scalar equations each, one for each spatial dimension. For example, Eq. $\eqref{stationary_o}$ can be written as $\omega_x(t) = 0$, $\omega_y(t) = 0$, and $\omega_z(t) = 0$. I interchangeably use both scalar and vector equations, so make sure you are spotting the difference.

Case 2: Spinning

Spinning is a commonly observed ball state in which there is no linear momentum of the ball, yet it is spinning like a top:

Like in the stationary state, the ball has no linear momentum:

\[\vec{v}(t) = \vec{v} \notag\]

Likewise, the ball remains in place:

\[\vec{r}(t) = \vec{r}_0 \notag\]

However, since the ball is rotating, it has an angular velocity. You’ll notice that the ball spins around the $z-$axis, relative to the coordinate system in Figure 3:

Figure 3. A ball spinning in place. In this coordinate system the table is in the $xy-$plane.

Spinning around the $z-$axis is not by chance–it is a constraint of the state, since if the ball had any components of its angular velocity in the $x$ or $y$ directions, it would create a friction with the cloth that would translate into a linear velocity. Since spinning is characterized by 0 linear velocity $(\vec{v}=\vec{0})$, angular velocity is strictly in the $z-$axis. In other words,

\[\omega_x(t) = 0 \notag\] \[\omega_y(t) = 0 \notag\] \[\omega_z(t) = \text{ }??? \notag\]

To characterize the $z-$axis angular velocity, I need to introduce some bull****.

I told you that in this model, there is a single PoC between ball and cloth. If such were truly the case, there is nothing to stop the ball from spinning forever (besides air, which I will ignore).

This is because a ball spinning in place has zero speed at the infinitesimally small PoC. In other words, the relative velocity between the ball and cloth at the PoC is 0, and this means there can exist no frictional force.

Of course, everyone knows that the ball does slow, which is proof that there does not exist a point of contact but rather an area of contact.

Rather than explicitly define an area of contact, which would greatly complicate the physics, we account for this embarrassing blunder of the model by introducing a phenomenological friction parameter that slows down the $z-$component of the ball’s angular velocity over time.

A phenomenological parameter, you say? It’s a parameter that is added to a model ad hoc, that explains a phenomenon (in this case, the slowing down of a ball’s rotation) that does not come from assumptions of the model. It’s what people do when they want to model an observation but their model is bad and does not cause the observation. Basically, its cheating.

After adding a phony friction term, we have our equations of motion solved:

Spinning equations of motion

Displacement:

\[\vec{r}(t) = \vec{r}_0 \label{spinning_r}\]

Velocity:

\[\vec{v}(t) = \vec{0} \label{spinning_p}\]

Angular velocity:

\[\omega_x(t) = 0 \label{spinning_ox}\] \[\omega_y(t) = 0 \label{spinning_oy}\] \[\omega_z(t) = \omega_{0z} - \frac{5\mu_{sp}g}{2R}t \label{spinning_oz}\]

Validity:

$0 \le t \le \frac{2R}{5\mu_{sp}g}\omega_{0z}$.

In Eq. $\eqref{spinning_oz}$, $\omega_{0z}$ is angular velocity in the $z-$axis at $t=0$, $\mu_{sp}$ is the coefficient of spinning friction, $g$ is the gravitational constant, and $R$ is the ball’s radius. The equation states that as time evolves, there is a linear decay in the ball’s angular velocity. Collectively, these equations are valid until the ball stops rotating, which happens when $\omega_z(t)$ is $0$. This occurs when $t=(2R\omega_{0z})/(5\mu_{sp}g)$.

Case 3: Rolling

Think of rolling as driving your car on concrete, whereas sliding would be like driving your car on ice.

In the former, your tires grip the road such that at the point of contact, there is no relative velocity between the tire and the road; each time the tire does one rotation, your car translates the circumference of your tire.

But on ice, there is a lot of slippage between the tire and the ice, and therefore a relative velocity; each time your tire does one rotation, your car moves far less than one circumference of your tire.

In physics textbooks, what I call rolling is actually called rolling without slippage and what I call sliding is actually called rolling with slippage. Sorry for the confusing terminology.

Alright, so let’s get on with it. Consider a ball that is rolling in the positive $x-$direction:

Figure 4. A rolling ball, that moves in the $x-$direction. The left panel shows a bird’s eye view, and the right panel shows a profile view. In this coordinate system, the angular velocity is in the $y-$direction.

Such a ball will move in a straight line until it comes to a rest. As it travels, it will be slowed down by a frictional force proportional to its velocity, which implies that its velocity will decay linearly with time:

\[\vec{v}(t) = \vec{v}_0 - \mu_r g t \hat{v}_0 \label{rolling_velocity}\]

Here, $\mu_r$ is the coefficient of rolling friction, $g$ is the gravitational constant, and $\hat{v}_0$ is the unit vector that points in the direction of the ball’s travel (according to this coordinate system, $\hat{v}_0 = \hat{i}$).

Integrating this equation with respect to time yields the displacement as a function of time:

\[\vec{r}(t) = \vec{r}_0 + \vec{v}_0 t - \frac{1}{2} \mu_r g t^2 \hat{v}_0 \notag\]

Now for angular velocity. To discuss this, I should formalize the concept of rolling, which is formally defined as the state in which the relative velocity, $\vec{u}(t)$, between the ball and cloth at the PoC is $\vec{0}$.

$\vec{u}(t)$ has two contributions: (1) the linear velocity of the ball, i.e. the velocity of the center of mass, and (2) the velocity between ball and cloth that exists because of the ball’s rotation. Their sum defines the relative velocity:

\[\vec{u}(t) = \vec{v}(t) + R \hat{k} \times \vec{\omega}(t) \label{rel_vel}\]

For example, here is a shot where I tried to make sure the cue ball only has linear velocity:

I think we all agree there is some amount of rotation, but let’s just ignore it.

Since the ball does not rotate, there is no angular velocity, so Eq. $\eqref{rel_vel}$ reduces to

\[\vec{u}(t) = \vec{v}(t) \notag\]

Similarly, here is a shot that only has a velocity between ball and cloth only due to the ball’s rotation.

Right at the moment of contact, the cue ball spins in place and therefore has no center of mass velocity. In this instant, Eq. $\eqref{rel_vel}$ becomes

\[\vec{u}(t) = R \hat{k} \times \vec{\omega}(t) \notag\]

In the particular case shown, the ball has top spin, so the cross product dictates that $\vec{u}$ points in the direction opposite the cue ball’s travel.

Both of the above examples are cases in which $\vec{u}(t) \ne \vec{0}$, so are therefore cases of sliding, not rolling. To be rolling $(\vec{u}(t) = \vec{0})$, these contributions must match each other:

\[-R \hat{k} \times \vec{\omega}(t) = \vec{v}(t) \label{roll_condition}\]

This refers to the condition in which every time the ball does a complete rotation about the $y-$axis (according to the axes defined in Figure 4), the ball must travel exactly one circumference (aka $2 \pi R$ in the $x-$axis). Unless this exact condition is met, a moving ball is sliding, not rolling.

For how particular this condition seems, it is interesting that in the game of pool, balls are most often rolling. The reason is that any sliding ball experiences friction that reduces the magnitude of $\vec{u}(t)$ until it is rolling. In that sense, rolling is somewhat of an equilibrium state.

Now that there is a mathematical condition for rolling, i.e. Eq. $\eqref{roll_condition}$, there is a lot we can learn about the angular velocity. According to our coordinate system in Figure 4, the RHS of Eq. $\eqref{roll_condition}$ is strictly in the $+x-$direction. That means the LHS must also be strictly in the $+x-$direction. Expanding the cross product on the LHS yields:

\[-R\hat{k} \times \vec{\omega}(t) = R \begin{bmatrix} \omega_y(t) \\ -\omega_x(t) \\ 0 \end{bmatrix} \notag\]

3 really important things result from this equation:

In order for the RHS to point in the $+x-$direction, as it must, $\omega_x(t)$ is necessarily 0. So no angular velocity in the direction of motion.
Using Eq. $\eqref{roll_condition}$, it follows that $\omega_y(t) = |\vec{v}(t)|/R$. Since $\vec{v}(t)$ is known via Eq. $\eqref{rolling_velocity}$, this equation solves the time evolution of $\omega_y(t)$. Note that $\omega_y(t)$ is strictly positive, which intuitively refers to the fact that in order to be rolling, the ball must have top spin, not back spin.
$\omega_z(t)$ is absent from this equation, which means that it is a free parameter: it can take any value.

Points 1 & 2 solve the time evolution for $\omega_x(t)$ and $\omega_y(t)$, respectively. Meanwhile, point 3 has consequences that you may find highly surprising. For example, we know that in the rolling state, the ball path is a straight line. Yet the model predicts this is true regardless of $\omega_z$. Is that really sensical? It may not match your initial intuition, but it does match the reality:

In the above example I apply a shit load of clockwise side spin $(\omega_z(t) < 0)$ and as you can see, the ball follows a straight line. Pretty hard to deny, but it might still be at odds with what you know about pool. For example, look at O’Sullivan and the gang “swerving” the ball by applying side spin:

So what’s the difference between those shots, and the one I manufactured? The fundamental difference is that these players are elevating their cue, which causes $\omega_x$ to be non-zero (angular velocity in the direction of motion). This “barrel-roll” rotation is what causes curved trajectories, otherwise known as swerve or masse. (More on the cue-ball interaction later). On the other hand, my shot did not have any significant amount of $\omega_x$, so whatever small amount existed quickly dissipated within fractions of a second, yielding an otherwise straight trajectory.

The takeaway is that $\omega_z(t)$ is decoupled from everything else, and evolves according to Eq. $\eqref{spinning_oz}$, which was dealt with in Case 2. The only thing left to do is write down the equations for a given frame of reference. Let’s use a frame of reference that is centered about the ball’s initial center of mass coordinates. Then,

Displacement:

\[\vec{r}(t) = (v_0 t - \frac{1}{2} \mu_r g t^2) \, \hat{v}_0 \label{rolling_r_general}\]

Velocity:

\[\vec{v}(t) = (v_0 - \mu_r g t) \, \hat{v}_0 \label{rolling_v_general}\]

Angular velocity:

\[\vec{\omega}_{xy}(t) = \hat{k} \times \frac{\vec{v}(t)}{R} \label{rolling_oxy_general}\] \[\omega_z(t) = \omega_{0z} - \frac{5\mu_{sp}g}{2R}t \label{rolling_oz_general}\]

where $v_0$ is the initial speed of the ball. Since angular velocity has 2 decoupled components, $\vec{\omega}(t)$ is represented by 2 equations. $\vec{\omega} _{xy} (t)$ defines the angular velocity projected onto the $xy-$plane, which is parallel with the table.

The above equations are defined in terms of $\hat{v}_0$, which can point direction in the $xy-$plane. If I take a frame of reference in which ball motion is in the $+x-$direction, I can drop the vector notation:

Rolling equations of motion (ball coordinates)

Displacement:

\[r_x(t) = v_0 t - \frac{1}{2} \mu_r g t^2 \label{rolling_rx_ball}\] \[r_y(t) = 0 \label{rolling_ry_ball}\] \[r_z(t) = 0 \label{rolling_rz_ball}\]

Velocity:

\[v_x(t) = v_0 - \mu_r g t \label{rolling_vx_ball}\] \[v_y(t) = 0 \label{rolling_vy_ball}\] \[v_z(t) = 0 \label{rolling_vz_ball}\]

Angular velocity:

\[\omega_x(t) = 0 \label{rolling_ox_ball}\] \[\omega_y(t) = \frac{v_x(t)}{R} \label{rolling_oy_ball}\] \[\omega_z(t) = \omega_{0z} - \frac{5\mu_{sp}g}{2R}t \label{rolling_oz_ball}\]

Validity:

$0 \le t \le \frac{\lvert \vec{v}_0 \rvert}{\mu_r g}$. If $\frac{2R}{5\mu _{sp} g} \omega _{0z} < \frac{\lvert \vec{v}_0 \rvert}{\mu_r g}$, then $\omega_z(t) = 0$ for $t > \frac{2R}{5\mu _{sp} g} \omega _{0z}$.

The chosen frame of reference (centered at ball’s initial coordinates, x-axis in direction of motion) is convenient, but annoying to deal when you are interested in knowing how the ball evolves in relation to the table coordinates. Suppose $\hat{v}_0$ relates to the table in the following way:

Figure 5. Coordinate system in which the table is described. $\phi$ relates the ball’s unit vector of motion, $\hat{v}_0$, to the table coordinates. The origin (0,0) is the bottom left pocket.

Then $\hat{v}_0$ can be expressed in terms of the table coordinates via the following rotation matrix:

\[R = \begin{bmatrix} \cos\phi & -\sin\phi & 0 \\ \sin\phi & \cos\phi & 0 \\ 0 & 0 & 1 \end{bmatrix} \label{rot_mat}\]

We can rotate Eqs. $\eqref{rolling_rx_ball}$-$\eqref{rolling_oz_ball}$ via Eq. $\eqref{rot_mat}$ and subsequently add an initial displacement vector $\vec{r}_0$ to Eqs. $\eqref{rolling_rx_ball}$-$\eqref{rolling_rz_ball}$ in order to rewrite the rolling equations of motion in the table coordinate system:

Rolling equations of motion (table coordinates)

Displacement:

\[r_x(t) = r_{0x} + v_0 \cos(\phi) \, t - \frac{1}{2} \mu_r g \cos(\phi) \, t^2 \label{rolling_rx_table}\] \[r_y(t) = r_{0y} + v_0 \sin(\phi) \, t - \frac{1}{2} \mu_r g \sin(\phi) \, t^2 \label{rolling_ry_table}\] \[r_z(t) = 0 \label{rolling_rz_table}\]

Velocity:

\[v_x(t) = v_0 \cos(\phi) - \mu_r g \cos(\phi) \, t \label{rolling_vx_table}\] \[v_y(t) = v_0 \sin(\phi) - \mu_r g \sin(\phi) \, t \label{rolling_vy_table}\] \[v_z(t) = 0 \label{rolling_vz_table}\]

Angular velocity:

\[\omega_x(t) = - \frac{1}{R} \lvert \vec{v}(t) \rvert \sin(\phi) \label{rolling_ox_table}\] \[\omega_y(t) = \frac{1}{R} \lvert \vec{v}(t) \rvert \cos(\phi) \label{rolling_oy_table}\] \[\omega_z(t) = \omega_{0z} - \frac{5\mu_{sp}g}{2R}t \label{rolling_oz_table}\]

Validity:

Case 4: Sliding

If you’re ended up here without reading Case 3 (the rolling case), you might consider briefing specifically the section on relative velocity, otherwise this won’t make any sense.

Sliding occurs whenever there is a non-zero relative velocity, $\vec{u}(t)$, between the ball and cloth at the point of contact. If you need to know whether you’re sliding, take this zaney flowchart questionnaire:

Figure 6. Determine whether or not you are sliding. $\lvert \vec{v} \rvert$ is the speed of the ball, $\omega _{\parallel}$ is the angular velocity in the direction of motion, and $\omega _{\bot}$ is the angular velocity that is both orthogonal to the direction of motion and parallel to the table. Note that as discussed in Case 3, $\omega_z$ does not influence $\vec{u}(t)$, and therefore has no impact on whether or not you are sliding.

If there is any curvature whatsoever in the trajectory of a ball, it occurs while the ball is sliding (assuming the table is perfectly level). If you shoot a draw or stun shot, the cue ball is sliding. Directly after an object ball is struck by the cue ball, it is in a sliding state until friction with the cloth drives it into the rolling state. Starting to get the idea? Good.

What separates the sliding case from the rolling case is that $\vec{u}(t)$ can point in any direction in the $xy-$plane. The most important thing to note about this case is that whichever direction $\vec{u}(t)$ points, a frictional force opposes it. This can lead to curved trajectories. To see how, consider Figure 7, in which a ball is initially moving in the $+x-$direction, along with angular velocity also in the $+x-$direction. This is the kind of spin that a screw has when screwed into wood, where the spin of the screw is about the same axis as the axis of motion–a rather unrealistic spin to impart on a ball in the game of pool, but useful for the purposes of demonstration:

Figure 7. A ball moving in the $+x-$direction with angular velocity also in the $+x-$direction. The left panel shows a bird’s eye view, and the right panel shows a side view of the table and ball, as if looking down the barrel of the cue stick. Because $\vec{\omega}(t)$ is parallel to $\vec{v}(t)$, the $R \hat{k} \times \vec{\omega}(t)$ component in Eq. $\eqref{rel_vel}$ is orthogonal to $\vec{v}(t)$. The below paragraph defines the force terms.

I am used to thinking about friction opposing the ball’s center of mass motion, and that frictional force is still present in the sliding case and is shown in Figure 7 as $\vec{F}_S$ (straight-line force). Yet, the $R \hat{k} \times \vec{\omega}(t)$ contribution to $\vec{u}(t)$ (see Eq. \eqref{rel_vel}) also creates a frictional force, $\vec{F}_C$ (curved-line force). The sum of these two force terms, $\vec{F} = \vec{F}_C + \vec{F}_S$, yields the net frictional force manifesting from the ball-cloth interaction, and is anti-parallel to the relative velocity, $\vec{u}(t)$. (Note that because $\vec{\omega}(t)$ was exactly parallel to $\vec{v}(t)$, $\vec{F}_C$ and $\vec{F}_S$ are orthogonal, but in general this is not true). Since there exists a force component, $\vec{F}_C$, which is orthogonal to the ball’s velocity, this ball will begin curving to the right (the negative $y-$direction)! The ball will continue to curve until $|\vec{u}(t)| \rightarrow 0$, at which point the ball enters the rolling state, where it will spend the rest of its days transiting a line. The equation governing $|\vec{u}(t)| \rightarrow 0$ is

\[\vec{u}(t) = (u_0 - \frac{7}{2} \mu_s g t ) \, \hat{u}_0 \label{rel_vel_evo}\]

where $u_0$ is the magnitude of $\vec{u}(t=0)$ and $\mu_s$ is the sliding coefficient of friction.

To establish the equations of motion for the sliding case, let’s again use a frame of reference centered about the ball’s initial center of mass coordinates. Additionally, I assume the initial center of mass velocity, $\vec{v}(0)$, points in the $+x-$direction. Then,

Sliding equations of motion (ball coordinates)

Displacement:

\[\vec{r}(t) = \vec{v}_0 \, t - \frac{1}{2} \mu_s g t^2 \, \hat{u}_0 \label{sliding_r_ball}\]

Velocity:

\[\vec{v}(t) = \vec{v}_0 - \mu_s g t \, \hat{u}_0 \label{sliding_p_ball}\]

Angular velocity:

\[\vec{\omega}_{xy}(t) = \vec{\omega}_{0xy} - \frac{5 \mu_s g}{2 R} \, t \, (\hat{k} \times \vec{u}_0) \label{sliding_parallel_ball}\] \[\omega_z(t) = \omega_{0z} - \frac{5\mu_{sp}g}{2R}t \label{sliding_perp_ball}\]

Validity:

$0 \le t \le \frac{2}{7}\frac{u _0}{\mu _s g}$. If $\frac{2R}{5\mu _{sp} g} \omega _{0z} < \frac{2}{7}\frac{u _0}{\mu _s g}$, then $\omega_z(t) = 0$ for $t > \frac{2R}{5\mu _{sp} g} \omega _{0z}$.

These are essentially the same as the rolling equations, except the acceleration terms in the $xy-$plane act in the $\hat{u}_0$ direction instead of the $\hat{v}_0$ direction, and the rolling coefficient of friction $\mu_r$ is replaced with the sliding coefficient of friction $\mu_s$.

We can express these in table coordinates by applying the rotation matrix (Eq. $\eqref{rot_mat}$), which yields

Sliding equations of motion (table coordinates)

Displacement:

\[r_x(t) = r_{0x} + v_0 \cos(\phi) \, t \\ - \frac{1}{2} \mu_s g \, ( u_{0x} \cos(\phi) - u_{0y} \sin(\phi) ) \, t^2 \label{sliding_rx_table}\] \[r_y(t) = r_{0y} + v_0 \sin(\phi) \, t \\ - \frac{1}{2} \mu_s g \, ( u_{0x} \sin(\phi) + u_{0y} \cos(\phi) ) \, t^2 \label{sliding_ry_table}\] \[r_z(t) = 0 \label{sliding_rz_table}\]

Velocity:

\[v_x(t) = v_0 \cos(\phi) \\ - \mu_s g \, ( u_{0x} \cos(\phi) - u_{0y} \sin(\phi) ) \, t \label{sliding_vx_table}\] \[v_y(t) = v_0 \sin(\phi) \\ - \mu_s g \, ( u_{0x} \sin(\phi) + u_{0y} \cos(\phi) ) \, t \label{sliding_vy_table}\] \[v_z(t) = 0 \label{sliding_vz_table}\]

Angular velocity:

\[\omega_x(t) = \omega_{0x} \cos(\phi) - \omega_{0y} \sin(\phi) \\ + \frac{5 \mu_s g}{2R} (u_{0y} \cos(\phi) + u_{0x} \sin(\phi)) \, t \label{sliding_ox_table}\] \[\omega_y(t) = \omega_{0x} \sin(\phi) + \omega_{0y} \cos(\phi) \\ + \frac{5 \mu_s g}{2R} (u_{0y} \sin(\phi) - u_{0x} \cos(\phi)) \, t \label{sliding_oy_table}\] \[\omega_z(t) = \omega_{0z} - \frac{5\mu_{sp}g}{2R}t \label{sliding_oz_table}\]

Validity:

$0 \le t \le \frac{2}{7}\frac{u _0}{\mu _s g}$. If $\frac{2R}{5\mu _{sp} g} \omega _{0z} < \frac{2}{7}\frac{u _0}{\mu _s g}$, then $\omega_z(t) = 0$ for $t > \frac{2R}{5\mu _{sp} g} \omega _{0z}$.

This concludes the section of ball-cloth interactions, at least for now.

Section II: ball-ball interactions

This section is dedicated to the collision physics between two balls.

When physically modelling a phenomenon, the sky is the limit in terms of how real you want to get. In consideration of the ball-ball interaction, a complete classical description would entail treating the balls as compressible objects–perhaps even modelling the pressure waves that emanate within each ball during a collision. Perhaps this treatment is most necessary during the break shot, and I would be very interested to know how the degree of realism of such a treatment compares to the more pragmatic approaches I will be taking. Speaking of which, here are the two models I will present:

Elastic, instantaneous, frictionless
Elastic, instantaneous (TODO)

In each of these models, multi-ball collisions are not considered, i.e. each interaction is pairwise.

(1) Elastic, instantaneous, frictionless

In this model, collisions are perfectly elastic, which means no energy dissipates as a result of the collision. This is not true for several reasons. First of all, pool balls make noise when they collide, and those sound waves are a form of energy dissipation from the system. Then there is heat generated via the collision, another form of energy dissipation. Are there more instances of energy dissipation? Those are the ones I can think of anyways.

Another assumption is that the balls interact instantaneously. Like everything, pool balls are viscoelastic objects and have some minute degree of compressibility. When objects compress, they exhibit a spring-like response, like how this bouncy ball compresses into the ground:

As seen in the clip, this creates an interaction between the bouncy ball and the ground that lasts a finite period of time. Pool balls are subject to the same phenomenon, to a degree orders of magnitude less exaggerated. However slight the effect may be, in reality pool balls interact over a finite period of time, a time that this model will ignore.

The final assumption is that the ball-ball interaction is frictionless, i.e. perfectly slippery. This implies that there is no transfer of spin from one ball to another, which is commonly known as throw.

The frictionless assumption is the worst of these assumptions, since friction between balls is what causes spin- and cut-induced throw, which are effects that exhibit substantial influence on shot outcome, and must be accounted for by amateurs and pros alike. In the next model, I will account for friction between balls.

For this model, I first tackle the simple scenario in which a moving ball strikes a stationary ball. Then, I handle the general case of 2 moving balls.

Case 1: stationary ball

Assuming the elastic, instantaneous, and frictionless model, consider a moving ball that hits a stationary ball, shown in Figure 8:

Figure 8. Ball A (blue) strikes ball B (red) with an incoming velocity $\vec{v}_0$ in the $+x-$direction. The unfilled circle shows where ball A ends up striking ball B. During contact, the line connecting the centers of the two balls (the line of centers) forms an angle $\alpha$ with $\vec{v}_0$. The outgoing velocity of ball B, $\vec{v}_B$, runs along the line of centers, and the outgoing velocity of ball A, $\vec{v}_B$, is perpendicular to $\vec{v}_B$.

If we imagine that Ball A is the cue ball and ball B is an object ball, this represents a “cut shot” of $\alpha$ degrees. We would like to know how to resolve this collision. What do I mean by “resolve”? I mean, Given the state of the balls the moment before the collision, what is the state of the balls the moment after the collision. Just like in Section I, the state of a ball is defined by its position, velocity, and angular velocity.

Some of these I can bang out right away. Suppose the collision happens between $t = \tau$ and $t = \tau + dt$, keeping in mind instantaneity of the collision dictates that $dt$ is infinitesimally small. There is thus no amount of time for the balls to change position in the moments immediately before and after the collision. Therefore

\[\vec{r}_A(\tau+dt) = \vec{r}_A(\tau) \notag\] \[\vec{r}_B(\tau+dt) = \vec{r}_B(\tau) \notag\]

One down. Since I’m not accounting for friction effects, there is no loss or change in angular velocity due to the collision:

\[\vec{\omega}_A(\tau+dt) = \vec{\omega}_A(\tau) \notag\] \[\vec{\omega}_B(\tau+dt) = \vec{\omega}_B(\tau) \notag\]

Two down. This leaves only the velocities of the balls, which certainly do change. To solve the outgoing velocities, I’m going to need some conservation of momentum and energy. Figure 8 details the scenario above. For convenience, the coordinate system is defined so that $\vec{v}_0$ points in the $+x-$direction.

Before getting into equations, what’s clear immediately is that we know the outgoing direction of Ball B just from geometry: it is parallel to the line that connects the centers of the balls at the moment of impact. That’s because this line marks the direction of force that Ball A applies to Ball B. Now, let’s figure out the outgoing direction of Ball B.

According to conservation of linear momentum, the momentum before the collision equals the momentum after it:

\[m\vec{v}_0 = m\vec{v}_A + m\vec{v}_B \notag\] \[\vec{v}_0 = \vec{v}_A + \vec{v}_B \label{p_1}\]

Concurrently, conservation of energy states that the energy before the collision equals the energy after it

\[E(\tau) = E(\tau + dt) \label{con_energy}\]

Since the model ignores angular momentum transfer, we need only account for the kinetic energy resulting from linear translation of the balls. Plugging kinetic energy terms into Eq. $\eqref{con_energy}$ yields

\[\frac{1}{2} m \lvert \vec{v}_0 \rvert ^2 = \frac{1}{2} m \lvert \vec{v}_A \rvert ^2 + \frac{1}{2} m \lvert \vec{v}_B \rvert ^2 \notag\] \[\lvert \vec{v}_0 \rvert ^2 = \lvert \vec{v}_A \rvert ^2 + \lvert \vec{v}_B \rvert ^2 \notag\] \[\vec{v}_0 \cdot \vec{v}_0 = \lvert \vec{v}_A \rvert ^2 + \lvert \vec{v}_B \rvert ^2 \label{E_1}\]

Plugging Eq. $\eqref{p_1}$ into the LHS of Eq. $\eqref{E_1}$ yields something very interesting:

\[(\vec{v}_A + \vec{v}_B) \cdot (\vec{v}_A + \vec{v}_B) = \lvert \vec{v}_A \rvert ^2 + \lvert \vec{v}_B \rvert ^2 \notag\] \[\lvert \vec{v}_A \rvert ^2 + \lvert \vec{v}_B \rvert ^2 + 2 \, \vec{v}_A \cdot \vec{v}_B = \lvert \vec{v}_A \rvert ^2 + \lvert \vec{v}_B \rvert ^2 \notag\] \[\vec{v}_A \cdot \vec{v}_B = 0 \label{perp}\]

The inner product of $\vec{v}_A$ and $\vec{v}_B$ is 0, which means that outgoing velocities of the 2 balls are $90^{\circ}$ to one another! Since the direction of $\vec{v}_B$ is known from geometry, the direction of $\vec{v}_A$ is known as well. To determine the magnitudes, I superpose the 3 velocity vectors on top of each other:

Figure 9. Geometrical relationships between $\vec{v}_0$, $\vec{v}_A$, and $\vec{v}_B$.

This diagram contains 3 critical pieces of information.

$\vec{v}_B$ makes an angle, $\alpha$, with the incoming velocity, $\vec{v}_0$. This is known because Ball A imparts an impulse to Ball B in the direction parallel to the line connecting their two centers of mass.
The sum of $\vec{v}_A$ and $\vec{v}_B$ is $\vec{v}_0$. This is known from Eq. $\eqref{p_1}$, the conservation of linear momentum.
$\vec{v}_A$ is perpendicular to $\vec{v}_B$. This is known from Eq. $\eqref{perp}$, which used both conservation of energy and linear momentum.

Given these facts, I can soh-cah-toa my way to the answer. Expressed in terms of $\alpha$ and $v_0$, I get:

\[\vec{v}_A(t+\tau) = (v_0 \sin\alpha) \, \hat{v}_A \notag\] \[\vec{v}_B(t+\tau) = (v_0 \cos\alpha) \, \hat{v}_B \notag\]

Putting it all together, we have our equations for the elastic, instantaneous, and frictionless ball-ball collision in the specific case where one ball is stationary:

Elastic, instantaneous, frictionless ball-ball collision (stationary case)

Displacement:

\[\vec{r}_A(\tau+dt) = \vec{r}_A(\tau) \label{rA_simple}\] \[\vec{r}_B(\tau+dt) = \vec{r}_B(\tau) \label{rB_simple}\]

Velocity:

\[\vec{v}_A(t+\tau) = (v_0 \sin\alpha) \, \hat{v}_A \label{vA_simple}\] \[\vec{v}_B(t+\tau) = (v_0 \cos\alpha) \, \hat{v}_B \label{vB_simple}\]

Angular velocity:

\[\vec{\omega}_A(\tau+dt) = \vec{\omega}_A(\tau) \label{oA_simple}\] \[\vec{\omega}_B(\tau+dt) = \vec{\omega}_B(\tau) \label{oB_simple}\]

Case 2: both moving

Relaxing the assumption that one ball is stationary may at first seem like a terrible idea–the introduced complexity must be horrible.

And that intuition is basically correct. Treating both balls as moving would be a nightmare. Yet even when both balls are moving, I don’t need to treat it that way. Instead, I can change to a frame of reference that moves with one of the balls. In such a frame of reference, that ball is stationary. A picture of the situation is shown in Figure 10:

Figure 10. In the left panel, Ball A (blue) and Ball B (red) are both moving and due to collide at the position of the unfilled circles. If $\vec{v}_B$ is subtracted from the velocities of both balls (see yellow vectors), the frame of reference is changed to one that moves with ball B, shown in the right panel. In this scenario, ball B is stationary, and the situation reduces to Case 1, and specifically the situation depicted in Figure 8.

In Figure 10 panel II, the balls should contact with the same orientation as shown in panel I (about $50^{\circ}$ to the $x-$axis), rather than as pictured.

Since physics behaves the same as viewed from all inertial reference frames, I am well within my rights to make this transformation during the collision. After solving the outgoing state post-collision, I can reverse the transformation and voila, I’m done. This makes Case 2 trivial, since it can be reduced to Case 1. Explicitly, the procedure goes like this:

Elastic, instantaneous, frictionless ball-ball collision (both moving)

First, make the following transformation so Ball B is stationary:

\[\vec{v}_B'(\tau) = \vec{v}_B(\tau) - \vec{v}_B(\tau) = \vec{0} \label{trans_B}\] \[\vec{v}_A'(\tau) = \vec{v}_A(\tau) - \vec{v}_B(\tau) \label{trans_A}\]

The velocities of the collisions are resolved via Eqs. $\eqref{vA_simple}$ and $\eqref{vB_simple}$, where Eq. $\eqref{trans_A}$ is substituted as $\vec{v}_0$. This yields post-collision velocity vectors $\vec{v}_A’(\tau + dt)$ and $\vec{v}_B’(\tau + dt)$, which can be transformed back to the table frame of reference via the inverse transformation (adding back $\vec{v}_B(\tau)$):

\[\vec{v}_A(\tau + dt) = \vec{v}_A'(\tau + dt) + \vec{v}_B(\tau) \label{inv_trans_A}\] \[\vec{v}_B(\tau + dt) = \vec{v}_B'(\tau + dt) + \vec{v}_B(\tau) \label{inv_trans_B}\]

Section III: ball-cushion interactions

The ball-cushion interaction is probably the most difficult to model accurately. There are so many factors to consider. The height, shape, friction, and compressibility of the cushion. The incoming angle, velocity, and spin of the ball. All of these have significant effects on how the rail influences the ball’s outgoing state. Let’s take a look in slow motion:

First, you can really see that the rail deforms substantially throughout its interaction with the ball.

Pool ball significantly deforming the cushion Source.

The implication is two-fold.

First, the interaction is non-instantaneous. In fact, it persists far longer than the ball-ball interaction. Yet finite-time interactions open up a can of worms for multi-body dynamics, since a second ball may join the party and collide with the first ball whilst it is interacting with the cushion.

For this reason, along with the inherent complexity of the soft-body physics, modelling the interaction as non-instantaneous is likely unfeasible.

Second, there is no single point of contact (PoC) between ball and cushion. Rather, the interaction occurs over a line of contact (LoC), each infinitesimal segment of which contributes to the applied force.

These complexities are why the ball-cushion interaction is the least accurately modelled aspect of pool physics simulations.

Did you notice that the ball pops into the air post-collision? This happens because the apex of the cushion is at a height greater than the ball’s radius, and so the outgoing velocity of the ball has a component that goes into the table. Consequently, the slate of the table applies a normal force to the ball, popping it up into the air.

Importantly, I want to distinguish between the ball-slate interaction and the ball-cushion interaction: the ball-cushion interaction creates the outgoing velocity of the ball, which under most circumstances has a component into the table. An infinitesimally small amount of time later, the ball-slate interaction occurs, which prevents translation into the table’s surface, and in some cases pops the ball into the air if its speed is great enough. This section deals strictly with the ball-cushion interaction, and in the next section I will treat the ball-slate interaction.

Given the complexity of the ball-cushion interaction, I must admit I feel a little in over my head. So far, I have considered 3 models: Mathavan et. al, 2010; Marlow, 1994; Han, 2005. Let’s take a look.

(1) Mathavan et. al, 2010

After searching the literature, the most complete treatment I have found is this work by Mathavan et. al (2010) entitled, “A theoretical analysis of billiard ball dynamics under cushion impacts”. They develop a model that must be solved numerically using differential equations, which is relatively complex. To get a rough idea of how involved this model is, check out the force body diagram in Figure 4:

Force body diagram from Mathavan et. al, 2010. source

This is the kind of thing that takes a long time to wrap your ahead around the meaning of the variables. Despite the model’s complexity, it is still very simple in comparison to reality because it assumes the cushion deformation is insignificant, and thus the interaction instantaneous. Despite this being a large departure from reality, they report really good agreement with experiment.

This is definitely the best model I could find that I would be willing to implement, but at this moment in time I don’t really want to solve differential equations on-the-fly every time there is a ball-cushion interaction. Perhaps I could solve the differential equations for the entire parameter space and then parameterize the solution space somehow, but for now I would like to avoid this model altogether and look for something simpler.

(2) Marlow, 1994

So Marlow has a book called “The Physics of Pocket Billiards” that I use as a reference for pool physics. In general, its very comprehensive, but in the case of the ball-cushion interaction he presents an incomplete and inconsistent treatment. Moving on.

(3) Han, 2005

I enjoy this treatment by Han, 2005, entitled, “Dynamics in carom and three cushion billiards”. It assumes instantaneity and negligible cushion deformation. It is simple, analytic, and appears to be physically plausible. That said, it is less realistic than Mathavan et. al, 2010.

Alright, so let’s go over the model. Han chooses the following frame of reference:

Figure 11. A ball colliding with a rail viewed from above. The frame of reference is defined so that the rail is perpendicular to the $x-$axis, and the $+x=$direction points away from the playing surface. The incoming velocity makes an angle $\phi$ with the $x-$axis.

At the instant of contact, $t=\tau$, the ball has a state $(\vec{r}(\tau), \, \vec{v}(\tau), \, \vec{\omega}(\tau))$, and immediately afterwards it has the state $(\vec{r}(\tau + dt), \, \vec{v}(\tau + dt), \, \vec{ \omega}(\tau + dt))$. Since Han assumes instantaneity, $dt$ is an infinitesimal amount of time. Since there is no funny business going on with cushion deformation, we know that the position at $\tau$ will equal the position at $\tau + dt$:

\[\vec{r}(\tau + dt) = \vec{r}(\tau) \notag\]

Resolving the velocity and angular velocity requires some geometry of the ball-cushion interface.

Figure 12. A ball colliding with a rall viewed from the side. It is the same scenario as in Figure 11. $R+\epsilon$ defines the height of the cushion, where contact is made with the ball. $\theta$ is determined uniquely by $\epsilon$, and defines the direction of the impulse the cushion imparts on the ball.

One of the most important determinants in this model, and in reality, is the height of the rail. This will determine the direction that the cushion applies force to the ball. The higher the cushion, the more the cushion redirects the ball into the table. Also, the higher the cushion, the more $y-$axis torque will be applied to the ball. The cushion is specified by $\epsilon$, the height above center ball, and uniquely determines $\theta$, the angle that the line between the PoC and the center of the ball makes with the $x-$axis:

\[\theta = \arcsin(\epsilon/R) \label{eps_theta}\]

You may wonder, why not have $\epsilon = 0$? Then $\theta=0$ and the ball won’t be redirected into the table.

This is idealistic thinking for 2 reasons. In practice, it is not uncommon for balls to leave the table when struck hard. If an airborne ball hit a rail with $\epsilon = 0$, it would go flying, so $\epsilon > 0$ protects against that.

Yet even if a ball remained firmly planted to the table, consider a rolling ball striking the cushion perpendicularly. It’s spin is in the $+y-$direction of Figures 11 and 12. When in contact with the rail, the rolling spin tries to push the cushion down, and in an equal and opposite manner the cushion pushes the ball up, sending the ball airborne.

If you’ve ever played on a table with rails that pops the balls up, its probably because $\epsilon$ is too low. For these reasons, in practice $\epsilon$ is typically $0.1 R$ to $0.2 R$. This isn’t a perfect solution to the airborne problem however, because as we saw in the slow mo video, redirecting the ball into the table can also cause balls to become airborne via the ball-slate interaction. That said, redirection into the slate is preferable to redirection into the air, since the slate’s low coefficient of restitution dampens most of the energy.

Using rigid body dynamics, Han solves the outgoing ball state for the scenario depicted in Figures 11 and 12.

After surveying the equations for consistency, I found 2 mistakes, that I outline in this worksheet. The equations below have these mistakes accounted for.

ball-cushion interaction (Han 2005 model)

Let all of these quantities be:

\[s_x = v_x(\tau) \sin\theta - v_y(\tau) \cos\theta + R \omega_y(\tau) \label{han_sx}\] \[s_y = -v_y(\tau) - R \omega_z(\tau) \cos\theta + R \omega_x(\tau) \sin\theta \label{han_sy}\] \[c = v_x(\tau) \cos\theta \label{han_c}\] \[I = \frac{2}{5} m R^2 \label{han_I}\] \[P_{zE} = m c \, (1 + e) \label{han_pze}\] \[P_{zS} = \frac{2m}{7} \sqrt{s_x^2 + s_y^2} \label{han_pzs}\]

Displacement:

\[\vec{r}(\tau + dt) = \vec{r}(\tau)\]

Velocity (if $P_{zS} \le P_{zE}$):

\[v_x(\tau + dt) = -\frac{2}{7}s_x \sin\theta - (1+e) \, c \cos\theta \label{han_vx_1}\] \[v_y(\tau + dt) = \frac{2}{7}s_y \label{han_vy_1}\] \[v_z(\tau + dt) = \frac{2}{7}s_x \cos\theta - (1+e) \, c \sin\theta \label{han_vz_1}\]

Velocity (if $P_{zS} > P_{zE}$):

\[v_x(\tau + dt) = -c \, (1+e) (\mu \cos\phi \sin\theta + \cos\theta) \label{han_vx_2}\] \[v_y(\tau + dt) = c \, (1+e) \, \mu \sin\phi \label{han_vy_2}\] \[v_z(\tau + dt) = c \, (1+e) (\mu \cos\phi \cos\theta - \sin\theta) \label{han_vz_2}\]

Angular velocity:

\[\omega_x(\tau + dt) = - \frac{m R}{I} v_x(\tau + dt) \sin\theta \label{han_ox}\] \[\omega_x(\tau + dt) = \frac{m R}{I} (v_x(\tau + dt) \sin\theta - v_z(\tau+dt) \cos\theta) \label{han_oy}\] \[\omega_z(\tau + dt) = \frac{m R}{I} v_y(\tau + dt) \cos\theta \label{han_oz}\]

Section IV: ball-air interactions

FIXME

Section V: ball-slate interactions

FIXME

Section VI: ball-cue interactions

FIXME

Conclusion

That’s everything–at least for now. My knowledge will (hopefully) increase over time, and when it does, it will be added to this post.

Next time I will be discussing the various algorithms for simulating a pool shot, and the approach I intend to take for my own simulation, which is called continuous event-based shot evolution.

The physics of pool/billiards was originally published by Evan Kiefl at Evan Kiefl on April 24, 2020.