Introduction

I haven’t posted in a while and I’m intending to write more often. I will start this new era with a non-techincal blog post about my life. In the beginning of May 2025 I went to Vietnam to visit the place my lovely girlfriend was born and raised. I met her amazing family and felt the most welcome I’ve ever been. So the best way to do them justice to my 10-day escapade is to write down my impressions and keep them for eternity in the data pit that is internet (probably will be used to train some AI). I will be detailing my experience in Vietnam through multiple lenses such as: the food, the life, the history and the nature. At last, I will be providing my opinion on my experience and some tips on how to best experience Vietnam if you happen to visit it.

The Welcome

After a long 2-connection flight to Vietnam I got greeted by this, a warm slightly exaggerated welcome by Jamie (aka. Ngoc) and her brother Trong.

This will remain my most memorable welcome by far.

The Food

I can’t be an experience blog post without a description of the food. After all, “a man’s love is through his stomach”.

The Vietnamese cuisine is defined by its harmonious balance of flavors, textures, and colors. The overarching theme centers around freshness and contrast: crisp herbs, savory broths, and vibrant vegetables are paired with tender meats, seafood, and rice or noodles flavored with variants of fish sauce. Except for the savory broths, most of the food is basically unflavored until you dip it in a sauce.

The fish sauce is the heart and soul of the cuisine, providing that umami and salty concentrated flavor. It forms the foundation for many essential sauces, such as nuoc cham (a tangy dipping sauce for spring rolls and grilled meats), nuoc mam gung (ginger fish sauce for poached chicken), and nuoc mam pha (a blended fish sauce for salads and noodle dishes). These sauces, each with their own blend of lime (or green kumquat), garlic, chili, and sugar, highlight the versatility and central role of fish sauce in Vietnamese cooking. I was surprised when I learned that the fish sauce is not a sauce made for fish food, but made WITH fermented fish.

Dishes are designed to be shared, encouraging communal eating and conversation. Whether it’s the iconic pho, fragrant bun cha, or simple street-side banh mi, every meal reflects a deep respect for natural ingredients. For example, the photo above shows the first dinner I had with Jamie’s family and relatives. I was greeted and welcomed by everyone. It was especially surprising the way they shared the food with me. It’s considered respectful to pick up food from the shared dishes and drop it on someone’s plate – I was constantly offered prerolled rolls, from the topmost plate in Figure 2. That was my favorite dish in this picture – it contained pig ears and some green leaves that where put there to be used as the rolling base (same as the rice paper is for spring rolls). In fact, most of the plates come with these large leaves that are used as the roller base.

Bia Ha Noi and the Drinking Customs

The pride of Hanoi is its crafted beer, Bia Ha Noi. This beer has a mere 4.5% alcohol, but my it managed to hit me at 7 glasses. It tastes similar to a white beer like Heineken, but is less bitter and easier to drink. It’s actually pretty light and it doesn’t get in the way of the flavor of the food.

When it comes to drinking, my impression is that people here gather and enjoy drinks together for hours on end. People rarely drink alone; instead, everyone shares the experience. It’s customary to clink glasses with at least one other person, and if only two people are drinking, they shake hands after each toast. I really liked the idea that no one drinks alone—it truly feels like someone is always there to share the pain, joy, or boredom with you. This theme of social life is quite common in Vietnam, and I really enjoyed it. Even though I don’t look Vietnamese, I was welcomed into this social circle and it made me feel at home.

The coffee: Cà phê phin

Of course it can’t be a post about Vietnam without mentioning the coffee. Cà phê phin is traditional Vietnamese way of brewing coffee. It is brewed using a small metal drip filter called a phin. Coarsely ground dark-roast coffee—usually robusta—is placed in the filter, hot water is poured over it, and the coffee VERY slowly drips into a cup below. The process takes several minutes and results in a strong, concentrated brew. It’s often served black or mixed with sweetened condensed milk, either hot or iced. This method highlights bold, earthy flavors and delivers a powerful caffeine kick. As a coffee connoiseur (aka. addict) I have to admit this is by far one of the strongest coffees I’ve ever had. Unlike arabica, this one has a more distict sourness. Below is a photo I took while enjoying a “Chari” coffee at the famous Ca phe Mai in Hanoi. I ended up buyin $60 worth of coffee from them including the “Paris Mai”, which apparently is pooped out by luwak, a squirrel-like animal.

My Favorite Dish: Bánh Cuốn

Bánh cuốn (steamed rice rolls) quickly became the highlight of my culinary journey in Vietnam. For breakfast, Ngoc, Trong, and I decided to ride to a tiny street food shop famous for it. There, the cook skillfully ladled a watery mixture of rice flour onto a thin cloth stretched over a pot of steaming water. The steam, cooked the batter into delicate, translucent rice sheets. These sheets were then filled with a savory blend of minced pork and finely chopped wood ear mushrooms, creating a combination of the squishy rice sheet and the chunky fillings.

The rolls were served with a side of sweet and sour dipping sauce, the same one used on spring rolls, and topped with crisp fried shallots. Each bite was a perfect harmonious dance of crunchy, silky and chunky/squishy textures as well as an elegant balance of sweet, sour, tangy and savory flavors. On the side, we ordered slices of pork sausage dipped in the sauce, complementing each other’s savory flavors. All in all, it was a dish that masterfully mixed flavors and textures beyond what I’ve ever had before.

Bún Riêu: Crab Noodle Soup

This dish is the most colorful and chaotic dish, in my opinion. It’s exactly what its name suggests, AND A LOT MORE. Straight up you will notice a vibrant red broth made from tomatoes and crab paste. The soup is filled with rice noodles, fried tofu, and chunks of crab meat. On top, there are green onions and fried shallots. The broth itself is tangy, slightly sweet, and savory, making it a great comfort food. It was a dish that surprised me with its complexity.

Most importantly in the figure above you should notice the side dish, fried bread. I’ve had fried bread in USA, but this one tases more crunchy. The way to eat it is to just put it in the soup and let it soak the broth – similar to how we eat regular bread in Albania with some stews.

My least favorite: Goat meat

After our boat trip in Tràng An (where King Kong was filmed), in Figure 6, we decided to have lunch in a famous restaurant where they serve goat meat. Unlike Albania, in Vietnam goat is not a common meat, so having a restaurant serve goat meat is rare.

In the photo below, you can see the final spread of all the dishes we ordered. We went with medium-rare goat meat with lime (the shredded one) and goat meat with fried garlic (the chunkier dish). As is typical, both came with a generous side of fresh plant leaves, which serve as a base to wrap and eat the meat with.

Surprisingly, what stood out the most to me wasn’t the food—it was the tea. We tried kumquat tea and a fruit tea, which I believe was dragonfruit. The fruit tea, in particular, was incredibly refreshing—especially after a day of hiking and exploring the ancient imperial capital of Hoa Lư.

Unfortunately, I have to admit the goat meat was a bit underwhelming. It was quite chewy, and since it was cooked with the skin on, it made things even tougher. Even the more shredded dish was hard to get through. On the bright side, the sides really made up for it—especially the little onions, which were a standout.

The controversial: Bún Đậu Mắm Tôm

This dish was amazing. It’s ranked #2 in my personal ranking. I really enjoyed the Mắm Tôm (shrimp paste sauce). The is dish made up of rice vermicelli (bún), crispy fried tofu (đậu), and a variety of accompaniments like boiled pork, chả quế (fried cinnamon pork sausage), and of course fermented shrimp paste–marinated intestines. The centerpiece is the shrimp paste. It’s a pungent, salty, deeply umami dipping sauce. The sauce is divisive even among the Vietnamese themselves due to it’s incredibly stron flavor. The sauce is mixed with lime juice, sugar, and chili to balance its intensity – so when you receive it you have to violently mix it until bubbles are formed. I enjoyed it because it was unique and I don’t plan to eat the pork just plainly boiled like that. After that dish Jamie and her brother gave me their seal of approval for becoming a Vietnamese citizen – I’m on the right track.

The dawg

Staying on the theme of controversy, I’ll now review one of the most unusual dish I tried: dog meat (thịt chó). This traditional specialty, especially common in northern Vietnam, is often grilled and paired with fermented shrimp paste (mắm tôm). The flavor was bold and gamey—reminiscent of goat meat, which is probably why I enjoyed it. My only critique was the chewiness, which stood out even more since I was still partially recovering my sense of taste at the time. In general, Vietnamese cuisine doesn’t seem to shy away from a bit of bite in its meats. Back in Albania, for instance, we tend to pressure-cook goat to make it more tender.

On the ethics of eating dog meat, I don’t side with what I see as selective outrage. I believe all animal life is equal (humans aside), and if someone is comfortable eating an octopus—an animal arguably more intelligent than a dog—it’s hard to justify placing dogs on a higher moral pedestal. Spare me the sentimentality: we are not divine arbiters deciding which species deserve to live and which do not.

Fertilized duck egg: Balut

If you thought dog meat was the peak of culinary controversy, meet balut—a fertilized duck egg with the embryo still inside, boiled and eaten straight from the shell. It’s a common street food, but to Western eyes, it’s the kind of thing that makes people clutch their stomach before they’ve even taken a bite. Below is a picture of a 17 day old fertilized duck egg.

Crack it open and you’re greeted by a mix of textures: tender egg white, rich yolk, and the slightly firmer, delicate crunch of developing bone and beak. Flavor-wise, it’s deeper and more savory than a chicken egg—closer to a meaty broth trapped inside an eggshell. You sprinkle some salt, maybe a dash of vinegar or herbs, and slurp it down while it’s hot.

I liked it. Not in the “oh, it’s weird but I’ll pretend to enjoy it for the story” way, but genuinely liked it. The broth-like richness was addictive, and the mix of textures worked for me. The only reason people recoil is because it forces them to confront the fact that meat comes from living things—something most conveniently forget while buying their sanitized, shrink-wrapped chicken breasts.

As for the morality debates? Please. If you’re okay with eating a full-grown bird, then eating it a few days earlier doesn’t suddenly make you a monster – it arguably spares the bird a lifetime of torture in a cage.

Agriculture

Due to the tropical climate Vietnam bolsters a vast selections of agriculture, many of which are not available in the US or Albania. I had for the first time sugar cane juice (absolutely love it), lychee, mangosteen, jackfruit and even coconut. I really liked mangosteen in particular. It’s a sweet, tangy, and juicy fruit, with hints of peach, strawberry, and citrus.

The nature

The nature in Vietnam is nothing short of fascinating. I came to realize this when Jamie and I went to Ninh Bình. We booked a boat trip to discover the beauties that attacted the movie producers of Kong: Skull Island. While gliding along the calm green waters of Tràng An I was surrounded by towering limestone karsts that rose dramatically from the river like ancient sentinels.

Draped in dense greenery, these cliffs created a scene that felt almost otherworldly, their rugged surfaces contrasting beautifully with the lush jungle below. The moody, shifting skies added a mystical atmosphere, casting patterns of light and shadow that danced across the water’s surface.

Our small wooden boat, hand-rowed by a skilled local boatwoman, moved quietly through the winding waterways. The gentle splash of the oars was the only sound breaking the tranquil silence. Around us, other boats carried travelers in bright orange life jackets, each person lost in awe at the breathtaking landscape. Tràng An’s peaceful beauty and grand scale made it one of the most unforgettable highlights of my journey through Vietnam.

I want to take a moment to acknowledge the wonderful, welcoming, and deeply underappreciated boatmen and women who work tirelessly to show visitors the natural beauty of Vietnam. Their warmth and generosity left a mark on me. After our tour ended, all the rowers gathered to eat the simple lunches they had brought from home. We joined them with our own snacks, and before long they were offering to share their food and asking about my story. It hit me hard, knowing what I’d learned earlier—that they earn barely 200,000 VND (about $7.60) for a single five-hour trip, and even in high season, they might only get one trip a week. Many are forced to take other jobs just to get by. And yet, despite such modest means, they still sat with a foreigner and insisted on sharing what little they had—proving themselves richer than many in the only currency that truly matters: a heart of gold.

The history

When it comes to history, I’ve always been fascinated. Before this trip, though, my knowledge of Vietnamese history was limited to a few YouTube videos and the one-sided perspective of the horrific Vietnam War. Through my dad’s stories, I learned that communist Albania at the time was a strong supporter of Vietnam, with radio broadcasts proudly reporting every achievement of the communist forces against the “imperialist American allies.”

Meanwhile, in the U.S., most of what I learned portrayed the Americans as justified and the Viet Cong as the villains. Given the more recent wars in the Middle East, I’ve come to approach these narratives with a grain of salt. I wanted to learn more about Vietnam’s history before the war to better understand what led up to the conflict, and to hear perspectives on the war from a different point of view.

The French colonialism

Even before my visit, I learned that the French had a major influence on Vietnam. Many buildings in Hanoi showcase classic French architecture, with features like ornate facades, tall arched windows, wrought-iron balconies, and wide boulevards lined with trees. The French introduced European-style villas, grand government buildings, and cathedrals, which still stand today as striking reminders of that era. Walking through Hanoi’s Old Quarter and French Quarter felt like stepping into a unique blend of Vietnamese and European styles. Colonial buildings now house charming cafés, boutique shops, and small museums, all contributing to the city’s distinct atmosphere.

In fact, French influence goes beyond architecture—some Vietnamese food is also inspired by French cuisine. One of the most famous examples is bánh mì, a crispy baguette sandwich that combines French bread, pate (pork liver) with local ingredients like pickled vegetables, cilantro, chili, and various meats or pâté. Dishes like bò kho (beef stew) and pâté chaud (meat-filled puff pastry) also reflect this fusion. It’s fascinating how colonial history has left such a tangible mark on daily life, not just in the buildings and streets, but in the food people eat and the way the city feels.

Life under a colonizer isn’t all fun and brioche. In fact, I took my first brush with a grim history lesson from a guided tour of the Hỏa Lò prison in Hanoi (a.k.a. the “Hanoi Hilton”). A small part of this man-made building of horrors is still kept in the heart of Hanoi – reminding the people of the resiliance and strength of their revolutionary freedom fighters.

Hỏa Lò Prison was built by the French colonial administration around 1896 in central Hanoi as part of France’s efforts to consolidate control over Indochina. It was the highest security prison in the area – in fact, they mandated that metal used on the door locks must come from France (Fig.14). The French bought the area for dollars on the dime from the residents of Phu Khánh village, which was well-known as a traditional ceramic and clay pot–making community. Known to the French as Maison Centrale, it was one of the largest prisons in French-occupied Southeast Asia, designed to hold up to 450 inmates but often crammed with thousands. Hỏa Lò was primarily used to detain Vietnamese revolutionaries, nationalist intellectuals, and anti-colonial activists, becoming a symbol of French repression and a rallying point for resistance movements seeking independence from colonial rule.

Many prominent communist revolutionaries were imprisoned at Hỏa Lò during the French colonial era. Ironically, what was meant to break their spirit became a breeding ground for resistance. Inside its grim walls, detainees formed secret study groups to spread Marxist-Leninist ideas, trained one another in political theory, and laid the ideological foundation for Vietnam’s future leadership. Among those held were Trường Chinh, who later became General Secretary of the Communist Party and a key architect of socialist policy in Vietnam, and Nguyễn Văn Cừ, also a Party leader and one of the youngest General Secretaries in its history. Their time in prison was not wasted—it became a period of political consolidation and underground organizing.

Escape stories from Hỏa Lò are legendary and remain a source of pride in Vietnam’s revolutionary narrative. During WW2, when Japan displaced the French colonial administration and occupied Vietnam, the prison system descended into turmoil. As a result, security weakened, and the frequency of successful prison breaks surged. The communist leadership highlights these escapes not just as acts of personal bravery, but as strategic moves that enabled revolutionaries to rejoin the anti-colonial struggle. Today, these stories are preserved as symbols of resilience and the unwavering will for national liberation.

Vietnam war

Naturally, no history of Vietnam would be complete without mentioning the civil war between the North and the South. In Hanoi, reminders of that turbulent period are everywhere—museums, monuments, and preserved war relics. One of the most notable sites is the Hỏa Lò Prison, which includes a section dedicated to its role during the war.

The prison (again)

After Vietnam was liberated from Japanese occupation, the new socialist government took control of the North. The prison was repurposed to hold American prisoners of war. Conditions were reportedly improved from the harsher state left by the French; prisoners had straw beds, access to games, and basic hygiene supplies. In a place under heavy bombing, concentrating prisoners in one location increased their odds of survival since they wouldn’t bomb a prison. This holds true, especially when many Vietnamese civilians were being bombed indiscriminately in the streets. American POWs famously nicknamed it the “Hanoi Hilton.” While the term was used sarcastically, it’s possible that the prison was statistically safer than being outside, where napalm strikes and friendly fire posed threats to GI’s.

The B-52 museum

Another spot worth mentioning for its display of war spoils is the B-52 Museum. As the name suggests, it’s dedicated to the American B-52 bomber—an aircraft that struck both fear and deep resentment into the hearts of the Viet Cong. Bringing down one of these ruthless, indiscriminate war machines was a major morale boost for fighters on the front lines. Back in the 1970s, such victories were even broadcast and celebrated over Albanian radio.

What really caught my attention was how both men and women were trained to operate anti-aircraft gun turrets, with many positioned strategically around Hanoi to protect the city. A detail I learned from Jamie’s dad made the picture even grimmer: to evade radar detection, B-52s would dump massive amounts of metal chaff. While intended to confuse tracking systems, those chunks still rained down on homes below, damaging property and sometimes injuring civilians.

Not only did they rely on anti-aircraft gun turrets, but they also deployed Soviet-supplied S-75 Dvina surface-to-air missiles—far more accurate than manual gunfire, but expensive and limited in number. Designed for high-altitude targets, these missiles were the real B-52 killers, able to bring down bombers before they reached their drop zones. Each launch had to count, as wasting one on a decoy wasn’t an option. When a missile did connect, it wasn’t just a tactical win—it was a statement that even America’s most advanced bombers could be blown out of the sky.

The MiG-21 was one of the North Vietnamese Air Force’s key weapons against B-52 bombers, and it went on to become one of the most produced supersonic aircraft in history. The museum features displays honoring distinguished pilots, complete with their kill counts and the aircraft they intercepted.

At the entrance of the museum, visitors are immediately greeted by a striking symbol of conflict—a destroyed B-52 bomber, a powerful reminder of the war’s enduring impact.

Uncle Ho (Bác Hồ) (aka. Hồ Chí Minh)

Hồ Chí Minh was the figurehead of North Vietnam’s unification war. From what I was told, he was a humble man who rejected the riches and spoils of war in favor of his people—sometimes with a frightening level of efficiency. As I saw for myself, he lived in a modest stilt house built beside a small pond, choosing it over the opulent French colonial governor’s palace just a short walk away. His sandals were cut from worn-out car tires, and his plain khaki suit was worn so often it became part of his public image. He personally read letters sent to him by ordinary citizens, sometimes giving direct orders to solve their problems without waiting for bureaucracy to grind into motion. Even in wartime, he was relentlessly pragmatic—approving strategies that made full use of every available resource, and refusing to waste manpower, supplies, or time, no matter the sentiment attached.

He is still deeply revered in Hanoi today. His body lies preserved in the mausoleum, where visitors file past in silence. After our quiet walk around him, we explored the building where he once worked and met with other leaders. What struck me most was learning that he had maintained good relations with the Albanian government at the time. In Fig. 20 you can see the meeting room where he hosted these leaders—preserved exactly as it was, frozen in time, much like the man himself.

Even the Vietnamese diaspora in France praised his efforts. As a gesture of appreciation, they presented him with a Peugeot 404.

Conclusion

Vietnam is a fascinating country rich in history and culture, home to a resilient and enduring people who have overcome countless challenges throughout the centuries and are in an upwards trajectory of economical growth. Their strength and spirit are deeply woven into the fabric of the nation, reflected in everything from vibrant traditions to rapid modernization. I would strongly encourage anyone to visit this country and experience the culture, tradition and the welcoming of the Vietnamese. I have never felt more welcome in a foreign country, where people would sit down and share their food with you just to hear your story and represent their country in the most positive light they personally could. Here are a few more recommendations from the official website.

Introduction

I’ve seen a lot of papers in the academia talk about static analysis in linux kernel’s source code. Some use old tools like KINT, some others just mention an LLVM use-def pass, but for a newcomer it’s difficult to write all of those on their own. In this blog post, I will go step by step into how to track a variable allocation, its data type and it respective kmalloc cache. I will be running everything in Ubuntu 20.04 but it can work in any OS if you try hard enough. For this analysis we will try to infer the kernel cache for each kernel object.

In this mini-project I will be using clang-13 with linux kernel 5.13.14 (same kernel as in my other blog post where I exploit CVE-2017-27666).

Setup

We start off with downloading and extracting the linux kernel code as follows:

wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.13.14.tar.gz
tar -xf linux-5.13.14.tar.gz
rm linux-5.13.14.tar.gz

Then we need to download clang-13 specifically release tag llvmorg-13.0.1 from github (check the code below). After downloading we patch it in such a way that when we compile the kernel we dump the LLVM bitcode before the optimizations happen (the linux kernel must be compiled with -O2 flag so we can’t turn it off). This patch allows the compiler to write the bitcode file with extension .c.bc for each .c file that is compiled. Download the patch from here. In short, you can make this clang using the following commands:

# setup llvm-13
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
git checkout llvmorg-13.0.1
git clone https://gist.github.com/f81771e4a8ea6cce8ec79279eb07d18c.git llvmpatch
mv llvmpatch/llvm_write_bitcode_13.patch ./
git apply llvm_write_bitcode_13.patch

# now compile clang in RELEASE mode
cmake -S llvm -B build -G Unix Makefiles -DCMAKE_BUILD_TYPE=Release
cd build
make -j$(nproc)

Note that these commands work for Linux. To build this on Windows just read their README from the github repo. If you are using TempleOS, unfortunately Terry Davis didn’t consider llvm-project holy enough to be supported(I made that up).

Compiling the kernel

Now that you have clang compiled let’s go back to the Linux kernel directory and compile it (after make defconfig) using this command line: make CC=<path to: llvm-project/build/bin/clang> -j$(nproc)

After compiling you should be getting an output that looks like Figure 1. Now in the compiled kernel directory we have a bunch of files that look like Figure 2. As you can see below we have a bunch of .c.bc files.

Analyzing the kernel

As a quick start lets download a modified version of KINT and compile it. This version is a modified one from the DirtyCred’s code. First let’s clone the github repo and compile it. The analysis binary is in ./build/lib/analyzer. We feed every single .c.bc file generated by the compiler as an argument for the analyzer and run it. To do that we go to ./build/lib/ and run this command in a bash shell that supports this weird apostrophe: ./analyzer `find <ROOT OF KERNEL SOURCE> -name "*.c.bc" ` 2> struct_cache_res.txt. The command first executes the find in the root of the linux kernel and prints out a list of all files that end with .c.bc. This list is the fed as parameters to the analyzer program. The 2> makes both stdout and stderr be written into struct_cache_res.txt file. This way all the output of the analyzer is stored in a file to be processed later as needed. When we run this command (after removing the file dump part) on the kernel we compiled above we get the results in Figure 3.

Explanation

Now that we know we can do this, let’s dive deeper into how it works exactly. Since this code has been repurposed by many other people, it has many variables that are not useful to us. The meat of the analyzer is in src/lib folder, so go in that directory and open VScode using code . command. The program starts from KAMain.cc. In line 162 the program loops through all the arguments (the .c.bc file paths), reads the LLVM IR (Intermediate Representation) and parses it into a LLVM::Module object. In the end it runs doBasicInitialization function that parses all the structures defined in that Module and populates it with info such as the fields/members and the struct size. This will be useful later to determine the cache based on the struct size. The thing that I modified is printAllStructsAndAllocCaches(); from StructAnalyzer.cc. The new function now calls getAllocCache() (in StructAnalyzer.h) for each structure. In Figure 4 you can see part of that code.

At first(line 187), the code loops through all the allocation sites and checks if the function that allocates the structure is kmalloc or other generic allocation functions (as shown in Figure 5). If any of them is not, then it will go into the first argument of the special function and grab the cache. For example kmem_cache_alloc takes as the first arguments the pointer to the cache. This pointer is usually a global variable, which is first NULL. At runtime this global variable will point to a kmem_cache structure, which is created/allocated after calling kmem_cache_create. My code goes to this global variable and checks if its data type is kmem_cache structure (line 218). If it is, then we loop through all of its users (the LLVM instructions that read or write this global variable) as shown in line 219 in Figure 5. After kmem_cache_create is called the return value needs to be STOREd into the global variable; not assigned, even though it looks like global_var = kmem_cache_create(...);. So the code finds the StoreInst from the users (line 220). In line 222 it goes backwards towards the source (the create function call) and in checks if the function call is kmem_cache_create (line 223). If true the code then extracts the first argument of this function calls which is a hardcoded string containing the name of the cache as shown here.

Future work

Of course, there is more room for improvement. When dealing with generic allocation calls (ie. kmalloc) we simply consider the objects to fall into the closest cache that fits it, however this may not always be the case. We can improve this by going backwards from the call graph of these functions to find the kmem_cache from kmalloc_slab function.

In this blog post I publish the results of a project I did during Thanksgiving. While eating an unhealthy amount of turkey I brought up the topic of AI and warned my parents about the risks of its misuse. One such risk being the biggest nightmare of every security practitioner – AI lowering the bar for petty skids to hack real systems after they got bored taking L’s in Fortnite.

Security is not a solved problem and I don’t see it being one in the near future (more code = more jobs). As security researchers, our job is therefore to increase the bar, such that an adversary with a low enough budget doesn’t do anything. As I was making the case, at some point in the conversation I thought: What if I can use a large language model (LLM) to help me with fuzzing a linux kernel module that I have no idea of?

Background

Fuzzing the Linux kernel is a widely studied topic. There are various fuzzing tools, such as Triforce AFL, HFL, Syzkaller, etc, the latter being the defacto state-of-the-art open-source kernel fuzzer. Syzkaller relies on kernel experts’ manually-written Syzlang descriptions to generate valid system calls. A description is a file written in syzkaller’s language (Syzlang) that describes how to correctly invoke a systemcall, using the correct parameters, structures and dependencies. For example a write into a file requires one to open the file and use it’s return value as a parameter for the earlier.

In Fig. 1 I’m showing an example of a Syzlang description (called description from now on) for the aio module. The description contains the following:

1. The required includes to write a valid C program. (yellow highlight)
2. The resources that need to be opened, such as file descriptors. (green highlight)
3. struct definitions that allow the fuzzer to accurately fuzz complex structures (yellow box)
4. const, enum defitions that restrict the fuzzer to values that make sense (pink bracket)
5. The system call defitions that allows the fuzzer to know what parameters to use at which positions and what data type they are. As shown in the figure, some parameters can be resources too. The fuzzer is made aware that it needs to initialize the resource using io_setup (blue highlight) by passing its integer pointer as an argument.

Experimental setup

As mentioned earlier, these descriptions are written manually by kernel developers. There have been efforts to automate the generation or even reduce these descriptions [1,2,3,4, 5], which is beyond the scope of this blog post (but an interesting followup nonetheless). Regardless they all come with some built-in assumptions, and wouldn’t generalize well for a random kernel subsystem. My assumption instead is that a kernel subsystem has a well-written documentation that instructs other non-kernel developers how to use it’s exposed system calls to interact with it. With this assumption in mind I hypothesize that a Large Language Model (referred as LLM from now on) can read it and understand how to translate that human-readable documentation (ie. a massive HTML page) into a description for Syzkaller. To test this assumption I picked a fairly complex subsystem, the KVM. This subsystem has a rich description in Syzlang, clocking more than 2000 lines.

In summary, the experiment is as follows. First, I chose to enable fuzzing on the first 40 system calls from dev_kvm.txt, such as openat$kvm, ioctl$KVM_CREATE_VM, etc. Second, I ran the fuzzer for 9 hours using the original manually written descriptions. Third, I found the documentation of KVM module from this article and fed it to Bing AI. Fourth, I removed everything from dev_kvm.txt and populated it with what the AI wrote to me. At last, I ran the fuzzer again for 9 hours using the AI-written descriptions.

A chat with the quickest reader

In fourth step, I chat back and forth with Bing AI. First I primed it by making sure that it reads the web page (that contains the KVM documentation) and answers using that as the reference. I believe Bing AI is using a form of information retrieval system on top of the LLM as in here [5], which makes it such that the queries are always referred to text from the documentation page. I opened the tab on the side using the Microsoft edge’s Bing button ans started asking the qeustion. In the figure below (Fig. 2), I am showing the first message I sent to Bing AI using GPT-4. I asked it to tell me what it reads and see if it can interpret the web page. As it can be seen, the AI was fully capable of reading the article and summarize the syscalls in 4 classes of system calls. This is not a simple rip-off of the original summary, because the part underlined in red is not there in the general description section of the documentation. This means that the GPT was able to read though each of the individual system calls and conclude that check extension is a feature of the system ioctls. As also shown in the figure there are plenty of other cases where the GPT is able to provide more concrete examples of each class of syscalls. In the device ioctls however it fails to mention that they can only be called by the process that created the VM, but it’s not a big deal in my opinion.

Next, I asked the GPT to generate a program that uses KVM_CREATE_VM and it wrote it in C. To save some space, I am not posting the interactions that were needed to ask it to write the C program. I then tested it to make sure it understood how Syzlang works and whether it can translate the KVM_CREATE_VM syscall back into Syzlang. The response is shown in Fig. 3. As you can see, the model is capable of understanding how Syzlang works and what information from a C program are required by it. Additionally it not only wrote the syscall I asked it, but also the syscall ioctl$KVM_CREATE_VM is dependent on, openat$KVM.

We need to go deeper…

As you can see above the ioctl$KVM_CREATE_VM syscall is correct, but it’s a specific one, so it cannot be used to fuzz, because none of the parameters can be modified by the fuzzer. So naturally, I asked GPT to make a generic version of this system call. In Fig. 4 you can see the output of GPT. It’s asstounding how it not only wrote a generic Syzlang representation but also explained every parameter, the dependencies and all the possible flag values and the header where they are defined. I was shocked at how well it described the system call and how deep was it’s understanding. This were my emotions when I read this output .

Well, now the genie is out of the bottle. Let’s just ask our overlord for more blessings. In Fig. 5 you can see our conversation. For the sake of brevity, I won’t be posting the entire result, only the beginning of it. The GPT hit the word limit and it didn’t generate all the syscall descriptions.

As I showed earlier there are 4 things in a description file: (1) the includes, (2) the resources, (3) the syscalls and (4) the struct and flag definitions. We have tackled the first 3. Now let’s ask GPT to tackle the 4th too and bundle up all into 1 so that we can copy and paste. In the same conversation I asked GPT:

write me a syzkaller description with all the kvm syscalls mentioned below:


openat$kvm
openat$sgx_provision
ioctl$KVM_CREATE_VM
ioctl$KVM_GET_MSR_INDEX_LIST
ioctl$KVM_CHECK_EXTENSION
ioctl$KVM_GET_VCPU_MMAP_SIZE
ioctl$KVM_GET_SUPPORTED_CPUID
ioctl$KVM_GET_EMULATED_CPUID
ioctl$KVM_X86_GET_MCE_CAP_SUPPORTED
ioctl$KVM_GET_API_VERSION
ioctl$KVM_CREATE_VCPU
ioctl$KVM_CHECK_EXTENSION_VM
ioctl$KVM_GET_DIRTY_LOG
ioctl$KVM_CREATE_IRQCHIP
ioctl$KVM_IRQ_LINE
ioctl$KVM_IRQ_LINE_STATUS
ioctl$KVM_GET_IRQCHIP
ioctl$KVM_SET_IRQCHIP
ioctl$KVM_XEN_HVM_CONFIG
ioctl$KVM_GET_CLOCK
ioctl$KVM_SET_CLOCK
ioctl$KVM_SET_USER_MEMORY_REGION
ioctl$KVM_SET_TSS_ADDR
ioctl$KVM_SET_IDENTITY_MAP_ADDR
ioctl$KVM_SET_BOOT_CPU_ID
ioctl$KVM_PPC_GET_PVINFO
ioctl$KVM_ASSIGN_PCI_DEVICE
ioctl$KVM_DEASSIGN_PCI_DEVICE
ioctl$KVM_ASSIGN_DEV_IRQ
ioctl$KVM_DEASSIGN_DEV_IRQ
ioctl$KVM_SET_GSI_ROUTING
ioctl$KVM_ASSIGN_SET_MSIX_NR
ioctl$KVM_ASSIGN_SET_MSIX_ENTRY
ioctl$KVM_IOEVENTFD
ioctl$KVM_ASSIGN_SET_INTX_MASK
ioctl$KVM_SIGNAL_MSI
ioctl$KVM_CREATE_PIT2
ioctl$KVM_GET_PIT
ioctl$KVM_SET_PIT
ioctl$KVM_GET_PIT2
ioctl$KVM_SET_PIT2
ioctl$KVM_PPC_GET_SMMU_INFO
ioctl$KVM_IRQFD
ioctl$KVM_PPC_ALLOCATE_HTAB
ioctl$KVM_CREATE_DEVICE
ioctl$KVM_REGISTER_COALESCED_MMIO
ioctl$KVM_UNREGISTER_COALESCED_MMIO
ioctl$KVM_SET_NR_MMU_PAGES
ioctl$KVM_GET_NR_MMU_PAGES
ioctl$KVM_REINJECT_CONTROL
ioctl$KVM_HYPERV_EVENTFD
ioctl$KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2
ioctl$KVM_CAP_HALT_POLL
ioctl$KVM_CAP_DIRTY_LOG_RING

With this prompt GPT replied as follows:

include <linux/kvm.h>
include <linux/kvm_para.h>
include <linux/kvm_types.h>

resource fd_kvm[fd]
resource fd_kvm_vm[fd]
resource fd_kvm_vcpu[fd]
resource fd_kvm_device[fd]

openat$kvm(fd const[AT_FDCWD], file ptr[in, string["/dev/kvm"]], flags flags[open_flags], mode const[0]) fd_kvm
openat$sgx_provision(fd const[AT_FDCWD], file ptr[in, string["/dev/sgx/provision"]], flags flags[open_flags], mode const[0]) fd_kvm

ioctl$KVM_CREATE_VM(fd fd_kvm, cmd const[KVM_CREATE_VM], arg const[0]) fd_kvm_vm
ioctl$KVM_GET_MSR_INDEX_LIST(fd fd_kvm, cmd const[KVM_GET_MSR_INDEX_LIST], arg ptr[out, kvm_msr_list])
ioctl$KVM_CHECK_EXTENSION(fd fd_kvm, cmd const[KVM_CHECK_EXTENSION], arg int32[kvm_extensions])
ioctl$KVM_GET_VCPU_MMAP_SIZE(fd fd_kvm, cmd const[KVM_GET_VCPU_MMAP_SIZE], arg const[0])
ioctl$KVM_GET_SUPPORTED_CPUID(fd fd_kvm, cmd const[KVM_GET_SUPPORTED_CPUID], arg ptr[inout, kvm_cpuid2])
ioctl$KVM_GET_EMULATED_CPUID(fd fd_kvm, cmd const[KVM_GET_EMULATED_CPUID], arg ptr[inout, kvm_cpuid2])
ioctl$KVM_X86_GET_MCE_CAP_SUPPORTED(fd fd_kvm, cmd const[KVM_X86_GET_MCE_CAP_SUPPORTED], arg ptr[out, kvm_x86_mce])
ioctl$KVM_GET_API_VERSION(fd fd_kvm, cmd const[KVM_GET_API_VERSION], arg const[0])

ioctl$KVM_CREATE_VCPU(fd fd_kvm_vm, cmd const[KVM_CREATE_VCPU], arg int32) fd_kvm_vcpu
ioctl$KVM_CHECK_EXTENSION_VM(fd fd_kvm_vm, cmd const[KVM_CHECK_EXTENSION], arg int32[kvm_extensions])
ioctl$KVM_GET_DIRTY_LOG(fd fd_kvm_vm, cmd const[KVM_GET_DIRTY_LOG], arg ptr[in, kvm_dirty_log])
ioctl$KVM_CREATE_IRQCHIP(fd fd_kvm_vm, cmd const[KVM_CREATE_IRQCHIP], arg const[0])
ioctl$KVM_IRQ_LINE(fd fd_kvm_vm, cmd const[KVM_IRQ_LINE], arg ptr[in, kvm_irq_level])
ioctl$KVM_IRQ_LINE_STATUS(fd fd_kvm_vm, cmd const[KVM_IRQ_LINE_STATUS], arg ptr[inout, kvm_irq_level])
ioctl$KVM_GET_IRQCHIP(fd fd_kvm_vm, cmd const[KVM_GET_IRQCHIP], arg ptr[inout, kvm_irqchip])
ioctl$KVM_SET_IRQCHIP(fd fd_kvm_vm, cmd const[KVM_SET_IRQCHIP], arg ptr[in, kvm_irqchip])
ioctl$KVM_XEN_HVM_CONFIG(fd fd_kvm_vm, cmd const[KVM_XEN_HVM_CONFIG], arg ptr[in, kvm_xen_hvm_config])
ioctl$KVM_GET_CLOCK(fd fd_kvm_vm, cmd const[KVM_GET_CLOCK], arg ptr[out, kvm_clock_data])
ioctl$KVM_SET_CLOCK(fd fd_kvm_vm, cmd const[KVM_SET_CLOCK], arg ptr[in, kvm_clock_data])
ioctl$KVM_SET_USER_MEMORY_REGION(fd fd_kvm_vm, cmd const[KVM_SET_USER_MEMORY_REGION], arg ptr[in, kvm_userspace_memory_region])
ioctl$KVM_SET_TSS_ADDR(fd fd_kvm_vm, cmd const[KVM_SET_TSS_ADDR], arg int64)
ioctl$KVM_SET_IDENTITY_MAP_ADDR(fd fd_kvm_vm, cmd const[KVM_SET_IDENTITY_MAP_ADDR], arg int64)
ioctl$KVM_SET_BOOT_CPU_ID(fd fd_kvm_vm, cmd const[KVM_SET_BOOT_CPU_ID], arg int32)
ioctl$KVM_PPC_GET_PVINFO(fd fd_kvm_vm, cmd const[KVM_PPC_GET_PVINFO], arg ptr[out, kvm_ppc_pvinfo])
ioctl$KVM_ASSIGN_PCI_DEVICE(fd fd_kvm_vm, cmd const[KVM_ASSIGN_PCI_DEVICE], arg ptr[in, kvm_assigned_pci_dev])
ioctl$KVM_DEASSIGN_PCI_DEVICE(fd fd_kvm_vm, cmd const[KVM_DEASSIGN_PCI_DEVICE], arg ptr[in, kvm_assigned_pci_dev])
ioctl$KVM_ASSIGN_DEV_IRQ(fd fd_kvm_vm, cmd const[KVM_ASSIGN_DEV_IRQ], arg ptr[in, kvm_assigned_irq])
ioctl$KVM_DEASSIGN_DEV_IRQ(fd fd_kvm_vm, cmd const[KVM_DEASSIGN_DEV_IRQ], arg ptr[in, kvm_assigned_irq])
ioctl$KVM_SET_GSI_ROUTING(fd fd_kvm_vm, cmd const[KVM_SET_GSI_ROUTING], arg ptr[in, kvm_irq_routing])
ioctl$KVM_ASSIGN_SET_MSIX_NR(fd fd_kvm_vm, cmd const[KVM_ASSIGN_SET_MSIX_NR], arg ptr[in, kvm_assigned_msix_nr])
ioctl$KVM_ASSIGN_SET_MSIX_ENTRY(fd fd_kvm_vm, cmd const[KVM_ASSIGN_SET_MSIX_ENTRY], arg ptr[in, kvm_assigned_msix_entry])
ioctl$KVM_IOEVENTFD(fd fd_kvm_vm, cmd const[KVM_IOEVENTFD], arg ptr[in, kvm_ioeventfd])
ioctl$KVM_ASSIGN_SET_INTX_MASK(fd fd_kvm_vm, cmd const[KVM_ASSIGN_SET_INTX_MASK], arg ptr[in, kvm_assigned_pci_dev])
ioctl$KVM_SIGNAL_MSI(fd fd_kvm_vm, cmd const[KVM_SIGNAL_MSI], arg ptr[in, kvm_msi])
ioctl$KVM_CREATE_PIT2(fd fd_kvm_vm, cmd const[KVM_CREATE_PIT2], arg ptr[in, kvm_pit_config])
ioctl$KVM_GET_PIT(fd fd_kvm_vm, cmd const[KVM_GET_PIT], arg ptr[out, kvm_pit_state])
ioctl$KVM_SET_PIT(fd fd_kvm_vm, cmd const[KVM_SET_PIT], arg ptr[in, kvm_pit_state])
ioctl$KVM_GET_PIT2(fd fd_kvm_vm, cmd const[KVM_GET_PIT2], arg ptr[out, kvm_pit_state2])
ioctl$KVM_SET_PIT2(fd fd_kvm_vm, cmd const[KVM_SET_PIT2], arg ptr[in, kvm_pit_state2])
ioctl$KVM_PPC_GET_SMMU_INFO(fd fd_kvm_vm, cmd const[KVM_PPC_GET_SMMU_INFO], arg ptr[out, kvm_ppc_smmu_info])
ioctl$KVM_IRQFD(fd fd_kvm_vm, cmd const[KVM_IRQFD], arg ptr[in, kvm_irqfd])
ioctl$KVM_PPC_ALLOCATE_HTAB(fd fd_kvm_vm, cmd const[KVM_PPC_ALLOCATE_HTAB], arg int32)
ioctl$KVM_CREATE_DEVICE(fd fd_kvm_vm, cmd const[KVM_CREATE_DEVICE], arg ptr[inout, kvm_create_device]) fd_kvm_device
ioctl$KVM_REGISTER_COALESCED_MMIO(fd fd_kvm_vm, cmd const[KVM_REGISTER_COALESCED_MMIO], arg ptr[in, kvm_coalesced_mmio_zone])
ioctl$KVM_UNREGISTER_COALESCED_MMIO(fd fd_kvm_vm, cmd const[KVM_UNREGISTER_COALESCED_MMIO], arg ptr[in, kvm_coalesced_mmio_zone])
ioctl$KVM_SET_NR_MMU_PAGES(fd fd_kvm_vm, cmd const[KVM_SET_NR_MMU_PAGES], arg int32)
ioctl$KVM_GET_NR_MMU_PAGES(fd fd_kvm_vm, cmd const[KVM_GET_NR_MMU_PAGES], arg const[0])
ioctl$KVM_REINJECT_CONTROL(fd fd_kvm_vm, cmd const[KVM_REINJECT_CONTROL], arg int32)
ioctl$KVM_HYPERV_EVENTFD(fd fd_kvm_vm, cmd const[KVM_HYPERV_EVENTFD], arg ptr[in, kvm_hyperv_eventfd])
ioctl$KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2(fd fd_kvm_vm, cmd const[KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2], arg int32)
ioctl$KVM_CAP_HALT_POLL(fd fd_kvm_vm, cmd const[KVM_CAP_HALT_POLL], arg int32)
ioctl$KVM_CAP_DIRTY_LOG_RING(fd fd_kvm_vm, cmd const[KVM_CAP_DIRTY_LOG_RING], arg int32)

kvm_extensions = KVM_CAP_IRQCHIP, KVM_CAP_HLT, KVM_CAP_MMU_SHADOW_CACHE_CONTROL, KVM_CAP_USER_MEMORY, KVM_CAP_SET_TSS_ADDR, KVM_CAP_VAPIC, KVM_CAP_EXT_CPUID, KVM_CAP_CLOCKSOURCE, KVM_CAP_NR_VCPUS, KVM_CAP_NR_MEMSLOTS, KVM_CAP_PIT, KVM_CAP_NOP_IO_DELAY, KVM_CAP_PV_MMU, KVM_CAP_MP_STATE, KVM_CAP_COALESCED_MMIO, KVM_CAP_SYNC_MMU, KVM_CAP_IOMMU, KVM_CAP_DESTROY_MEMORY_REGION_WORKS, KVM_CAP_USER_NMI, KVM_CAP_SET_GUEST_DEBUG, KVM_CAP_REINJECT_CONTROL, KVM_CAP_IRQ_ROUTING, KVM_CAP_IRQ_INJECT_STATUS, KVM_CAP_ASSIGN_DEV_IRQ, KVM_CAP_JOIN_MEMORY_REGIONS_WORKS, KVM_CAP_MCE, KVM_CAP_IRQFD, KVM_CAP_PIT2, KVM_CAP_SET_BOOT_CPU_ID, KVM_CAP_PIT_STATE2, KVM_CAP

As you can see the the definition of the structs is still missing. GPT ran out of words before finishing typing the whole file. So I asked it to write the struct definitions such that the system calls you wrote above can be compiled. And GPT replied the struct definitions. I noticed that these structs were explicity in the KVM documentation but GPT was smart enough to know how to convert kernel-defined data types to the ones defined by Syzkaller or the userland include clauses. At the end of the post I will link the full GPT-generated description

Man against machine

I pulled syzkaller commit 8321139737ed27c in 2 different folders. In one I compiled it and in the other I replaced /sys/linux/dev_kvm.txt with the one generated via Bing AI. Then I let both run on the system calls I mentioned here. I let both run for around 9 hours, on 1 VM using --debug flag of syz-manager.

In the Fig. 6 I’m reporting the results of running both fuzzers. As you cna see in the left, the manually-written system calls showed a coverage of 6536 code blocks when it ran for 9h50min while the GPT-generated one showed a coverage of 5659 when it ran for 9h17min. The runtime is not exactly the same but I saw the fuzzers started to plateau. Additionally, the manually generated descriptions showed a max signal of 16813 while the GPT-generated one showed only 14110. Either way it took me less than 30 min with GPT to generate dev_kvm.txt. Did mankind win this ? I will leave it up to your judgement.

Conclusion

To sum up, in this blog post I briefly evaluate the ability of a LLM to automatically understand a kernel subsystem’s documentation to generate Syzlang descriptions, to guide the kernel fuzzer. The results show that for a subset of syscalls of KVM the manually-written rules outperform the kernel fuzzer. This result is not all-encompassing - other kernel subsystems may show different results. Additionally, this method assumes that a well structured and detailed documnetation on how to call the API is present. This might not always be the case. Nonetheless, GPT demonstrated a significant understanding of the KVM subsystem and Syzlang and I it not only shortened the time to write a description, but also made it more accessible for a kernel developer with limited knowledge of Syzlang to write one for their own kernel module. To support open science here is the dev_kvm.txt file that that the GPT helped generate.

Preface

Before I get started with this post, I want to give a shouout to OpenAI’s dall-e for generating this whacky cover photo. I will keep this trend of AI cover photos from now on.

Now lets start by laying the basis of this post. We will be assuming the threat model of an attacker that is running on user-level privileges. Basically, the attacker needs to be able to reach a privileged state that allows them to execute as the root account. Commonly, exploit authors rely on reaching kernel-context code execution to execute their shellcode which is basically commit_creds(prepare_kernel_cred(0)) as it was explained better by trickster0’s blog post. Another common method is to overwrite modprobe_path to point to a file that we can control. When modprobe is executed our file is executed using root privileges.

Some other methods of privilege escalation are not even reliant on kernel bugs, but to misconfiguration of the Operating System. Carlos’ tool, PEASS-ng, is a suite of scripts that are designed to find misconfigured system resources (such as files, directories, sockets etc). After looking at what the scripts are searching for, I noticed that privilege escalation involves:

1. writing into root-owned and/or unwritable files or memory
2. read into root-owned and/or unreadable files or memory

With that in mind, privilege escalation can be achieved if the attacker can reach any of the 2 states above. We can utilize the linux kernel bugs to reach those states. In fact I demonstrated what kernel structures can be corrupted to reach those states in SCAVY. I highly encourage the readers to look into that paper.

Introduction

In this blog post, I will explore a new approach at exploiting CVE-2022-27666. The explotation technique involves overwriting the f_mapping pointer in struct file. As shown in SCAVY, the corruption would allow a read and write into root-owned files. I will explain the technique in the next sections after introducing what the vulnerability is and how we leverage its capability. Basically, the exploit gets write capability to /etc/passwd and insert this line: albocoder:$1$KCPMXNrz$RkFUDj69PHe.T4cGUqzv91:0:0:root:/root:/bin/bash. I will explain more about the payload in the later sections. Our exploit is tested on Ubuntu 21.10 that runs Linux kernel 5.13.0-19-generic.

Vulnerability

As Etenal has described in his excellent blog post, CVE-2022-27666 is a vulnerability in Linux esp6 crypto module where the receiving buffer of a user message is an 8-page buffer, but the sender can send a message larger than 8 pages, which creates a page-wide buffer overflow. For this blog post I will rely on the same starting capability as the prior blog post, an 8-page memory write with the constraint of being the consecutive pages from the vulnerable object. For a more detailed explanation of the vulnerability please read Etenal’s blog.

Exploitation

Our exploitation involves an 8 step procedure. In this section I will briefly summarize our whole attack then we will go into details and reasons why we chose this method in the subsections below.

First, we need to leak the f_mapping pointer from struct file (src). Then overwrite that with the mapping of the struct file of /etc/passwd. I decided to use the same kernel memory read primitive from the previous blog post to leak the file pointer. Basically, we overflow the datalen field from struct user_key_payload to read over the next pages.

struct user_key_payload {
	struct rcu_head	rcu;		/* RCU destructor */
	unsigned short	datalen;	/* length of this data */
	char		data[] __aligned(__alignof__(u64)); /* actual data */
};

In the pages next to user_key_payload we allocate a large number of struct vm_area_struct (src) that map from 2 files, a dummy file that is opened with read/write permissions and our victim /ec/passwd. We use the primitive mentioned above to leak vm_file pointer of the vm_area_struct that mapped the pages of /etc/passwd. I will get on the details how to find out which vm_file points to which of the 2 files later. Next, we use the arbitrary read via msg_msg to leak the struct file->f_mapping pointer for the struct file of /etc/passwd. And at last we leverage the arbitrary write primitive that relies on msg_msg+fuse technique to overwrite the f_mapping pointer of the struct file of the dummy file.

In summary our exploitation will be as follows:

1. Massage the heap to account for noise and get the desired layout for the next step.
2. Allocate the 8 vulnerable skb pages followed by the 8 pages of user_key_payload.
3. Allocate some pages of vm_area_struct next to user_key_payload in a way that some map a dummy file with read/write privileges and some map the /etc/passwd with read privileges.
4. Overwrite datalen from user_key_payload and read into the page filled with vm_area_struct’s of the 2 files.
5. Leak the vm_area_struct->vm_file pointer that points to struct file of the dummy file and /etc/passwd.
6. Use the msg_msg arbitrary read to read the file->f_mapping pointer of the struct file of /etc/passwd.
7. Use the msg_msg arbitrary write primitive to overwrite the f_mapping pointer of the struct file of our dummy file with the leaked value from step 5.
8. Write into /etc/passwd a line to create a new root-level account

Important machinations of page allocator

The Linux kernel page allocator is a simple type of buddy allocator with a very basic API. To request a page allocation one can call __alloc_pages(type, order, nodemask)(src). The parameters are (1) type which specifies if the page(s) will be kernel, user or DMA page(s), (2) order which specifies the power of 2 for the number of pages to be allocated and (3) nodemask which is used to specify which NUMA node to use for allocation. We won’t worry about the NUMA node, since by default the kernel is compiled with a single NUMA node. We also don’t need to worry about the type since the slab allocators all request GFP_KERNEL page types and we can’t mess with that as the attacker. The only thing we care is the order, which specifies how many 2^order pages are we allocating. In the end it returns back a page object pointer that encapsulates all the pages that were allocated. As it’s made clear the buddy allocator can only allocate a certain number of pages from the sequence: 1, 2, 4, 8, 16, .... Similarly, to free the pages one can call __free_pages(struct page*, order) (src) which tells how many pages to free from the starting page.

The allocator works in a FIFO manner, meaning that the latest pages to be freed will be returned to satisfy an allocation, but this is not very important to us. What is more important to our exploit is that when freeing an order of pages the allocator will merge consecutive lower order pages if they can make up a higher order one. For example if 2 order 0 pages (meaning 2⁰=1 page) are freed and they are consecutive they will be put into the freelist of order 1. This will come useful when we chose how to mitigate noise on order 3 pages. Additionally, to assure that 2 order 3 allocations are consecutive we must exhaust the order 3 freelist and force it to split order 4 pages in 2 order 3’s. I won’t bore you out with more details on the buddy allocator since I already linked the source code of the allocator that our host is using, but if you are interested to know more about its inner machinations check it here which is also where I ss’ed figure 1.

Additionally, I want to point out that the slab allocator compartmentalizes allocation requests. A call to kmalloc(size) will round the requested size to the closest power of 2 and find the compartment (offically named cache) that has objects of that size allocated in its pages. For example, a call to kmalloc(54) will allocate the object in the kmalloc-64 cache. However, the slab allocator also allows one to request allocations from special caches. For example, a mmap() call will cause the kernel to call kmem_cache_alloc("vm_area_struct",sizeof(struct vm_area_struct));, which will allocate a new vm_area_struct in the special cache with the same name. Similarly, a call to fork() or clone() among other allocations will also allocate a struct task_struct in the task_struct cache. To view all the caches and statistics about the number of allocated objects a root user can simply cat /proc/slabinfo. The output looks like figure 2. In the default Ubuntu 20.04 LTS there seem to be 188 caches.

The 5 integer columns in the figure above indicate the following:

1. number of active objects (slots that are allocated)
2. total number of object (slots across all pages allocated for this cache)
3. object size (in bytes)
4. objects per slab (number of slots allocatable in a slab)
5. pages per slab (number of pages to allocate whenever a new slab is needed)

A slab is a set of 2^order of pages. Lets illustrate this with an example. Below I added what my task_struct cache currently looks like. This means that a single struct is 6400 Bytes which is more than 1 page in my system (PAGE_SIZE = 4096). Therefore, 1 structure takes ~1.56 pages. Currently there is no need to allocate more pages since there are 735-639 =96 free slots in this cache’s freelist, however if all slots were taken the SLUB allocator would call __alloc_pages(GFP_KERNEL, 3, NULL); and get 8 new consecutive pages that have 5 slots to satisfy the allocations.

task_struct          639    735   6400    5    8 : tunables    0    0    0 : slabdata    147    147      0

Massaging the memory

First we need to make sure that if any cache need pages their allocations don’t mess with our memory. To do that I follow the same logic as Etenal except for the last step that frees all the objects. Therefore the noise mitigation involves 3 steps:

1. drain the freelist of order 0, 1, 2, 3.
2. allocate tons of order-2 objects(assume it’s N), by doing so, order 2 will borrow pages from order 3.
3. free every half of objects from step 2, hold the other half. This creates N/2 more object back to order-2’s freelist.

This way all objects of order 2 and below will fall in the holes created in step 3. We expect there to be few order 3+ objects so we hope that all our allocations of order 3 pages will be consecutive since after step 1 there are no more order 3 pages scattered around, so the allocator takes 8 pages from order 4. As I illustrate in the GIF below the 4 page hole will satisfy allocations of order 2 to 0. Of course N is in the range of 10-30 so there will be up to 30 holes of 4 pages to accomodate the noise.

Leaking pointer to struct file

Once we have amortized the noise we immediately start with the exploit. As mentioned earlier, I will reuse the same struct user_key_payload oob read from the previous exploit. I allocate the vulnerable order 3 skb pages, then allocate 1 user_key_payload object of size 2049 (1 Byte more than 2048) so that it gets allocated in kmalloc-4k cache, which is an order 3 slab. Therefore a new slab request from this cache will allocate order 3 pages which will be next to our vulnerable object instead of falling in the noise holes. I didn’t change Etenal’s method of having of 1 user_key_payload object per kamlloc-4k slab, because it helps a lot with mitigating problems with order 3 noise. For more info on that to have 9 trials of consecutive kmalloc-4k slabs, which is because we can allocate up to 9 key payload structures. For more information on why this limit exists check their blog post. Figure 4 is an illustration of how it would be if we put all user_key_payload in one slab making the overflow useless if the user_key_payload didn’t get allocated in the 8 pages next to it. Figure 5 shows how a more improved noise-resistant overflow looks like, both taken from Etenal’s blog post.

So far so good, now lets start with our novelty. Once we have the correct slab next to our vulnerable object we need to find an object to leak. I looked into a few candidates that contain a pointer to struct file. My initial idea was to find a structure in dirtycred paper that has a pointer to struct file and fell in kmalloc-4k or any order 3 slab. However I found that vduse_iova_domain correspond to vdev device which is not supported in the stock Ubuntu 21.10 that I’m exploiting. So next I tried to change the exploitation strategy and go for corrupting struct task_struct instead. As we show in our paper there are multiple fields in task_struct that we can corrupt, however there were 2 problems: (1) when calling fork() the kernel would allocate task_struct along with hundreds of other structures, which would leak out of the noise holes we made earlier and mess with our exploit’s memory layout, (2) we would first need to leak the memory address of task_struct since our primitive is a read and not a write. Specifically for problem 2 we needed to know where the task_struct is allocated so that when we use msg_msg’s arbitrary write we know which address to write to.

Given these 2 problems I decided on a different apporach, which I also introduced in the beginning. I start by first opening 2 files: a dummy file (lets call it /tmp/dumdum) with read and write permissions and /etc/passwd with read permission (because we can’t write into it as a non-root user). This causes kernel to allocate 1 struct file for each open(). One idea is to just open a bunch of files with the hope that a page of struct files gets allocated next to user_key_payload. However, we have the same problem here as we did with task_struct, which is that we don’t know the struct file’s location in memory to overwrite. Therefore, I decided to mmap() the opened files instead, which creates vm_area_struct structures for each successful mapping. As you can see below the structure has vm_next and vm_prev which are used in the context of a doubly-linked list. Since every allocation will link to the previous and next we can find the exact page address. Additionally having vm_start and vm_end allows us to better search for the structure since we know these values from the return value of the mmap() call.

struct vm_area_struct {
	unsigned long vm_start;
	unsigned long vm_end;
	struct vm_area_struct *vm_next, *vm_prev;
	struct rb_node vm_rb;
	unsigned long rb_subtree_gap;
	struct mm_struct *vm_mm;
	pgprot_t vm_page_prot;
	unsigned long vm_flags;

	struct {
		struct rb_node rb;
		unsigned long rb_subtree_last;
	} shared;

	struct list_head anon_vma_chain;
	struct anon_vma *anon_vma;
	const struct vm_operations_struct *vm_ops;
	unsigned long vm_pgoff;
	struct file * vm_file;
	void * vm_private_data;

#ifdef CONFIG_SWAP
	atomic_long_t swap_readahead_info;
#endif
#ifndef CONFIG_MMU
	struct vm_region *vm_region;	/* NOMMU mapping region */
#endif
#ifdef CONFIG_NUMA
	struct mempolicy *vm_policy;	/* NUMA policy for the VMA */
#endif
	struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
} __randomize_layout;

So lets summarize our exploit so far. As I illustrate in Figure 6, we massage the heap and poke holes, then allocate the vulnerable object and user_key_payload, opent the 2 files and mmap them. Then we overwrite user_key_payload->datalen field and read out of bound into the page(s) containing vm_area_struct.

As we see, we don’t want to poke too many order 2 holes or all the mmap() calls will be falling into those holes. The way we allocate vm_area_struct is my mmap‘ing the dummy file 3 out of 4 times and the passwd 1. This ratio is important because it makes it easy to find which vm_area_struct->vm_file points the struct file of /etc/passd and which one points to the file of /tmp/dumdum. So let’s talk more in detail how we are doing this. After I open the 2 files I mmap them as shown below.

void* mapped[3000];
int fd = open("/etc/passwd", O_RDONLY);
int fd_dummy = open("/tmp/dumdum", O_RDWR);
for (int i = 0; i < 3000; i++) {
	if (i % 4 ==0)
		mapped[i] = mmap(NULL, 0x1000, PROT_READ, MAP_SHARED, fd, 0);
	else
		mapped[i] = mmap(NULL, 0x1000, PROT_READ|PROT_WRITE, MAP_SHARED, fd_dummy, 0);
}

As we can see above, vm_area_struct has the __randomize_layout which makes it harder to find the offset of each field. I personally found a hacky way to do it, which a user-level attacker wouldn’t be able to do. So I created a fake kernel module (say fake.ko), compiled it and then just ran pahole -C vm_area_struct fake.ko and it would show exactly how the structure is defined in the running kernel. If you want to learn more about this hack here is a stackoverflow post. Since vm_area_struct has many null fields and 2 known-value fields it’s fairly easy for an attacker to figure out the random layout.

So now we have leaked at least 1 page full of vm_area_struct. Now we need to find which f_mapping pointer points to /tmp/dumdum and which one points to /etc/passwd. To do that we simply count all the occurrences. The one occurring more often is the address of struct file of our dummy file. Additionally, while we don’t need to we can also count all vm_next and vm_prev to find the start of the page that contains our vm structures and from there we can find the address of every structure that we leaked.

Closing in the exploit with pre-written primitives

After allocating 3000 vm_area_structs we no longer have order 2 noise holes, so first we re-massage the heap and use the same msg_msg arbitrary read technique to read file->f_mapping value from the struct file of /etc/passwd. As we can see below the file structure has __randomize_layout too. I again used the same hacky way to get the correct layout, but I believe one can leak both struct files and figure out the pattern given that there are shared values such as f_op pointer. For more on how this read primitive works check Etenal’s blog post.

struct file {
	union {
		struct llist_node	fu_llist;
		struct rcu_head 	fu_rcuhead;
	} f_u;
	struct path		f_path;
	struct inode		*f_inode;	/* cached value */
	const struct file_operations	*f_op;

	/*
	 * Protects f_ep, f_flags.
	 * Must not be taken from IRQ context.
	 */
	spinlock_t		f_lock;
	enum rw_hint		f_write_hint;
	atomic_long_t		f_count;
	unsigned int 		f_flags;
	fmode_t			f_mode;
	struct mutex		f_pos_lock;
	loff_t			f_pos;
	struct fown_struct	f_owner;
	const struct cred	*f_cred;
	struct file_ra_state	f_ra;

	u64			f_version;
#ifdef CONFIG_SECURITY
	void			*f_security;
#endif
	/* needed for tty driver, and maybe others */
	void			*private_data;

#ifdef CONFIG_EPOLL
	/* Used by fs/eventpoll.c to link all the hooks to this file */
	struct hlist_head	*f_ep;
#endif /* #ifdef CONFIG_EPOLL */
	struct address_space	*f_mapping;
	errseq_t		f_wb_err;
	errseq_t		f_sb_err; /* for syncfs */
} __randomize_layout
  __attribute__((aligned(4)));	/* lest something weird decides that 2 is OK */

Then, the prior exploit uses a msg_msg arbitrary write primitive to overwrite modprobe_path, a common target for attackers. I modified their arb_write() code to overwrite the file->f_mapping of the struct file of the dummy file. After the overwrite succeded the read(fd_dummy,buf,6); returned root:x, which is the content in /etc/passwd. SUCCESS!

All that was left was to write into the file and close it. So I decided to create a root account with the name albocoder and password erin. Of course Linux doesn’t just allow for passwords to be saved in plaintext so I hashed it with the command openssl passwd -1 erin. This generates a hash that Linux will get to when we supply erin as the password in su. So first I fseek to the beginning of the file and then call write() for albocoder:$1$KCPMXNrz$RkFUDj69PHe.T4cGUqzv91:0:0:root:/root:/bin/bash. Last, we call exit(0) and boom, Bob’s your uncle…

Now we simply do su albocoder insert erin as the password and the id command returns the uid 0. Since I tried to keep most of the code the same as the previous exploit code I will only publish in the github repository the files that have changed here.

Video demo

vm_area_struct exploit

Read our paper, that was accepted at USENIX 2024.

Ack

Thanks to Etenal who helped me understand his exploit code and troubleshoot the exploit crashing when trying to corrupt task_struct by calling fork() 100 times.

This post is a gentle overview of our paper. I will present some other interesting results we didn’t include in the paper that might interest the broader audience and the industry.

Introduction

Dynamic malware analysis is the go-to tool for clustering malware into families[1][2][3][4] or detecting malware in cases where static analysis would fail[5][6], such as polymorphic malware. Prior work have proposed techniques to detect malware based off the execution traces[7][8][9][10][11]. However, even with these impressive detection rates, malware outbreaks still happen and pack a serious punch. So there must be some discrepancy between what analyst see in the labs and how malware behave in the wild. In our paper we studied the behavior of malware in the wild, for a dataset collected across 5.4M machines across the globe. In this blog post I will showcase some more results including some case studies. In our work, we answer the following questions:

1. How much variability is there in malware behavior in the wild across machines and time?
2. What parts of its behavior varies more?
3. How does this variability affect existing malware clustering and detection methods?
4. Can we still find invariants in a malware’s behavior? What parts of the behavior profile are more likely to yield invariants?

For the sake of brevity, in this report I will briefly report our results on questions 1, 3 and 4, while providing some motivating case studies.

Dataset

We are able to answer all the aforementioned questions through a dataset of program executions, we collected from 5.4M machines across 113 countries. In Figure 1, we show a distribution of the program executions in our dataset. Evidently, the majority comes from USA, China, Russia and the European countries.

As we see from the table above the most popular windows OS version in 2018 was Win 7 build 7601. This may no longer be the case in 2021 and the threat landscape may not be the same, but the malware behavior must be more evasive. Therefore the behavior variability in the wild should only be higher.

Variability

We measure the variability in terms of missing and additional actions in malware, PUP and benign execution traces across machines and time. In this blog post I will only focus on cross-machine variability.
For more details on the exact definition of variability and other results I would encourage you to check section 2 in the paper.

In the boxplots in Figure 2, we plot the number of missing/addional actions across machines for all malware, PUP and benign samples. As you can see, at least 50% of the malware samples in our dataset have around 59 missing/additional events in the trace where 33 are file creations (Figure 2b).

To better zoom in on this variability we built a Splunk dashboard with information about difference between 2 traces. Below I will present some of the variability we found in the wild and the cause of it.

Case study: The Ramnit worm (VT link)

An interesting case was that of Ramnit worm. The analysts from cert.pl blog have confirmed that if the Ramnit worm is executed with non-admin privilleges it will try to privillege escalation. In the vulnerable version of Windows 7 it will exploit CVE-2013-3660. From exploit-db we found a confirmed working exploit code, which seems to create mutexes. The large amount of Mutex creations may happen because the malware keeps running the exploit until it succeeds.

Going back to Figure 3, the green bar plot in the middle is the absolute difference in the number of actions for each event type. As we can see from the blue graph on the left, trace 1 has around 50 mutex creation actions while the plot on the right doesn’t have any. This result shows what the analysts of cert.pl blog have manually seen.

This environment-sensitive behavior, however happens very often[12] in malware thus it’s to be expected that the behavior variablity in the wild is the largest among the 3 categories (malware, PUP and benign).

Case study: The Darkcomet RAT (VT link)

Another case of variability across machines or time is the Darkcomet RAT. In particular case, variabilty is hard to be correctly attributed to a root cause, since the RAT may change its behavior in a specific machine (as instructed by the attacker) or through time if the attacker start an attack campaign (ie. DDoS). Nevertheless, we notice that in trace 2 the malware creates Registry keys while in trace 1 it doesn’t, meanwhile in trace 2 it creates a lot more files. This could be due to new commands issued by the C2.

Invariant

With all this behavior variability can the analyst still detect the malware? We turn our attention to host-based IDS systems (ie. Splunk, Sumo logic, Qradar or other SIEM tools). In this section, we measure the effectiveness of most common

Methodology

To answer this question, we first scrapped all the rules from Sigma to analyze their composition. In Figure 5, we show 2 SIGMA rules, in which we split the parameters’ values into tokens via windows delimiters. The tokens are shown in red. To highlight the importance of this tokenization we perform it for all the SIGMA rules and we consider a full token to exist if the rule shows 3 tokens with no regular expresssion character in between, this way we eleminate cases where only a part of the token is present, such as the green box. We noticed that around 70% of all the existing open source rules have at least 1 full token in the middle, as shown in black. Considering our approach, this is a lower bound estimate on the true value of full token matching rules.

I’m not highlighting all the tokens with all their resepective colors, but you get the point.

This is important because now we know that if 1 of these tokens is doesn’t match the entire rule will not. So we go back to our dataset and split all the parameters of all our execution actions into windows delimited tokens. Our goal is to analyze the prevalence of these tokens in the wild.

Basically we are not interested in common tokens (ie. exe, windows32, setup, cmd), but in malware-specific tokens (ie. wnry) since the latter will be used in a SIEM signature. Therefore, the first step from our analysis is to remove all tokens that appear in any of the benign samples and all tokens that appear only in 1 machine (randomness). We assume the analyst will use their knowledge or a data-driven method to remove randomness. The leftover from the culling are considered to be malware-specific tokens. We consider the set of all the tokens left as the invariant, since an analyst can create IDS rules that check the presence of at least 1 of the tokens, in the action’s parameter.

The invariant is a bag/set of tokens.

Gotta catch em’ all

The first measurement we conduct is to evaluate the number of minimum number of machines an analyst would need to capture all malware-specific tokens. The question we answer is: How many machines are needed to capture all malware-specific tokens for most common parameters?. We consider common, the parameters most used by the SIGMA rules. Of course, this assumes that the malware author will execute the malware at the perfect time and system to obtain the largest amount of tokens. This estimate is of course a lower bound, but it serves to illustrate the difficulty in obtaining the tokens needed for SIEM signatures.w

As we see from Figure 6, for different parameters it requires different amount of machines to get all the values. The hardest to capture seems to be the file name where only around 85% of the malware yield all their malware-specific tokens within 10 machines. In SIGMA, the dropped file’s name makes up for ~12% of the rules.

On the other hand, the command line is used in ~40% of the SIGMA rules and rightfully so. Figure 6 confirms that it takes less machines to get the malware-specific tokens.

How successful in detection are the tokens?

While we show how difficult to catch are some tokes from some of the parameters, we also want to know which tokens are the best to be used in detection. For this we measure the ratio of machines in the wild each malware-specific tokens appears. We name the ratio of machines the token appears in the wild as coverage.

In Figure 7, we measure the coverage of all non-benign tokens an analyst will find in a malware execution trace. The graph can be interpreted as the success rate (coverage) an analyst would get if they picked 1 single random token to make their signature of. Evidently, file path tokens (subdirectories) and command line tokens (CMD parameters) seem to yield better coverage.

But an analyst will not use 1 token and go their merry way. They will pick a all tokens (since by definition they don’t appear in benign samples). In the following sections we will determine how many machines an analyst needs to get the largest coverage. In the paper, we measure how often the analyst needs to reexecute the malware to keep the signatures up-to-date.

Optimal number of machines

In this section we measure the success of detection in the wild, for a set of tokens extracted by N machines. We assume that the malware would behave in the analysts’ sandboxes as if they are running in the wild. The question we answer is: “In how many machines should the analyst execute the malware sample to get the largest coverage of the token set?” We define as coverage, the amount of machines in the wild where 1 of the tokens in the set appears. In a more formal definition, a machine is considered “covered” if the intersection of the “signature” set with the set of tokens in that machine is not empty.

But how can an analyst generate N VMs to resemble N random machines in the wild? - We discuss that an analyst can use a random VM generator like SecGen [13] with the features proposed in Spottless Sandboxes paper [14]. Of course vendors have an idea on the distribution of those features in the wild, so for analysts in such companies generating VMS that resemble the true population of users’ machines in the wild is not a big challenge.

From Figure 8, we notice that an analyst needs to execute the malware sample in 3 machines to get the best coverage using file name tokens. Adding more machines only gives diminishing returns. For more results please refer to our paper.

Effects of behavior variability in malware clustering

As I mentioned earlier, clustering is a very popular method to deal with polymorphic malware samples. Analysts use it to determine if a newly seen executable belongs to a malware family based off the behavior it shows in the sandbox. However, they usually only use 1 execution per malware sample to determine the cluster (ie. malware family) the sample belongs to.

In this section we will analyze how effective clustering is in the wild. The closest clustering paper that uses features similar to ours is that by Bailey et al. [15]. Our goal is not to find the malware families, but to argue the validity of clustering results. The core idea is that executions from the same sample/hash should fall in the same cluster. For that we pick 4 random executions per malware sample and perform the clustering.

Since we pick 4 executions per sample we count the number of clusters those 4 executions fall in. In the table above we show the malware samples for each the 4 executions fall in 1,2,3 and 4 different clusters. We noticed that 33% of the malware samples have executions in 2 different clusters, therefore if we were to interpret each cluster as a malware family it’s not clear which family they belong to. Surprisingly, 1% of the malware have executions in 4 different clusters.

Conclusion

It has been known, for over a decade, that malware samples can change their behavior on different hosts and at different points in time, but no study has yet measured this variability in the real world. In this paper, we report the first analysis of malware, PUP and benign-sample behavior in the wild, using execution traces collected from 5.4M real hosts from around the world. We show that malware exhibits more variability than benign samples. The causes may be different based on the malware type, vulnerabilities in the victims’ machines etc.

We then assess the prevalence of invariant parameter tokens that are commonly used to derive behavior based signatures for malware. Our results suggest that analysts should re-execute the malware samples 3 weeks after first receiving them to update their behavior models.

At last we show that an analyst should be cautious of the malware bahavior variability when clustering them.

Our findings have important implications for malware analysts and sandbox operators, and they emphasize the unique insights that we can gain by monitoring malware behavior at scale, on real hosts.

Any comments of feedback? Email me or comment using Disqus below 😉 (must disable ublock origin to see it)

Introduction

In this blog post I will explore the commercial malware sandboxes. It appears malware is allowed to access the internet in many sandboxes . So with that access I decided to collect all the environmental features I could think of and send them to my discord channel. Since Python is the language with the largest library support ever (maybe?) I wrote a python script to take many environment features through psutil, platform, cpuinfo etc. The whole python code is here.

Then we use pyinstaller to create a single executable (~10 MB ) using a command like this: pyinstaller --onefile --noupx sandbox-env-stealer.py. Then for the sake of completeness we compressed the executable in the latest upx and uploaded to VirusTotal. After a few crashes we ended with 4 samples:

• 47ed17bdea1dab10fdee…
• 07a783fc3ae6a065dc0b…
• e472c0493a9a35b7975c…
• 88d38301327da310c5c0… (disregard the file name )

Of course, you can download the data I collected and play for yourself.

Experiment setup

I submitted one of the first samples on 17/03/2020 (because corona lockdown was getting boring). Then life happened. After deciding to revisit this project I submitted the other 4 samples around March 2021. I collected the following features (some ommited for brevity):

• Windows version + arch
• CPU name + core count
• local and internet IP address
• CPU counters
• disk partitions and counters

What did learn?

OS version

First, I noticed that 78% of all the sandboxes run Windows 7 build 7601 (the most famous pirated version ). That accounts for 40 out of the 51 executions. The table below illustrates it all. As it seems, the only versions are Windows 7 (build 7600 and 7601) and some flavor of Windows 10. A malware sample will have a high chance of being run on a real machine if the detected OS is anything but the following.

Timeline analysis

In this section I will analyze some runtime features of the sandboxes. Here there are some interesting features malware authors can use to quickly identify sandboxes. There are also some lessons I learned from looking at the executions.

First off, I wanted to see how many executions we were getting per malware and how often. This is particularly important since in our paper we showed that a malware need to run in 3 random environments at least every 3 weeks.

It appears, for the oldest malware sample 10 executions appear on the same day as the submission, 1 execution 1 day later, 3 executions 33 days later and 2 executions 62 days later. This means that the sample was deemed interesting 2 months from its first “appearance”. However, we can’t conclude that this is what happens for all the samples, with different number of VT engine detections or with more interest from individual AV vendors. For the record, I did get an execution report back from the first sample just yesterday, after clicking the “reanalyze” on VT . Either way, the malware was executed about 2 to 3 times every month, which is close enough to 3 weeks (that we recommend in our paper), but we demonstrated in the paper that on average 1 week stale of data decreases the detection rate.

Environment analysis

This sections will show some environment features that the malware can read.

The running processes

A common routine seen on many malware and benign samples is that or iterating the running processes. I (as in python libraries) use a similar routine to retrieve the running processes. The table below shows some of the running processes and the number of machines they were seen to run on. Something interesting we can see here are the “special” programs. In some machines we see the appearance of bitcoin-qt.exe, infinium.exe etc, while in some others we see steam.exe, SteamService.exe etc, in some others filezilla.exe or centralcreditcard.exe. This is usually done to see if the malware is a crypto miner, a game hack, a file infector or a point-of-sale malware respectively.

Unfortunately, we can also see things like cuckoo-47ed17bdea1dab10fdee7f61dff8b8f33ad5d092b3e1e5f0f5a3522a27798183.exe or 1a7446534577bab0984f5eb275bdf1f43ed92dfc.exe which is simply the checksum hash of the sample. An attacker can simply compute the popular checksums (the ones on the details section in VT) and see if its name is any of those and terminate .

Machine names

One thing I wanted to see is the username the malware in the sandbox will run on. For the most part I was underwhelmed. It appears the machine name remains the same across executions, which may be easy for an attacker to just submit bogus “malware” just to harvest all the machine names. In terms of the context I did see that sometimes the sample was executed as Administrator meaning that some sandboxes give the “malware” admin privilleges. This is done to make sure that malware can execute.

Hardware analysis

CPU and memory analysis

Next we are interested to know what CPU are the sandboxes running on? I realized the platform.processor() was not returning the actual CPU name, but it was too late when I realized so I used the available information from this command and scrapped some tables online to get the processor name (here is the pkl file that aggregates all data). In the latest version I added this awesome library called cpuinfo, so in case you need to collect your own data the cpuinfo will do the work.

As hypothesized, the machine has the wrong number of cores for the CPU name (VICTORY ). In the 9 machines, I found that the most used CPUs are Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz, Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz and Intel(R) Xeon(R) W-2140B CPU @ 3.20GHz. The machines seem to have 1 core for the most times but the name is that of a cpu with 4 or 8 cores.

I also checked the CPU utilization rates. Hypothetically, the sandboxes would have low CPU utilization, since they are only meant to run the malware.

While most of the sandboxes have a utilization less than 20% there are still cases where sandboxes have 80% or even 100%. I believe the CPU utilization cannot be a feature to distingush between sandboxes and real machines.

Lastly for this section we look into memory. How much memory can commercial sandboxes spare?

It appears most of the machines have 1 to 2 GB of RAM to spare. In 2 cases I found machines with 512MB of memory like its 2010 . Props to the AV vendor(s) that allocated 4, 8 and 16GB for a sandbox to run a malware sample.

Disk Partitions

An interesting result here is that in ALL sandboxes there seems to be 1 disk partion of type cdrom . Hypothetically this may happen because during the VM creation the Windows installation disk was left inserted (more work needs to be done to verify this).

For the main disk partition (where Windows is installed) we notice quite a spectrum of sizes, but the usage in general is quite low (except for the 32GB disks with high usage because Windows is installed there). In 2021 I wouldn’t expect there to be many machines left with 32GB in the main drive, so this may raise some suspicions from the attackers prespective.

Network Interfaces

In general 2 to 3 interfaces and 5 network interfaces in 1 sandbox. The most prevalent are of course Loopback Pseudo-Interface 1 and Local Area Connection. Then I noticed some pattern of isatap.{<random GUID here>} and Teredo Tunneling Pseudo-Interface. Upon further analysis I found that these interface exist to enable IPv6 communication, so there is nothing special that an attacker can use here .

Battery

Not a single sandbox had a battery. Attackers right now pulling a high IQ strat .

Time analysis

One thing its hard to keep up to date while restoring the snapshot (at least in Windows) is local time. Its a bit more difficult when you consider the geolocation the machine is supposed to be at. For this I collected the local time, the global UTC time and the external IP address of the sandbox and metadata for that IP. Thanks to ipfy, just-the-time, and ipinfo for the awesome service. The process I followed is pretty simple. I use the data from ipinfo to get the geolocation and convert the UTC time to the local time for that geolocation, then I calculated the time skew between the time of the geolocation and the sandboxes local time.

As it appears, around 65% of the sandboxes have some sort of a time skew (and I believe the other 35% are simply lucky to VPN on the same timezone or not even VPN at all, will get to this later ).

This is a very low effort feature a real malware can use to check if its inside a sandbox given that such info can be checked for free. Even through a CnC server the attacker can measure whether the malware has made it into the sandboxes. Throughout our paper we also noticed that there are some real users’ machines with an outdated clock (sometimes out of date by about 50 years ) however this made up for less than 0.001% of the machines in the real world not 65%, so a malware author may simply choose to forgive these outdated machines just to be safe from analysis. And from a defender’s prespective, PLEASE UPDATE THE CLOCK BEFORE ROUTING THE NET TRAFFIC TO NARNIA.

IP analysis

This is the meat of the blog post, in my opinion. Thanks again to ipinfo for the 7 day free trial. As the most interesting to me, I first looked at “Who owns these IP addresses? Who are the companies/ISPs?”. As shown in the table below the network traffic in all the sandboxes is routed through VPNs that they get from dedicated servers. It appears CrowdStrike has their own IP range that they use to route traffic (registered under their official name , hide the pain Harold).

Now we want to know where exatly are these IPs from. With this much data it’s hard to say which IPs are owned by the AV vendors, since they are also just buying VPN access, but I wanted to see where they are buying from. It appears Russia, US, Canada and Germany are the most prominent countries, and in Russia, Moscow seems to be the city with the highest number of IPs.

Conclusion

There is no silver bullet to detect sandboxes, but there are some features and bugs the attacker can use to detect them. On the other hand the sandboxes can also cover these weaknesses. It’s all about that arms race .