TheShadowEevee's Blog

Gridania Roundup - The Advent of `.`

Sakura Yoshida — Tue, 03 Feb 2026 00:00:00 GMT

Gridania Roundup
This post is part of the Gridania Roundup series, a mock Newspaper based out of the MMORPG Final Fantasy XIV. This is a creative writing series, and all information within should not be taken as fact.
This post was unfortunatly delayed for some time. The initial draft was written on December 17^th, 2025.

By Sakura Yoshida, Staff Reporter of the Gridania Roundup
Published 17th Sun, 6th Umbral Moon

A crowd has gathered in the Coerthas Western Highlands tonight, just beyond the Falcon’s Nest, to witness a spectacular creature, previously unknown to Eorzean wildlife biologists. The mysterious creature bears no name, though it takes an apperance similar to the common Steinbock or Antelope Doe. Many have taken to naming the creature, with some names including “Doe”, “Charles”, or “That Thing”. However, concensus has generally landed on the name ..

Staff Reporter Sakura Yoshida watching . in it's natural habitat.

I spent multiple hours on the scene with one of the . in the Western Highlands, chatting with adventurers and tourists. The crowd was often quite large, at one point exceeding 40 individuals. At least one Fan Club was started at the scene, and many tourists were riding Antelope Doe steeds to commemorate the occasion.

. was first sighted sometime on the 15th Sun of the 6th Umbral Moon by an unknown adventurer, and word of the strange creature quickly spread. The Temple Knights have determined . to be a relatively harmless creature and one of the safest they’ve encountered in the Western Highlands to date, leaving curious individuals little reason to not venture to the Highlands themselves in search of ..

The phenomenon has caused a short rise in the tourism to the Ishgard despite the slow travel season. Locals from the Firmament were unprepared for the increase in foot traffic during the Starlight season,

We interviewed Ashi Kar’toh at the scene for comment. [The following interview is edited for berevity.]

Sakura Yoshida:
Hello Hello, I’m Sakura Yoshida with the Gridania Roundup. Can I interview you about the recent appearance of .?

Ashi Kar’toh:
What’s there to ask?

Sakura Yoshida:
For starters, how do you feel about .?

Ashi Kar’toh:
Whereas a deity such as Skogs Fru would empower one to feel awestruck, . instead warrants a feeling of calm and serenity

Sakura Yoshida:
Mhm, mhm… And where do you think . came from? Leading researchers haven’t found a conclusive answer to this question?

Ashi Kar’toh:
It’s her presence, in the same way that one may feel empowered when hearing a speech by a smart and noble man. Though it may even go further than her contageous attitude and peaceful way of life - perhaps her aether itself is strong enough to be sensed by others.

Sakura Yoshida:
Interesting take, I can’t help but see your logic. If I may ask on last question, is there anything you would like the readers to know about .?

Ashi Kar’toh:
Praise Skogs Fru.

Sakura Yoshida:
I… I see. Well, thank you for your time today, Ashi.

The Appeals of Homelabbing

Lumi — Thu, 01 May 2025 00:00:00 GMT

Of my five blog posts prior to this, I consider three of them “real” posts. These are “Renaming an Active Node in a Proxmox Cluster”, “Notes for Self-Hosting an ATProto PDS”, and “Welcome to the ATmosphere!”. In each of these prior posts, I made some allusion to my personal Homelab, a set of hardware spread between two (or three, if you count my Oracle Cloud server) locations running nearly everything that is accessible on shad.moe and konpeki.solutions. I think it’s about time I actually write about this homelab, and more-so why I have a homelab in the first place.

Definition: Homelab#
”…Homelab is the name given to a server (or multiple server setup) that resides locally in your home and where you host several applications and virtualized systems for testing and developing or for home and functional usage.”
“This server can be a simple tower or small PC or a Raspberry Pi like device or a repurposed professional server that you can acquire from companies who discard them due to their age but are still usable.” - Helder - Linux Handbook

Actually, I disagree with this definition slightly. Where Helder specifies that the server “resides locally in your home”, I think that can be loosened a bit. This is indeed generally true, but there can be exceptions - your definition of a homelab can be different! At its core, a homelab is a machine or set of machines in your control on which you can run basically whatever you want! Some people make lists of their “recommended” services to host, but I won’t be doing that. What I host can vary quite a bit from what you may want, so instead I’ll direct to this directory of selfhostable apps that you can explore at your leisure.

Instead, my aim in writing this is to talk about my personal homelab, and why you should consider creating your own!

Supplemental Information Notice
Much of this post consists of information about my own homelab setup. For this reason, a lot of what I talk about may contain links to other resources about this software / hardware. If you are interested in learning about a piece of my homelab, feel free to follow the links! A lot of the discussion about my setup will be information dense, and the links will help to better explain a given topic for your own use.

My Hardware#

I’ve talked in some length about my own Homelabbing equipment in my previous posts. At the time of publishing, my setup currently consists of two Dell PowerEdge R640 servers and one Dell Optiplex 5040. I also have two Oracle Cloud free tier VPS servers. The networking side of things is much more varied.

Currently, the two R640s are in a dorm room on the Purdue University campus, which are soon to be moved as the semester winds up. For those two servers, an HP ProCurve 2910al-48G switch is used for local communication (as well as other clients in my dorm room), which is routed through an old Netgear WNDR3400 router running dd-wrt, which is connected to Purdue’s network. At home, an hour and a half away, my Optiplex sits in a laundry room with a direct ethernet connection to the router there.

Communication between these servers is handled via Tailscale, on their free tier. I am a fan of Tailscale, and have explored alternatives (Headscale, Netbird, Self-Managed Wireguard) but kept returning to Tailscale in the end. All World-Accessible endpoints (mostly everything hosted on shad.moe, konpeki.solutions, or their subdomains) are routed to the two Oracle Cloud VPSs, which have the primary purpose of proxying traffic to the internal servers.

So… Why bother homelabbing?#

Personally? I find homelabbing fun. Some people lab for learning, others to reduce their reliance on corporate oversight, and many others lab for reasons I don’t know. My homelab is what I jokingly call a “High Downtime Environment”. I break things, often, and the perceived usefulness or interest any given puzzle piece provides determines if it gets fixed or stays broken. This is a good thing! This freedom allows me to do a lot more than I could in a constrained environment.

I also enjoy playing with new software. Having a Homelab provides a perfect environment to tinker and put things into production! The most recent example I have is Anubis. I do care a bit about resiliency, and security, so Anubis immediately struck my interest.

What is Anubis?#
“Anubis weighs the soul of your connection using a sha256 proof-of-work challenge in order to protect upstream resources from scraper bots.”
“This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.” - Anubis Documentation - April 2nd, 2025

I heard about Anubis directly from the source: a Bluesky post made in late March by the original developer, Xe Iaso. If you haven’t heard of them before, I highly recommend checking out their site and blog! Anubis was a tool created to solve a specific problem: AI Scrapers bullying websites with overwhelming amounts of requests in their thirst for more data. I won’t be able to do right by the first of Iaso’s blog posts on this, so I highly encourage checking their post out yourself.

Anubis is the latest example of software I thought was interesting and/or could serve a useful purpose to me. Soon after I learned about Anubis, I overhauled my outward-facing networking (previously a number of muddled proxies between the world and my servers), and simplified the flow while implementing Anubis at the same time. Anubis doesn’t work on some of my subdomains - it’s not supposed to work on everything! - but for those it does, it now protects. This includes my Forgejo/Git instance and my SearXNG instance for example.

However, it’s important to know that homelabbing can be time consuming if you let it, and you will oft find something broken for you to fix.

There’s Always Trouble in Paradise#

Even now, as I begin the process of writing this blog post, I had to stop writing to fix a bug I noticed. I was on Bluesky, and I attempted to follow the account @techaro.lol, the company behind Anubis. I noticed however that while the BlueSky AppView reported for a moment that I now follow the account, BlueSky quickly forgot this fact and re-displayed the blue “follow button”. After some investigation, I discovered that this was because of a change I had made on a whim, which had consequences I hadn’t anticipated. Let me walk you through the journey I took to get this fixed - I documented each step as I took it to include in this post.

First, I tried to access my Nginx logs to check the network traffic to my self-hosted data server. I grabbed my phone to open the Proxmox mobile app, but I soon remembered I had been working on moving Nginx to a NixOS-based machine instead, so I would need to access that machine rather than one of my Proxmox containers. Unfortunately, the SSH key to connect is on a hardware token my phone wasn’t immediately able to use. So, I grabbed my laptop - and found it dead. After plugging it in and turning it on, I attempted to SSH into the Nginx container to discover that I still couldn’t access it; I had changed the SSH port earlier and forgot to allow the new port in the firewall. Luckily, one quick configuration change later, and I was in! And yet, after all of that, the access logs looked fine. Any calls to the PDS were returning code 200 (HTTP code for “OK”) and no errors were being posted.

What gives? Well, further research online and in the “AT Protocol PDS Admins” Discord server turned up similar occurrences for instances with WebSockets incorrectly configured. This couldn’t be my issue though - I had just added proxyWebsockets = true; to all my Nginx configurations some hours ago! It couldn’t be possible that WebSockets weren’t being proxied. …Right?

Wrong, and it’s because I made a mistake. The configuration file that defines the url for the PDS instance, $/machines/servers/barrier-nginx/nginx/solutions/konpeki/default.nix, had apparently been overlooked and never had this configuration set! Of course, adding the single missing line immediately resolved my issues. Problem solved! Now I can follow Techaro, and re-like the few post I had attempted to like in the downtime.

A Place to Try New Things, and Break Old Ones#

I host a fair amount of services on my personal hardware. For example, even this blog is running in two parts, an Astro frontend and my ATProto Personal Data Server. Both of these are hosted on different LXC containers, further separated from the Nginx server which is hosted on a free Oracle Cloud machine (one of the 1GB ram, 1 oCPU machines). While this may seem intimidating, I manage my efforts to prevent it from being a burden. Keeping my homelab together can be a chore at times, but it should never become like a job!

Allowing myself these freedoms opens up doors to the ability to make changes that you otherwise couldn’t in a more “production” or “corporate” setting. Most companies obviously wouldn’t allow an employee to haphazardly prop up software or reconfigure hardware that interests them with the ease you can in a homelab. In the same vein, I would be fired on the spot if I started completely reinstalling an entire server’s operating systems just because I “wanted to try NixOS”.

In fact, some people do treat their Homelabs as if they were in a company setting! I personally do a mix of both - many of my services run on the shad.moe domain, often more personal services like my blog or my RSS Aggregator. However, I also make use of the name Konpeki Solutions for anything I target to be more like a company. This includes the domains konpeki.solutions and konsol.internal, which host my BlueSky PDS.

What is Konpeki Solutions?
Konpeki Solutions is a strange construct. It is an example company name, a legal name (currently as a DBA, with plans to incorporate using the name if/when I need to form a Corporation for something), and the setting of a fictional corporation. This fictional corporation and it’s employees (the universes’ characters) will someday appear in some form brought to life by me and my sibling @axoblu. I will write more about Konpeki Solutions / KonSol in a future post.

This type of organization of your homelab can help guide you in certain directions, and is a good way to focus your efforts! Sometimes having self-imposed constraints can make things more interesting and easier to work on then having no idea what to do next.

On top of this, this environment allows me to make mistakes and learn at a pace that far surpasses what i could achieve without. Simply having a two-node Proxmox cluster has given me knowledge that many people wouldn’t have prior to working in a more corporate Proxmox environment - and even in that case, I have a much deeper knowledge of some of aspects than the average tech would. (See my Proxmox Cluster Post for an example. I’d bet most people haven’t had to hand repair a Proxmox Cluster!)

How does one start a homelab?#

You can make your homelab as simple or as complicated as you want! There are plenty of reading resources on creating a homelab, and I’ve included some of my personal favorites.

r/homelab on Reddit - A Reddit Community of other Homelabbers. Typically more hardware focused.
r/selfhosted on Reddit - A Reddit Community of people self-hosting services for their own uses. Typically more software focused.
Self-Host Weekly Newsletter - A Weekly Newsletter that shares updates in the self-hosting community, including new software, updates to software, and other general news.
Self-Host Apps Directory - A directory of self-hostable apps, managed by the same person behind the Self-Host Weekly Newsletter.

As well as these resources, I’ve drafted a few different categories of ways you can start a homelab below. Feel free to read them all, or skip to the category that interests you most!

Disclaimer
The following suggestions are a mix of personal experience and information I’ve heard from friends and online. Always know what you’re buying, what the policies on return are, etc. TL;DR - Treat your money with the care it deserves!

I don’t want to purchase anything#

There are many ways to begin a homelab without spending a cent! Whether you want to get a taste of the experience or you want to avoid ever spending money, you have plenty of options. Some good places to start are with software that runs in Docker containers, as you can easily run a handful of these in the background on an existing desktop machine. If you have old hardware laying around, such as an old laptop, this can also make for a great start to a Homelab! Installing a Linux distribution (preferably without a graphical environment if it will be a server) is a great way to breathe some extra life into old tech you have laying around, and often times what you thought was e-waste can be a decent host.

I want to own my hardware#

Personally, each of my physical servers were obtained thanks to the Purdue University Surplus Store. I’ve spent 350 US Dollars there, which seems like a lot at first glance, until you compare to the hardware. Each of my two R640s were obtained for 100 US Dollars each, while the Optiplex and the HP ProCurve switch were about 50 US Dollars each. These are really good deals on older server hardware, and everything worked great! I’ve since upgraded the RAM in the Optiplex and did have to purchase storage for each machine, as they don’t come with their storage installed for data security reasons. If you don’t have a similar store nearby, there are other places you can look for deals. Government and other Surplus sites online may offer hardware if you can arrange shipping, and sites like eBay can have some good deals as well.

Of course, you don’t need enterprise grade hardware either! In fact, for a lot of people this is an anomaly. Like if you were to purchase nothing, old laptops and desktops can be great servers. You can often get these for cheap (know the worth of what you’re buying though!), and in some areas even free! Ask some friends if they have old tech they don’t want anymore, hit up your local e-waste collections facility (some are amiable to letting you have some of their collected hardware, but most aren’t. They have to make money somehow!), or even check with your workplace, school, etc. to see if you can requisition some old hardware destined for recycling or destruction. As with before, in almost every case you’ll have to provide your own storage, and these methods are more likely to net you broken tech. For some people, fixing their hardware is part of the fun! For others, inspect anything before you buy or otherwise requisition it if possible.

I want to rent a server#

Renting a server is another way to get a machine to self-host a homelab. I can’t recommend any service providers myself, but a number of Cloud Service Providers (CSPs) offer Virtual Private Servers you can rent on a timed basis. You’ll often expect to pay for CPU hours, Network Ingress/Egress, etc., but CSPs have the advantage of uptime guarantees, fast networks, and no physical maintenance requirements. If you’ve heard of Amazon Web Services, Google Cloud, or Microsoft Azure, you have an idea of what a CSP is. Many CSPs have free trials and free tiers, but they often end up with unexpected charges. Keep this in mind if you go this route!

Oracle Cloud Free Tier
Oracle Cloud has a generous Free Tier for customers to try out, but I cannot recommend it despite the fact I use it myself. Oracle has a long history of shutting down free tier customer’s machines^[1], deleting entire accounts^[2], and preventing sign ups in the first place if they happen to decide they don’t like your email, location, credit card, etc.^[3] Unless you’re willing to deal with these headaches, avoid Oracle Cloud.

Conclusion#

To some people, a homelab is a daunting idea. They may have heard stories about the amount of time, or effort, or cost some people put into their own homelabs, or maybe they just don’t know where to start. My goal in writing this post is of two parts - first, to write about an interest of mine. Secondly, however, I also hope to convince you - the reader - that a homelab isn’t this overwhelming concept. As with many things, a homelab is what you make out of it. You can put in as much or as little effort as you want. Maybe you want to host one or two services for yourself, or maybe you want to create an entire miniature network. Maybe you want to learn about Microsoft Active Directory, or maybe you want to participate in a community like DN42, participating in a peer-to-peer BGP routed network!

All this is to say, don’t concern yourself with making a “perfect” or “correct” homelab. Anyone who has their own homelab will tell you that it can never be perfect. Something will always break, something can always be added, something can always be tweaked. In my own opinion, a homelab is not a static entity, it evolves with you.

Authors Note
I always try to write my blog posts semi-professionally, but always ending with a “Conclusion” feels almost too professional… I aim to write for a wider audience, often as an outlet for whatever I feel like writing about. If you have anything I may find interesting, or comments about this (or any!) post, don’t hesitate to reach out on Bluesky or via email!

References#

^[1] r/oraclecloud on Reddit - How possible Oracle shut down my always free resources instance?
^[2] r/oraclecloud on Reddit - Just like everybody says, Oracle Free Tier suddenly deletes your account
^[3] r/oraclecloud on Reddit - Can’t sign up for free tier

Commit Overflow 2024 Log - A Purdue Hackers Adventure

Lumi — Thu, 06 Mar 2025 00:00:00 GMT

Welcome to my Commit Overflow 2024 Log!#

Commit Overflow is a Purdue Hackers event that is held yearly over the Winter Recess at Purdue University.

This year, Commit Overflow ran from December 18^th to January 12^th, with a total of 43 participants. Every participant starts Commit Overflow by creating a thread in the forum channel (this years being #🟩commit-overflow-2024). Then, the goal is to “commit” something every day over winter break! Commit is in quotation marks here, as a commit doesn’t have to be code. As long as it’s furthering a creative project, it’s a commit!

The event is held to encourage making things over the long break. Not only are you encouraged to make progress over time, but you can follow what others are doing as well simply by following their respective threads! In addition, two unique Badges - one for commiting 10 or more days total (further engraved with your highest streak if it’s over 10), and another for anyone who managed to commit daily throughout the entire event, ending with a streak of 26 - were given out to participants meeting their requirements.

The format of this post will be a daily update log of my work, similar to what is being submitted to my Commit Overflow thread. Much of this is backfilled from my notes and work shown in my Commit Overflow thread. I consider myself a badge collector (I have every badge since Hack Night 5.0, and I don’t intend to miss one any time soon), so the promise of badges was tempting. At the same time, I was already looking for an excuse to work on something over break, so this was the perfect excuse.

Note to readers#

Beyond this point, this post will take on a much more relaxed tone. While everything is written with the intent of being in a post, it is also still a journal of sorts. The writing here won’t necessarily meet the same standard as my other writings do.

Each day will have the commit proof I submitted for Commit Overflow. Where applicable, this will either be a URL to my Forgejo instance (Previously Gitea, see Day 21) or a screenshot.

December 18^th#

While looking for inspiration to start Commit Overflow, I was going to take a look at my website. However, I was interrupted by a little icon on my GitHub homepage. Turns out, GitHub Copilot Free was announced a mere six hours before I began working.

This annoyed me, as I don’t like Copilot much, so my project for the day became the creation of a Firefox Add-on that blocks certain Copilot related requests on https://github.com and https://github.githubassets.com, as well as removed some elements from the DOM related to Copilot.

See: Github-Copilot-Blocker

December 19^th#

Commit 1#

Picking up from the previous day, Github Copilot Blocker received Chromium browser support. This was basically plug-and-play as the code is quite simple to begin with. All that needed to be done was packaging a .crx and updating the Readme.

See: Commit Comparison

Commit 2#

My personal website shad.moe has been in disrepair for awhile now, to the point of the entire site being a 404 for much of it’s existance.

Today marks the fourth revival attempt in “Website But Again The Sequel To The Trilogy”. Today’s commit is the creation of a base layout in Astro with Shad/CN. A sidebar with filler information and a site theme was setup and repository was initialized. Vercel was used to stand the site up on my domain.

See: Commit

Editors Note
This commit refers to my website at shad.moe. All work that was done on this was thrown out at the start of February and the repository has since been deleted. The commit URLs will reference an archive on GitHub rather than my Forgejo instance.

December 20^th#

Since I didn’t know what the site would actually look like yesterday, I looked for some inspiration. Then, I spent some time refreshing my CSS knowledge by messing around with flexboxes. I used that to setup a further frame for how I wanted the site work to progress, and I fixed some issues with the Shad/CN sidebar.

See: Commit

December 21^st#

I spent some time today to replace my Gitea instance at git.konpeki.solutions with Forgejo after hearing about it in the Purdue Hackers Discord server for some time. Forgejo is a Hard Fork of Gitea, but Forgejo maintains compatibility with Gitea. That means I could simply replace the Gitea binary with a Forgejo binary and call it a day!

Editors Note
As of Forgejo v10.0 / Gitea v1.22, a transparent migration from Gitea to Forgejo may no longer be possible. See “Gitea 1.22 is the last version to allow a transparent upgrade to Forgejo” on Forgejo’s website.

I didn’t do that. I had some issues related to my configuration with Gitea in the past, so I decided it would be a better use of my time to create a brand new instance then clone each repository and push them to the new instance.

This worked pretty well. I created a new LXC container on one of my Proxmox instances and installed Forgejo. I configured it locally, connected authentik, and that was done. All I had to do after that was install Tailscale and update the proxy_pass in my Nginx configuration to point to the new container.

December 22^nd#

Ah, Nextcloud. It’s great software, but I manage to have so many issues with it. My Nextcloud instance had been broken for weeks, so I opted to reinstall it.

Step 1: rm -rf /. Yes, I actually did this. I was annoyed and wanted to so something about it so I took it out on the soon-to-be defunct container and ran the command that must not be ran. The container was Alpine flavored, so I didn’t need --no-preserve-root - it just worked.

Step 2: Actually installing a new Nextcloud instance wasn’t that bad. I spun up a new LXC container after deleting the previous, installed Docker, and used the Nextcloud AIO installer. I also had to update the Nginx proxy_pass directive.

See: Commit - Nginx Configuration Update

December 23^rd#

After hearing some off-hand complaints and thinking about how easy it would be to automate at least a chunk of the Commit Overflow threads for organizers checking commits on various days, I decided to actually do something about it.

Introducing “Commit Overflow Test Bot”! Better name pending. This Discord bot can be activated by running /commit-count in a Commit Overflow thread, and it will automatically check for at least one of the following each day:

Git Commit URLs (Via Regex)
Attachments (Pictures, Audio, etc.)
”🟩” Reactions

This last check allows Organizers to mark a text-only message as a commit. The bot then outputs each day with a commit (alongside the verification method from above) and each missed day. Below that is a readout of the total amount of commits and the threads longest streak.

See: Commit

December 24^th#

Smaller commit today, as I was celebrating the holidays with family. I made a commit out of updating authentik, fixing configuration errors in Nextcloud, and reinstalling my private Komga instance.

I then rebooted the server due to mounting errors, hoping to clear them, and accidently knocked the server offline.

December 25^th#

Small commit again due to holidays. Fixed the broken Proxmox server. Then, reconfigured Proxmox Backup Server to allow my backups to be accessible with PBS offline.

December 26^th#

See: Grow With Google

December 27^th#

I spent part of today reinstalling Windows on my desktop computer after a year of being in a semi-broken state, then spent another part debugging and fixing a WinGet package for one of two remote desktop tools I use, NoMachine.

See: Pull Request

December 28^th#

See: Grow With Google

December 29^th#

Commit 1#

Renamed the Commit Overflow bot to Commit Overflow Helper, updated the containers timezone to be correct (Needed to be ETC - 6 to match 6am being the day change), then fixed Pull Requests to be a valid type of Git link.

See: Commit

Commit 2#

Then, I fixed a bug I noticed yesterday in winget-create that was the result of schema changes overtime. It was making multiple manifests crash the tool.

See: Pull Request

December 30^th#

A bug with dates in the Commit Overflow Helper Discord bot was found a few days ago, and I finally got around to fixing it here. There were nine total commits, but the end result was a two line change.

I also created an icon for the bot, supposedly representing a bucket of commits.

See: Commit

December 31^st#

Commit 1#

Further progress was made in completing the Grow with Google Cybersecurity Certificate coursework. Module 5 of 8, “Assets, Threats, and Vulnerabilities”, was completed.

Commit 2#

Once again had to update the timezone for the container the Commit Overflow Helper Discord bot was hosted on. This time it was switched to GMT-11.

January 1^st#

This is an… “interesting” one. The commit for today day ended up being a worldbuilding adventure in another Discord server with some of my friends.

It started with one person in the server, AxoBlu, changing their server nickname to “Obsidian Night” as a joke. This devolved into lore being created for the “character”, as well as many others creating their own personas and lore, with my own name being changed to “Princess Pawsalot”. It was mainly meant to be a purposly terrible joke, but it was taken and ran with.

In the end, we had come up with multiple different countries, kingdoms and even a pirate fleet, all encompased on a continent we titled “JOPFI” (The initials of our characters). This was multiple hours of goofing around, and the end result was really fun to look back at.

January 2^nd#

See: Grow With Google

January 3^rd#

See: Grow With Google

January 4^th#

Commit 1#

Small life commit, went with family to try Axe Throwing for the first time!

Editors Note
Images not shared for privacy reasons.

Commit 2#

See: Grow With Google

January 5^th#

Today is mostly minor things bundled into one commit. SyncThing was broken, so I re-setup the connection between my Laptop and Desktop, and synced my Documents folder between them. I then worked on some local homelab documentation, and worked in a group call to update old permissions on the Purdue Hackers GitHub Organization.

January 6^th#

Minor commit for today, I spent the day preparing to move back in to my residence hall on Purdue’s campus. Once moved in, I spent a few hours shuffling around Proxmox LXC containers to bring that part of my Homelab back online.

Editors Note
Some Clarification - University Residences at Purdue has a policy of unplugging everything in your room over break. As a result, critical services were transfered from the two local Poweredge servers to the remote Optiplex server, and the Poweredges were taken offline until my return.

January 7^th#

Commit 1#

I did an audit of my Homelab network, identifying each endpoint and the network pathing between the public web and an endpoint, as well as the pathing for machines inside the network.

Editors Note
The network graph was the proof of work for this commit, but I have opted not to attach it because it was pretty bad, exposed a fair amount of information about the internal networking of my Homelab, and would not embed well here.

Commit 2#

Once again a minor change was made in Commit Overflow Helper. I did a formatting pass with Prettier, and the code was updated to not include the current day as a missed day.

See: Commit

January 8^th#

Commit 1#

Worked on a small joke website dubbed “The Pit”, a list of companies and people who have in one way or another annoyed me or a friend. The website takes the list and plays a looping animation of them falling into a pit. The list was changed to work in a JSX, and a button was added to view a full table of occupants.

See: Commit

Commit 2#

Bluesky and the AT Protocol have been on my mind quite a bit recently. I quite like the AT Protocol, and I wanted to do something with it (and my PDS server) other than just have my Bluesky account. While researching, I discovered that one of the data fields returned when looking up a user, the service field, is fairly arbitrary. When an AppView needs to know where to find a service, it searches based on the service key. Since this is something a service queries versus a list a service has to parse through, you can realistically append any data you want harmlessly.

So I created the “service” #have_a_fox_image. You can guess what it does. TL;DR, it’s a static URL pointing to a public domain image I grabbed and uploaded to my Nextcloud instance.

January 9^th#

I came up with an idea yesterday. I was looking at things I could do with ATProto, and stumbled upon haileyok/blug, a blog built using the AT Protocol as it’s backend (essentially). I decided to see how difficult this would be, it’s possible all of this gets thrown out.

As of today, posts are already loaded when linked to directly, sans some markdown.

See: Commit

January 10^th#

Commit 1#

Made a variety of fixes and changes to my blog. These were mostly changing how data was passed around to better use the ATProto system I setup rather than Frontmatter.

See: Commit

Commit 2#

Fixed a date parsing bug in Commit Overflow Helper

See: Commit

January 11^th#

ATProtocol post fetching is finally mostly implemented, and Vercel now doesn’t have build errors. The page is now previewable live!

See: Commit

January 12^th#

Final day, and with it comes a variety of fixes and changes from the todo list I made. As of now, the blog is in a state I’m happy with, and I may push it in a few days once I write the corresponding blog post!

Editors Note
I didn’t intend for pushing this to be a full month, but I ended up cleaning the blog up much more before finally pushing the new version. I talk about this more in Welcome to the Atmosphere!. You’re reading this post on the new blog!
Also of note, the blog is now hosted locally on my Homelab infrastructure. Vercel (nor any other platform) hosts this blog.

See: Commit

Grow With Google#

Over Winter Recess, the Grow With Google Cybersecurity certificate was offered for free to Purdue students. I spent some time on December 26^th, 28^th, 31^st and Jan 2^nd completing this certificate.

Welcome to the ATmosphere!

Lumi — Sun, 02 Feb 2025 00:00:00 GMT

Update
As of March 28^th, 2026 this post is outdated. While the technical challenge was interesting and fun to experiment with, tech debt and performance issues resulted in this blog being taken off of ATProto ~14 months after publishing. The ATProto version of the blog is archived and accessible at https://git.konpeki.solutions/TheShadowEevee/atproto-blog.

In a previous post, I touched on Bluesky and the AT Protocol (or, ATProto for short). Since that post, I’ve looked further into the AT Protocol, and have come to be enticed by the concept. It’s not necessarily a novel system, but it is a system that’s becoming more widely used by the day.

Definition: The AT Protocol#
“The AT Protocol stands for “Authenticated Transfer Protocol,” and is frequently shortened to “atproto.” The name is in reference to the fact that all user-data is signed by the authoring users, which makes it possible to broadcast the data through many services and prove it’s real without having to speak directly to the originating server.” - atproto.com^[1]

This was a fun project to work on and I’m glad to have everything in a working state! I’ll talk about my thought process behind this and some of the inner workings of the blog and the AT Protocol. At the end, I’ll also include a few caveats; this isn’t a perfect system after all!

View the Blog Source
If you would like to host this yourself, or would like to learn how this all works, the source code is available on my Forgejo instance and Github!

The What#

Before I get into any details, I should explain the title of this blog post. My personal blog, blog.shad.moe, is now an ATProto AppView! What this means specifically is that my posts are no longer markdown files stored in a git repository. Instead, each post is an entry on my Personal Data Server (or PDS)!

It’s similar to a post on BlueSky. If you go to my Bluesky profile right now, you will see a few posts. Clicking on one directs you to a URL resembling https://bsky.app/profile/{Handle or DID}/post/{rKey}. The first blank is one of two identifiers for a Bluesky account; either a “handle”, a domain name like username, or a unique DID (Decentralized Identifier)^[2] for an account. If a handle changes, the DID stays the same unless a new account is made.

The second part is the key. Or quite literally, the Record Key (rKey for short). The rKey looks like a random string of characters at first, but it’s often actually something more. The most common rKey is the TID format, but there are other formats that may be used. The important part is that the rKey is a unique key that when passed to a PDS server alongside a “repository” AKA your handle or DID, will return a specific piece of content. In the case of Bluesky, that would look like a post.

Circling back to my blog, a familiar format is used! As the blog is intended for one person, the profile is hardcoded in for ease. With that being the case, the url scheme this blog follows drops the profile part to become https://shad.moe/posts/3lgbnigwdzk2r/! The rKey 3lgbnigwdzk2r is the key to this specific post on my PDS, and thus is all the information needed to render the post.

The Why#

While I already was in control of my blog posts and the platform they were on, moving to the Atmosphere opens up federation of the posts, potentially increasing their view. In addition, I’ve been looking for a project I could use the AT Protocol on for some time now, cause I simply think it’s cool.

The AT Protocol is, as described by the developers, a social protocol. Despite being a relatively new system, many awesome projects all ready exist, testing the limits of the protocol! A short list of my favorites include BlueSky, WhiteWind, GrayHaze, and ATFile.

The How#

The way I made this work is rather interesting, and pulled from a lot of existing community knowledge to cobble together. Perhaps the greatest help and inspiration was WhiteWind. Whitewind is a pre-existing ATProto based blogging service, and their lexicon is what I’m using for my posts. Specifically, each blog post has a type of com.whtwnd.blog.entry. Since these posts are inherently public, they can be viewed from my blog, WhiteWind’s website, or a PDS viewer such as PDSls.

Similar projects that helped#

I referred to haileyok/blug and rwietter/atproto-blog both multiple times during work on the transition. haileyok’s blug project specifically pushed me to implement Redis caching for posts on the backend. If you attempt to do something similar to this, please use a caching system of some sort. One downside of ATProto is that you generally have to make a web request for each view; a proper cache will immensly cut down traffic to your PDS and speed up loading times on your blog.

The Blog’s Design Itself#

If you’ve visited my blog before, you’ll notice it looks the same as it did before! I put a strong effort into retrofitting ATProto into saicaca/fuwari, the Astro-based blog view I used up till now. I really like the design of Fuwari, and didn’t want to sacrifice it for this.

Loss of Metadata#

One of the biggest challenges of retro-fitting was the loss of metadata. Fuwari uses Frontmatter to store important data about a post, like it’s title, description, category, and tags (among a few other things). However, the WhiteWind Lexicon for com.whtwnd.blog.entry defines different data! The only matching field is the title field. To work around this, I did something I still consider a bit cursed. Markdown allows the use of HTML; I took advantage of this to use an HTML comment to act as a Frontmatter field of sorts.

It’s a strange solution, but it allows passing values Fuwari would have previously seen as what I’ve deemed “Extended Data”. When the blog view pulls a post, it will be parsed for a string like the following: 

This string can be broken down into a header, a footer, and a JSON object. This JSON object is extracted when the header and footer are found, and serialized to be passed along as metadata. Notably, no AppView should display this data in the post! Both Fuwari and WhiteWind’s AppViews parse markdown, hiding the comment from the reader.

Verceln’t#

Something else of note is that Vercel no longer hosts this blog. I spent way too long on trying to get post fetching to work, but Vercel refused to build the project to have dynamic paths or serverless functions no matter what I tried. This gave me the excuse I needed to move from Vercel to my own infrastructure! This blog is currently hosted on an Optiplex that serves as the outer edge of my Homelab network, but will eventually be moved onto the Dell Poweredge R640 servers I mentioned in a previous blog post.

The Caveats#

As I mentioned before, the Atmosphere is not all sunshine and rainbows. There are a few notable issues with this system that are important to understand before undertaking this adventure.

Network Requests#

Every time someone loads the blog, a web request is made. And often, more than one. This is something that is unavoidable; the entire point is that the blog AppView needs to call out to my (or the authors) PDS. This can result in longer load times as the web requests are made, and can be delayed if something is wrong with the network.

However, this is acceptable in many cases. In the case of network issues, it’s highly likely that your blog would be inaccessible in the first place since users can’t connect to it. In the case of sending many requests that can potentially take some time to resolve, this is relieved through the use of caching. Specifically, a KV database, Redis, is used in the backend to cache posts locally.

Different AppViews#

The first caveat is the potential for a post to differ slightly depending on where it’s viewed. As mentioned before, each post can be viewed directly on my blog or on Whitewind. It may also be viewed by any other AppView that chooses to implement the Whitewind lexicon. While these posts are written in Markdown, which should be universal, there can be differences in how Markdown is rendered. A great example is Admonition cards: on my blog, they create a nice block to grab a readers attention! On other AppViews, these aren’t rendered and instead appear as text.

Image: An Admonition card as it would appear on my blog#

Image: An Admonition card as it would appear on WhiteWind#

Posts are Inherently Public#

Finally, it’s important to note that posts are always public (unless you’ve configured your PDS to require authentication to view certain things. YMMV.), so if you’re drafting a post that you don’t want to be viewable, you should hold off on posting it to your PDS.

This is something I am fine with. My flow is to use the WhiteWind editor (eventually an editor on my own blog) in “Anyone with the link” privacy mode. This hides the post on Whitewind unless the reader has the exact rKey, and my blog is configured to not show these posts but to still allow direct URL access without caching. This enables quick reviews everytime I save the post, and allows me to share the unlisted link with friends for review. Once a post is ready, simply changing the privacy mode to public “publishes” the post, making it appear on the blog’s front page and enabling my RSS.xml to pick up on the post and distribute it.

(Potential) Future Plans:#

Finish the editor at blog.shad.moe/editor
Add comments and reactions sections
Change URL Schema to allow multiple users? b.s.moe/author/{acc}/post/{rKey}?
- Currently possible but probably won’t be implemented for my blog
  - Changing URL schema could/would break SEO again
  - Maybe have Single / Multi-User toggle in code?

Conclusion#

I don’t have much of a conclusion for this post. This is a project I started near the end of Commit Overflow (Post soon on this event!). I began on January 8th, 2024 and finished January 21st, 2025, which is less time than I thought until I typed it out. This project was honestly quite motivating, and I’m glad that I went through with it instead of scrapping it like I almost did on two seperate occasions. All in all, I’m excited for this new system! If anyone wants to try this for themselves, you can find the repository on my Forgejo instance and Github as mentioned above. If you have any questions, feel free to shoot me a message on BlueSky or through email!

References#

^[1] Quick start guide to building applications on AT Protocol - atproto.com
^[2] ATProto DIDs - atproto.com

Notes for Self-Hosting an ATProto PDS

Lumi — Thu, 12 Dec 2024 00:00:00 GMT

Recently I’ve started using Bluesky, a decentralized social media platform built on the AT Protocol. When I was first looking into Bluesky, something that attracted my interest was the ability to host your own Personal Data Server, or PDS. To quote the AT Protocol Glossary:

PDS (Personal Data Server)

A PDS, or Personal Data Server, is a server that hosts a user. A PDS will always store the user’s data repo and signing keys. It may also assign the user a handle and a DID. Many PDSes will host multiple users.

A PDS communicates with AppViews to run applications. A PDS doesn’t typically run any applications itself, though it will have general account management interfaces such as the OAuth login screen. PDSes actively sync their data repos with Relays.

All of that boils down to “This is where your user account and associated information is stored”. I personally host a private PDS at konpeki.solutions - my Bluesky handle is @theshadoweevee.konpeki.solutions. This is entirely local! Any time someone access my profile or a post, a request is sent to the domain https://theshadoweevee.konpeki.solutions.

A lot more goes into this that I haven’t touched on (like the Firehose) but I’ll stop here for now. It’s time to complain a bit.

The AT Protocol has, in my opinion, great documentation on how the protocol works. It’s worth browsing through the documentation if you’re curious. Where it falls flat, however, is documenting any non-default configuration of the PDS.

Configuring Email#

Configuring an Outgoing SMTP Server is necessary for Email Verification and Password Resets (both nice to haves). There is documentation for this now, but at the time I set up my PDS server, it was restricted to a Pull Request and even then, the current documentation doesn’t really clarify what an SMTP URL is very well.

The trick to this is configuring an environment variable. My environment variable file is located at /pds/pds.env, yours will likely be in a similar spot. You’ll append the following to the end of your environment variables:

1
PDS_EMAIL_SMTP_URL=smtps://<Username>:<API Key / Password>@smtp.example.com:465/
2
PDS_EMAIL_FROM_ADDRESS=<Username>@example.com

In some configurations for the url, such as mine, the username must be accompanied by the domain. In this case, replace the username with <Username>%40example.com where example.com is your domain name.

After configuring, restart your PDS server. If you used the install.sh / Official PDS installation method, you can run systemctl restart pds.

Configuring Reverse Proxies#

Here’s a much more annoying task: Configuring a reverse proxy. The PDS Server runs it’s own Caddy instance, but there are many reasons you may want a reverse proxy in front of that. My main reason was that the PDS server should be accessible over Tailscale to my Nginx server, which then serves to the world.

On top of this, I wanted to use my username + domain root and not a subdomain (No theshadoweevee.pds.konpeki.solutions!). However, this raises the issue of konpeki.solutions being the home of my PDS while I still wanted to host content on the domain. I did some… funky workarounds to get this working.

Each of the following assumes I’ve already created an account on my PDS with the username theshadoweevee.konpeki.solutions using pdsadmin account create and that I know it’s DID, did:plc:krbzbucjaj76xjob6ju47ilo, from pdsadmin account list.

Setup A/AAAA records for your handle#

Your handle must be a domain you have control over in some fashion. For all intents, your handle is also a domain. A DNS A/AAAA record needs to be configured for theshadoweevee.@ which will point to a server you control (where your reverse proxy is hosted, in this case). Ensure the server being pointed to has a valid SSL certificate for your handle, or a wildcard for the domain.

Verifying your handle via DNS#

If you have access to your DNS records, this is the easiest method to verify a handle and requires no setup on the PDS Server or reverse proxy. Head to your domain’s DNS records, and create a new TXT record with the following, substituting the Username and DID:

1
Name: _atproto.theshadoweevee
2
Content: "did=did:plc:krbzbucjaj76xjob6ju47ilo"

Note the quotation marks around the content, this is a requirement in TXT records. Wait for your DNS records to propagate, then check your handle with the Bluesky Debug Page.

Verifying your handle via HTTP#

If DNS isn’t an option, or if you want to have both verification methods setup, then you can also verify your handle using a file. This file should be at https://theshadoweevee.konpeki.solutions/.well-known/atproto-did, where your handle is the domain name. Inside atproto-did, the contents should simply be your DID without quotes, such as did:plc:krbzbucjaj76xjob6ju47ilo.

Proxying actual content: Handle#

At this point, you aren’t actually proxying any data from your PDS. To proxy a handle domain, you only need to proxy one path (that I’m currently aware of). Proxying /xrpc to your PDS server will pass through any HTTP API requests made to your handle domain. It is important that you correctly proxy websocket requests.

A nice to have you can implement on a handle-only domain is redirecting to your Bluesky profile. You can do this by redirecting or rewriting / to https://bsky.app/profile/did:plc:krbzbucjaj76xjob6ju47ilo, replacing the DID with your own username or DID (both work, however the DID survives handle changes in the future).

Example Nginx Configuration can be seen on my Gitea instance.

Proxying actual content: PDS#

Proxying your PDS root domain is more complicated, as there’s more paths to be aware of. As before, ensure you have a valid SSL configuration and websockets are proxied. Then, you’ll need to proxy each of the following to your PDS server:

/xrpc
/.well-known/oauth-protected-resource
/.well-known/oauth-authorization-server
/oauth
/@atproto

Proxying these ensures HTTP API requests get through (/xrpc), as well as a few other user management services such as OAuth. When Discord added Bluesky to their connections list, this was an issue I ran into as I hadn’t proxied the OAuth endpoints. This resulted in Discord returning an HTTP 500 error when trying to link the accounts.

Example Nginx Configuration can be seen on my Gitea instance.

Wrap Up#

The AT Protocol is something I hope to see used more in the future, though currently I take some issue with the problematic self-hosting documentation. With enough googling and some investigative work however, there isn’t an issue that I haven’t deemed unsolvable yet.

I ended up running into my issue with Bluesky/Discord linking, fixing the issue, and writing the majority of this in one sitting at a campus Starbucks, so I feel I have to give some credit to the baristas who made my drink, and to the people on campus who put up with my complaining. Sometimes the most difficult issues become simpler with something to drink and a friend to vent to.

Renaming an Active Node in a Proxmox Cluster

Lumi — Wed, 11 Dec 2024 00:00:00 GMT

Renaming Instructions
Are you here for the instructions? Click here to skip ahead!

So, there’s a bit of a story here. Back in September of this year, I made an impulsive purchase at the Purdue Surplus Store. Specifically, I aquired two Dell PowerEdge R640 Rack Servers for $100 USD each. I won’t go into the specs, but these were a nice upgrade to my Homelab environment from my then current Dell Optiplex 5040 SFF. (Also a Purdue Surplus purchase)

It took some time to get the servers setup and connect to their iDRAC controllers, but once I found a Micro USB cable and the drivers I needed, I was ready to install an operating system! If it wasn’t clear, I went with Proxmox.

Each of these machines was loaded up with a fresh install of Proxmox, and following my loose naming scheme of Touhou based hostnames, they were named moriya-pve1 and moriya-pve2, referencing the Moriya Shrine. Then, I installed Tailscale on each machine, and clustered them into the Moriya-Cluster

Early Mistakes#

If you’ve worked with Proxmox clusters before, some of these mistakes may seem obvious. But when working on my own environment, I like to do things to see what they do, and clean up the mess later. Let the following mistakes be a warning to those who homelab the same way.

Joining a remote server to the cluster#

Very early on on the first creation of moriya-cluster, I had the bright idea to join my Optiplex to the cluster as well. This was not smart. To quote the Proxmox Documentation, “[Corosync] needs a reliable network with latencies under 2 milliseconds […] to work properly”. Corosync is the name of the service that handles cluster management in Proxmox. The main problem here, is that the Optiplex was over 75 miles north. The latency was… more than two milliseconds. The cluster threw a fit. Since I had no containers yet, this was the cause of the first full Proxmox reinstallation.

Not planning names ahead of time#

My homelab hostnames follow a pattern. The Optiplex is Momiji-Server, for example. So, the decision to use Moriya-PVE1 and Moriya-PVE2 was a poorly made one. Sure, the names worked, but they were certainly not ideal. For this reason, I opted to rename them on November 14th. This is “the incident” that I’m writing about now.

Making changes with limited time#

I opted to rename my nodes, a major change, with only an hour to work with. At the time, I was sitting in a public space, waiting for a friend of mine to return with a U-Haul carrying a Robot Arm and Controller from a freight company. This is a whole other story, but in the realm of this post, what matters is I only had about 45 minutes to do repairs. This meant the whole cluster was down for many hours until I was free again.

What actually happened#

Renaming a node in Proxmox is not an easy task. There is a wiki page on how, but it lays out the following requirements.

The node must be empty (No containers, VMs)
The node must not be clustered
- This rule can be ignored, but there is an extra step if you do

I read this, and in my infinite wisdom thought “what’s the worst that could happen?”. So I renamed my nodes. Nodes that were clustered and had multiple LXC containers. That, and I renamed them both simultaneously.

I updated the /etc/hosts and /etc/hostname files (ignoring the optional files), and restarted the servers. Notably, I didn’t update the corosync.conf file, but that was only a minor issue compared to what actually went wrong.

After fifteen minutes, the servers connected back to my Tailscale network, and I opened the Proxmox webpages. Read-only. Forgetting to edit the corosync.conf caused a quorum failure, which I was lucky able to fix using the terminal access provided by my iDRAC controllers (accessible over Tailscale).

Another reboot, and the core of the problem became evident. There were now four nodes listed in the Server View. The newly renamed servers kanako-server and suwako-server were there, but they listed no LXC containers. There should have been 7 containers, so this was concerning. Also displayed was Moriya-PVE1 and Moriya-PVE2, the old names. The LXC containers were found under those nodes, and all were offline as the non-existant nodes couldn’t be found. Uh oh.

Investigating#

I promptly began searching for solutions. I had already spent a lot of time troubleshooting, and the group of people that had gone with the U-Haul were already on their way back.

My research was conclusive… I should reinstall my server and restore my containers from backups. In a seperate incident that same day, I had made my backups inaccessible as I switched to a new storage system. So I didn’t have backups. Oops.

One Proxmox forum user, shodan, then suggested to restore the backup of /var/lib/pve-cluster/config.db. This made me curious, and turned out to be the key to restoration.

Instructions: Changing a Hostname on an Active Node#

Click here to view the instructions!

Disclaimer
The following commands can cause irreversible damage and data loss!
It is advised to understand these steps yourself before running them. These commands have only been tested in a sandbox, and vary slightly from my recovery steps.
I am not liable for any issues that may arise from these commands.

Version Notice
These steps were ran on Proxmox VE 8.2.7, and may or may not work on other versions.

Restarting Nodes
Do not restart any nodes until reaching the restart step! Restarting nodes early can cause a number of issues and junk folders. It is recommended to use the reboot command.

Recovering Nodes
If you need to recover a node that you already renamed, try changing the hostname back. If that doesn’t work, you will need to manually change inode and parent ids in the database for each folder and relevent file.

Choose new hostnames for one or more nodes
Connect locally to the Proxmox server
- Do not use the web interface, as the next step will temporarily stop the services it needs to run
Change the hostname in /etc/pve/corosync.conf

Stop all running Proxmox services ^[1]

Terminal window

1
killall -9 corosync
2
systemctl restart pve-cluster
3
systemctl restart pvedaemon
4
systemctl restart pveproxy
5
systemctl restart pvestatsd

Change the hostname in /etc/hosts and /etc/hostname
On the renamed node, run:
Terminal window
```
1
sqlite3 /var/lib/pve-cluster/config.db
```
Making note of the entries, run:
```
1
SELECT * FROM tree WHERE type=4;
```
- Alternatively use an online viewer such as this one
- The most important fields are inode (First Column) and name (Seventh Column)
Identify the entry for each node you are renaming
For each node being renamed, run:
```
1
UPDATE tree SET name = "NEW_NAME" WHERE inode=0;
```
- Change NEW_NAME to the new hostname, and 0 to the inode number for the entry of the node
If the node is in a cluster, repeat steps 2-9 on each node in the cluster, even if it’s not being renamed
- This is important, as in a cluster environment corosync can sync the old database to the server, deleting the changes
Restart each modified node
- Ideally, have an alternative access plan for the servers; if there is an error, the web interface may not come back up over the network
Edit /etc/pve/nodes/NEW_NAME/ssh_known_hosts and correct the hostname on all nodes - Replace NEW_NAME with the new hostname

Conclusion#

The clear moral here is to listen to warnings, especially when given by multiple sources. I was clearly warned by the documentation and multiple forum posts, however I opted to do it anyway, as I consider my homelab a breakable environment. Proxmox is certainly good at what it does, but it also cares a lot about it’s state. Messing with configurations Proxmox expects not to change was as bad of an idea as I was warned, but luckily I was determined to make it work.

References#

^[1] spirit - Stopping all proxmox services on a node

A writeup of KC7's New2Cyber 2023 event

Lumi — Thu, 16 Mar 2023 00:00:00 GMT

Update 11/20/24
When reviewing this post, I noticed some changes to the KC7 platform. All links have been updated, but will retain the original name as written. For reference, the old New2Cyber challenge is now called “Envolve Labs - With a twist!”.

The KC7 New2Cyber Game was a Cybersecurity challenge hosted on the KC7 platform alongside the SANS New2Cyber Summit on March 14th. It was a “Jeopardy” style challenge, meaning it was broken into sections of questions you could answer at your leisure. This writeup will consider the main part of the event as sections 2 and 3, given they hold the majority of the KQL work and all of the incident discovery.

For more information on this event, you can visit the KC7 page for it at SANS New2Cyber - KC7 (kc7cyber.com). This page includes the dataset used, a link to the event leaderboard/submission page, and their training guide. The training guide is a very well written explanation of Module 1 meant to guide you through learning the basics of KQL while completing most of Section 1.

Going forward, the bulk of this event will be referred to as “the security incident” or just “incident”. This writeup will both go over the event’s main portion giving a narrative of the incident and guiding through many of the steps taken throughout.

Warning: Spoilers / Answers Ahead
This event has finished, but the organizers at KC7 have said it will stay live indefinitely. If you haven’t tried the event yet, give it a go yourself first! This is a good learning tool to go in blind for at first; you can always come back here for guidance along the way.
Sections 2 and 3 of this event are discussed to the point of basically giving away solutions below. Sections 4 and 5 are not included and can be mostly found with some googling.

The Scenario#

View the KC7 New2Cyber training guide to read the full scenario! SANS New2Cyber - KC7 (kc7cyber.com)

Paraphrased scenario from the training guide:
Today is your first day as a Junior Security Operations Center (SOC) Analyst at Envolve Labs Corporation. Envolve Labs is a (fictional!) 2012 med-tech startup with a goal of designing flexible vaccine tech. Envolve Tech has 4 key partners (listed in the training guide) that will be seen throughout this event.

Envolve Tech has hired you as they realize they need to invest more in Cybersecurity efforts. Envolve Labs collects log data about the activities its employees perform on the corporate network. These audit logs will be the subject of your investigations as you determine whether the company is being targeted by malicious actors.
_{All credit for the above scenario goes to KC7!}

Activity Summary#

After receiving the tip on the potential phishing domain, a larger investigation was started. One person had clicked on the malicious domain and investigating this further led to discovery of potentially 2 different potential malicious actors.

Hereby referred to as Actor A and Actor B, these two actors are for now suspected to be different based on their methods and time of intrusion. Actor A used a wide scale phishing campaign to steal credentials and deliver malicious files meant to allow exfiltration of data. Actor B appears to be a different, more advanced actor using more targeted phishing using compromised emails, and more persistent tools on infected systems.

Actor A could potentially be a group of malicious actors, as many domains and IP addresses showed similar methods and similar timing. There were 3 total groups of interconnected IPs and domains that fit the exhibited behavior, and the similarity in their operations are what linked the 3 groups as one larger actor.

Actor Methods of Operation#

Actor A#

Actor A used a simple phishing scheme in order to get a handful of employees to share login information or download malicious files. The extent of this campaign appears to have aimed to gain login credentials, which were used to download files from the email system.

Actor A operated primarily on January 4th, 8th, and 9th.


Recon	Collected Email Addresses of multiple employees, did research on important company figures (i.e., the CEO)
Weaponization	Unknown
Delivery	Shared malicious files and URLs through email via phishing campaign
Exploitation	Unknown
Installation	Unknown
Command and Control	N/A
Actions	Used access to user accounts to download critical files over open internet

Actor B#

Actor B appears to be a more advanced actor than Actor A. Actor B used similar methods of initial infection, sharing malicious files via phishing emails. However, there were less emails suggesting a more targeted approach. On top of this, the infected files were better hidden using file names such as “EnvolveLabs_Research_Tool.7z”, and “updater.dll”. These files were designed to be stealthy and remain undetected.

The updater.dll file that extracted itself from the .7z file allowed a direct, persistent connection to the malicious actor. Once the updater.dll was installed, it appears to drop the file “infector.exe” and operate it via a Windows Scheduled Task. Together, these two files opened up paths for the actor to control the computers via Ligolo and PuTTY. In total, only 7 employees were infected, again suggesting a more targeted attack.

Actor B operated primarily on January 5th.


Recon	Likely to have researched various positions within the company
Weaponization	Use of common and specialized file names are meant to avoid detection and maximize attack ability
Delivery	Shared malicious files through email via phishing campaign
Exploitation	Extraction of a malicious 7z file resulted in infection
Installation	updater.dll was used to create a persistent infector.exe
Command and Control	Actor opened paths for remote control via Ligolo and PuTTY
Actions	Unknown

Impact#

The impact level of this event is hard to determine. However, this appears to be a Mid-High severity incident (on an arbitrary scale) given what impact we do know of.

Actor A has gained access to a large number of employee accounts, compromising security in more ways than they may know how to take advantage of yet. They have already compromised email accounts in order to download seemingly sensitive files (based purely on file name) and may be able to use the compromised accounts to send emails spreading their phishing campaign further.

Actor B on the other hand has direct, persistent access to multiple systems potentially owned by higher ups, putting unknown amounts of information at risk. Actor B’s method of infection makes it hard if not impossible to tell what files may have been exfiltrated already using the logs provided here.

Discovery Process and KQL Examples#

The discovery process for this incident will follow the order of the questions asked in the KC7 answer submission page. Not all questions will be explored as there were 39 questions total in Sections 2 and 3, with some being repeats or not closely related enough to others to make a quick narrative from.

The KQL queries used during the discovery process will be shared to show what is happening and make it easy to follow along. This should ideally be understandable if you haven’t looked at the KC7 challenge but will be a good guide if you are actively working on it or have completed it.

Disclaimer: All URLs, IP addresses, Names, Emails, and other data is generated for this event by the KC7 team, and any mention is NOT an endorsement or claim of malicious intent for or against listed entities.

Section 2 - Introducing the Hackers#

Section 2 has a general focus on identifying the suspicious domain and learning initial information about our first actor.

Questions 2.1 - 2.4#

The incident discovery begins with a suspicious domain, immune.tech. In the training guide, it is stated that the domain was found by a security researcher and publicly shared as a suspicious domain found while looking into phishing activity.

Since this domain was potentially related in phishing activity, we should check to see if any of our employees have received emails with a link to this domain. We can search with the following KQL query.

1
Email
2
| where link has "immune.tech"

This query shows all the emails received with a link to immune.tech, which will be used to answer the questions. This query checks the Email table, searches all entries in the link column for the text immune.tech, and returns any positive results.

Questions 2.5 - 2.8#

Knowing that multiple employees received an email with the suspicious domain, the next step is to identify if a user clicked on the link at any point. We can search the OutboundBrowsing table to identify any potential clicks.

1
OutboundBrowsing
2
| where url has "immune.tech"
3
-- One result was returned with a src_ip of 192.168.0.234, we can search the Employee table to see who this was
4
Employees
5
| where ip_addr has "192.168.0.234"

It appears that “Marissa Stephens” did in fact click the link in one of the emails. It is likely this was a phishing email, and they may have given away their login to the actor behind the domain. We should check logins to their account.

1
-- Visible in the Employee table, usernames are first two letters of their first name, and their full last name, all lowercase.
2
AuthenticationEvents
3
| where username has "mastephens"

As we worried, it appears that there was a login to Marissa’s account 9 hours later from IP address 88.150.11.111. Now, this could be Marissa logging in, we can’t be certain yet. Let’s check if this IP has been used to log in before.

1
AuthenticationEvents
2
| where src_ip has "88.150.11.111"

It appears this issue runs far deeper than anticipated. We can use a new KQL term, count, to get the number of returned results. Running the query again with | count added on a new line shows a total of 21 logins from this IP address, all on separate accounts. Not only is this very likely to be a malicious actor, but it also seems they have access to more than expected.

Tips for Questions 2.9 - 2.13#

2.9 - .gzip is a file extension, try searching for a URL ending in this extension.
2.11 - Searching for this person as a recipient will turn up multiple results, look for a suspicious sender or misspellings!
2.13 - Use the distinct query on the link field to isolate unique instances.

Section 3 Examples - Hackers sending malware docs#

Section 3 begins to focus on another type of threat. The bad actors have sent multiple malicious documents over email, from a variety of emails, some being compromised emails from partner companies. Our second actor makes an appearance for a short time here.

Questions 3.1 - 3.5#

The continuation of the discovery process starts with a new suspicious domain, notice.io. This domain was called out for having a specific malicious file, “Critical_Security_Path.docx”.

Following up on this domain, we need to see if anyone was targeted.

1
-- How many emails with this domain were recieved
2
Email
3
| where link contains "notice.io"
4
| count
5
-- Who sent the emails, and what was the subject line
6
Email
7
| where link contains "notice.io"

It appears this email has found it’s way into some employee’s inboxes. Using the same methods as section 2, it’s possible to find out that one of the employees clicked the link, and what time the hosted file was downloaded at.

Questions 3.7, 3.12, 3.14#

Between questions 3.5 and 3.16, the questions are similar to previous questions. These 3 however are new.

1
/* 3.7 - What other domains are hosted on the same IPs as notice.io? */
2
-- Using PassiveDns, we can determine the IP address for notice.io
3
PassiveDns
4
| where domain contains "notice.io"
5
-- Then, we can test the returned IP addresses and see what other domains they might resolve to
6
PassiveDns
7
| where ip contains "21.101.248.158"
8
/* 3.12 - How many IPs does scanverify.com resolve to? */
9
-- Similar to part one of 3.7, see what IPs scanverify resolves to and count them
10
PassiveDns
11
| where domain contains "scanverify.com"
12
| count
13
/* 3.14 - What is the name of the file hosted on scanverify.com? */
14
-- One way to do this is to check OutboundBrowsing for someone who has already navigated to the file, and copy the file name from the end of the link
15
OutboundBrowsing
16
| where url contains "scanverify.com"

Questions 3.16 - 3.26#

Starting here, we go deeper into what this actor is doing for recon and how they are infecting machines once their email is clicked on.

First, we can look into what recon the actor may be doing. They may have searched for some systems, and what we’re looking for is guided by Q 3.16 and 3.17.

1
-- The full query is "helpdesk ticket system", but using just helpdesk works better since we can't use spaces
2
InboundBrowsing
3
| where url contains "helpdesk"
4
-- The next query asks what policy they're looking for. The following returns multiple results, some with the search we're looking for of "Password Expiration Policy"
5
InboundBrowsing
6
| where url contains "policy"

Now we want to look into what the “EnvolveLabs_Research_Tool.7z” file does. We’re told a .dll was dropped on the machine after downloading the 7z file, and we have to find the name of it.

1
-- Find IP Addresses that downloaded the file
2
OutboundBrowsing
3
| where url contains "EnvolveLabs_Research_Tool.7z"
4
-- Find the employee to get their computer hostname
5
Employees
6
| where ip_addr contains "192.168.1.129"
7
-- Search for file creations using the computer hostname
8
FileCreationEvents
9
| where hostname contains "5RT9-MACHINE"

This reveals a .dll file being created about 40 seconds after the .7z file is downloaded, named updater.dll. We should now check to see if any commands have been run on this computer.

1
ProcessEvents
2
| where hostname contains "5RT9-MACHINE"

This reveals the updater.dll file running 2 commands, “whoami” and a command using Windows Task Scheduler.
_{Note: This command (starting with schtasks) is the answer to Q 3.20. Use this information to complete Q 3.21; Q 3.20 does not exist due to an error.}

We can take the task scheduler command and check to see if other systems may have run it.

1
-- The full command was omitted here. It is recommended to search using the entire command in most scenarios.
2
ProcessEvents
3
| where process_commandline contains "schtasks"

Now that we know more machines are infected, we need to check for this file on other machines.

1
FileCreationEvents
2
| where filename contains "updater.dll"

Using the sha256 hashes provided, we can see 4 unique files with the same name, on 4 different machines. We can search the hash of one of them on VirusTotal to see how many antivirus programs detect it as malicious. We can search for known tunneling tools to see if they have been used on these infected machines. First we can check for the use of ligolo, as well as see if any other tools are being used.

1
-- We're asked to look at a specific employee first
2
Employees
3
| where name contains "Michelle Beal"
4
-- See what IP address is being tunnled to
5
ProcessEvents
6
| where hostname contains "UY2V-LAPTOP"
7
| where process_name contains "ligolo"
8
-- Check for other systems and programs contacting the found IP address
9
ProcessEvents
10
| where process_commandline contains "158.129.161.214"
11
-- Check to see how many ligolo tunnels are in use across the network
12
ProcessEvents
13
| where process_name contains "ligolo"
14
| count

Finishing this, that’s the extent of the discovery asked for. If you’d like, feel free to look around more and play with the data, and make your own assumptions on what is going on behind the scenes! Who knows, I may be wrong entirely with my assumptions on the number of actors and their roles. It’s up to how you interpret the data in front of you.

Conclusion#

If you are interested in participating in future KC7 events, I recommend checking out their website at https://kc7cyber.com/. I had a great time talking with others in the slack channel created for this event, and I can safely say that Question 5.20 has scarred many of us for life with how many attempts were poured into it. (The answer was a hack and leak attack by the way! You can read some interesting news stories if you google it.) Question 20 was the last question I had left and was the only question that resulted in me finishing everything second instead of first. Congrats to oclku9rt for pulling that out from under me for the first 100% completion!

All in all, this was a great event, and I had a great time working through some of the problems presented. The KC7 platform seems to be a great learning tool with this hands-on exploration into a realistic environment simulating an actual security event you may encounter as a Security Analyst. As a participant who hasn’t done something like this before, KC7 did a great job making this engaging while still keeping everything at a good skill level for those participating in the SANS New2Cyber Summit, from anyone brand new to the Cyber scene as well as long time members of the field.

TheShadowEevee's Blog

Gridania Roundup - The Advent of `.`

The Appeals of Homelabbing

Definition: Homelab#

My Hardware#

So… Why bother homelabbing?#

What is Anubis?#

There’s Always Trouble in Paradise#

A Place to Try New Things, and Break Old Ones#

How does one start a homelab?#

I don’t want to purchase anything#

I want to own my hardware#

I want to rent a server#

Conclusion#

References#

Commit Overflow 2024 Log - A Purdue Hackers Adventure

Welcome to my Commit Overflow 2024 Log!#

Note to readers#

December 18th#

December 19th#

Commit 1#

Commit 2#

December 20th#

December 21st#

December 22nd#

December 23rd#

December 24th#

December 25th#

December 26th#

December 27th#

December 28th#

December 29th#

Commit 1#

Commit 2#

December 30th#

December 31st#

Commit 1#

Commit 2#

January 1st#

January 2nd#

January 3rd#

January 4th#

Commit 1#

Commit 2#

January 5th#

January 6th#

January 7th#

Commit 1#

Commit 2#

January 8th#

Commit 1#

Commit 2#

January 9th#

January 10th#

Commit 1#

Commit 2#

January 11th#

January 12th#

Grow With Google#

Welcome to the ATmosphere!

Definition: The AT Protocol#

The What#

The Why#

The How#

Similar projects that helped#

The Blog’s Design Itself#

Loss of Metadata#

Verceln’t#

The Caveats#

Network Requests#

Different AppViews#

Image: An Admonition card as it would appear on my blog#

Image: An Admonition card as it would appear on WhiteWind#

Posts are Inherently Public#

(Potential) Future Plans:#

Conclusion#

References#

Notes for Self-Hosting an ATProto PDS

PDS (Personal Data Server)

Configuring Email#

December 18^th#

December 19^th#

December 20^th#

December 21^st#

December 22^nd#

December 23^rd#

December 24^th#

December 25^th#

December 26^th#

December 27^th#

December 28^th#

December 29^th#

December 30^th#

December 31^st#

January 1^st#

January 2^nd#

January 3^rd#

January 4^th#

January 5^th#

January 6^th#

January 7^th#

January 8^th#

January 9^th#

January 10^th#

January 11^th#

January 12^th#