Introduction

I’ve been a huge fan of VR ever since we saw some of the first few consumer products hit the market in around 2015/2016, and so back then I got myself a HTC Vive headset and controllers and upgraded aspects of my gaming PC to accommodate the more strenuous requirements at the time. The game Beat Saber has particularly piqued my interest (in addition to a few other discoveries such as Space Pirate Trainer, SUPERHOT, and Pistol Whip). I also like occasionally streaming my ventures on Twitch. However there are some significant differences when trying to stream from the perspective of a typical keyboard+mouse/controller non-VR game, versus from the perspective of a controller/tracker VR game.

Current VR Setup

The setup I’ve had for some time is an original HTC Vive Headset from around 2016, complete with the 2016 era controllers. It comes with two lightboxes, which are used to track the position of the headset and controllers with a remarkable amount of accuracy and smoothness.

The VR headset itself supports a resolution of 2160×1200, which is about 1080×1200 per eye. To support this sort of resolution, you generally needed a pretty beefy graphics card at the time, so here mine is supported by a GTX 1080.

Streaming VR vs non-VR

All the VR games generally have a non-VR window that is displayed on the desktop monitor. You can easily capture that output using some popular off-the-shelf software, such as OBS. The view is somewhat indicative of what’s visible in the headset, usually from one eye, but of course unless the game supports adjusting the FOV, the default FOV has a hard time showing everything that’s going on. There’s also the problem of, unlike controlled in a mouse/keyboard/controller situation, human bodies actually sometimes have very jerky movement. As the wearer of the headset you don’t notice since your eyes naturally locks onto targets while you move, however for external viewers it can be very disorienting.

Compare that to this of Halo: Reach where you can get a better field perspective of what’s going on.

HTC Vive Trackers Setup

You’ll want three trackers, two for your feet and one for your waist. This is the typical setup for most Full-Body tracking situations. The trackers themselves only come with a camera mount for attachment, so it’s up to you to figure out how to attach them to your person. I went ahead and bought the TrackBelt straps from Amazon and they work well enough. Note: despite the picture, the straps DO NOT come with trackers! Each Vive Tracker currently retails for $99 USD.

It’s fairly easy to set up a single HTC Vive Tracker. It’s harder to set up 3. Kudos to this reddit user for figuring out the careful set of steps, reposted here:

• Plug in Tracker A Dongle
• Plug Tracker A into a wired USB hub that is not occupied by the Tracker A Dongle
• See SteamVR find Tracker A (it will be lit green).
• Unplug Tracker A’s wired USB cable.
• Tracker A will now be gray, but you can right-click on it
• Right click on Tracker A in SteamVR, select “pair Tracker”
• Hold button on Tracker A for 4 seconds until blue light is flashing
• Tracker will pair (sometimes can take 2-3 tries, be patient)
• Label the Tracker correctly in Steam (as left foot, right foot, or waist!)
• Turn off Tracker A (right click in SteamVR and power off Tracker. If you do it by holding the button down for six seconds ‘till it powers off you may force it to un-pair because it will cycle through the pairing process again)
• UNPLUG TRACKER A DONGLE
• Plug in Tracker B Dongle and repeat all steps with Trackers B, and then C
• Plug in all 3 Tracker Dongles, Turn on both controllers, and then power up all 3 Trackers. You should see everything lit up.

If you have lost track of Tracker A, B, etc. I recommend labeling your dongles and your trackers:

R = Right foot, L = Left food, W = Waist.

You’ll of course want to correctly label each tracker as you add them, since otherwise it’s basically impossible to distinguish which from which.

Setting up LIV

I spent a lot of time trying to get avatar setup with Beat Saber back in April 2020. At the time I was armed with only a Kinect, which provided emulated Vive trackers to try and make the feet and waist visible to Beat Saber Custom Avatar. All the results at the time were extremely poor; the avatar would twist itself/get mangled, which was just not a good look. I had to start thinking about moving soon after, so I dropped that project.

Since then, I got the HTC Vive Trackers and attempted the whole setup process again. I couldn’t figure out how to get it working with Custom Avatar out of the box, and also by that time I had spent hours trying to get them configured and pairing properly on the computer. Through some more googling, I discovered a piece of software called LIV. While I was fully expecting to have to watch some more YouTube or jump on a Discord to figure out how to install the software, I was pleasantly surprised that you could just download the software on Steam. So that’s what I did.

The guide is fairly straightforward regarding setup. You have install the LIV driver which provides more virtual trackers into SteamVR, and typically you also need to start the compositor any time you want to stream. The compositor provides a nice window separate from the game window and separate from the Steam VR mirror display. For this reason I recommend disabling the Steam VR mirror display, and to reduce the resolution of the native game mirroring if you don’t want to stress out your video card.

To pull up the LIV menu, you need to stare at and point at the LIV icon floating on the ground.

Adding and configuring an avatar is very easy with LIV. I recommend for each new play session to re-calibrate your avatar to the trackers, since you’re likely putting on the trackers slightly differently each time, and otherwise it will distort your avatar to viewers.

If you don’t want to use the stock standard boring avatars, you should go look for some online. Note that not all avatars support Full-Body Tracking, and actually if you are more savvy with 3D modeling software (e.g. Blender), there are a few guides out there, such as this one. But if you’re not into that, I recommend adding tag:FBT and tag:Full Body Tracking to refine your search results. I am currently using Sour Miku Black v2.

If you are wanting to fiddle with the LIV camera settings I recommend using a gamepad or controller.

When you’re done, you can close the LIV menu by staring and pointing at the ground again.

Streaming with StreamLabs OBS

Now when it comes to connecting all the pieces together to actually stream your VR content, you’ll need to probably get an account on a streaming website such as Twitch or YouTube. You’ll also need streaming software; I used to use OBS but more recently I’ve switched to StreamLabs which has roots in OBS. If you connect your Twitch account to StreamLabs, it will set up the output stream more or less automatically; alternatively if you have a prior OBS profile it can import those settings. If all else fails, you can refer to the Twitch Streaming Configuration Suggestions.

Within OBS or StreamLabs, you have the opportunity to set up different Scenes; this more or less shows different kinds of views on the stream, and is commonly used with Transitions for cases where you have a custom Scene for intermissions, breaks, etc. from gameplay. I am not a serious streamer, so I just use it to organize different game streaming “profiles” depending on what kind of game I want to stream. Within a Scene you can configure a number of different Sources, that can range from captured window output to custom media to custom browser-based UI widgets.

Speaking of widgets…

Setting up Beat Saber Visual Stats in Stream

This one is wholly optional and has nothing to do with Full-Body Tracking but I found it to be a really nice indicator on the stream of specifically the song and difficulty level that I was currently playing. It’s primarily powered by Beat Saber HTTP Status which exposes Beat Saber stats on a local websocket, and then from there you can host a UI that listens periodically to that port. It’s specifically very nice for browser/HTTP-based UI widgets that can be added to your favorite capture software.

Interestingly I vaguely remembered this mod being available on ModAssistant, but I couldn’t find it. However it seems at the time of this writing, the most recent version of HTTP Status supports one patch version behind the current one, so I suspect it’s not shown for those reasons. So instead you can install it manually. If that doesn’t work out of the box, it’s fairly trivial to use strings and xxd to find the Beat Saber version hardcoded in the DLL, and change it slightly to support the most recent version.

On its own, this mod doesn’t do anything, so you’ll need to find something that reads from the websocket and provides an overlay. For that I just cloned reselim’s Beat Saber Overlay, and published it via github pages. I then created a browser UI widget and pointed it to the appropriate URL, which for me is https://worldwise001.github.io/beat-saber-overlay/.

Final Thoughts

Overall this project happened over several hours of research distributed over several days; originally I embarked on this quest in April, and it took time for (a) the Kinects to arrive for testing, (b) the Vive Trackers to arrive for testing, and (c) for me to move to a different country and set up VR in a better space. But I am fairly pleased with the final results, and so I hope this post will help connect the dots of some of the articles that were scattered around the internet, or at the very least provide some entertainment value documenting my different roadblocks along the way.

Good luck and happy VRing!

PostScript: Things that didn’t work

The first thought I had was trying to figure out how to get a third-person camera view working with Beat Saber. I had seen some folks demonstrate that capability on YouTube in the past, so I knew it was possible, and I figured it was through modding of some kind since modding is fairly popular in that community.

ModAssistant

There are a few different guides to aid first-time modders, including the BSMG Wiki and Beast Saber; I used some combination of these guides. While many claim that you can technically set everything up manually, what you probably want to use is a piece of software called ModAssistant. I was remarkably surprised at how easy this was to set up. You download the latest release, and simply run the executable. It doesn’t actually install anything, so keep that executable aside from your Downloads folder if you don’t want to lose it.

I love some of the user-friendliness of this windows. Note the Game Version and the ModAssistant version tracked in the lower left. You have to click “I Agree”, which will install the basic scaffolding (probably by overwriting certain DLLs/binaries). If nothing seems to be happening, you can click on the “Mods” icon on the left hand side. That brings up something similar to the following window:

Perhaps non-intuitively, you want to click “Install or Update” which will install all the mods indicated by the Checked Boxes. It should then show the green installed version numbers. As you can tell there are some core libraries that are generally required for all the other mods; digging into the source code it seems like it provides a bunch of useful convenience methods for modders.

An important caveat is that the Beat Saber releases change semi-frequently, and this seems to be a frequent source of breakage. I don’t remember the last time I had to handle this, but if you are fine with wiping away your mods and starting fresh, you can do the following:

• Open your Steam library, and uninstall Beat Saber. Note that this will not remove the modded content.
• Navigate to your Beat Saber install directory and nuke the contents. This is usually <your steam install folder>/steamapps/common/Beat Saber.
• Open up Steam again and isntall Beat Saber.
• Repeat the above steps to reinstall mods.

Setting up CameraPlus

CameraPlus is a Beat Saber mod to configure custom cameras (specifically for popular third-person cameras) to view the player in Beat Saber. It commonly is also used in tandem with Custom Avatars. Setting up CameraPlus is only partially intuitive. It’s easy to install through ModAssistant, as shown in the screenshot above, but it does take some finagling to get the view working as you intended. Some of the gotchas I ran into include:

• The currently maintained source for CameraPlus is by Snow1226 not xyonico (the original author). Digging into commit history seems to show a change in maintainership a few times, but it’s good to see that this is still actively being maintained for bugfixes and features.
• CameraPlus uses the existing Desktop window screen to render its view. It turns out if you enable Smooth Camera, that overwrites the CameraPlus settings so you’re stuck with still a first-person view. However infuriatingly, you don’t notice until you record yourself in-game, since in the menu screens, CameraPlus appears to work perfectly.
• Configuring the camera involves watching a YouTube video. If you are discovering CameraPlus through ModAssistant and bad Google SEO, you will not easily find any of these things.

Once you have CameraPlus install, the way to configure it is from the Desktop Window of Beat Saber, not from within the VR headset. If you right-click on the window, the CameraPlus menu interface pops up and you can do things such as add/remove cameras, and switch between First-Person and Third-Person view.

If you’re not happy with the Third-Person camera view, you can put on the VR headset and move it around with your pointer. Hold the trigger to grab it, let go of the trigger to let it go.

At the end of the day, I didn’t end up using CameraPlus for my streaming since I found LIV to be slightly easier to set up. But more on that later.

Setting up Custom Avatars

The Beat Saber Custom Avatars mod is intended to pair with CameraPlus to provide a representation of your person holding the sabers in-game. Unfortunately for some reason this isn’t installed through ModAssistant, so you have to follow the instructions yourself. Note you probably don’t need to manually install DynamicOpenVR, since that is provided by ModAssistant. However you do need to download and install (read: extract the contents) of Custom Avatars in your Beat Saber install directory, i.e. usually <your steam install folder>/steamapps/common/Beat Saber.

At the time of this writing, there’s actually a bug with Custom Avatars, so check the thread and download the pre-release rc2 version to see if it fixes the bug for you (it did for me).

Once the files are in, the left screen upon startup now has a Mods Tab with an Avatars section you can configure. From there you can choose the appropriate avatar you want to “wear”. Note that the menu show Full-Body Tracking support, however the legs move really weird when we pick a Full-Body Avatar. That’s because this is only using trackers on the headset and the controllers aka hands, and so the avatar “guesses” where your feet might want to be and move them appropriately.

There’s one problem with using the Template Avatar and that’s this:

Emulating Vive Trackers with Kinect

Note: this does not use LIV, and sets up virtual vive trackers using a Kinect. It probably is not what you want!

TBD, but it’s gnarly.

Some time ago I had upgraded my work laptop from Mojave to Catalina. The result of that, as a lot of you probably know, is that due to the fact that it now only supports 64-bit applicatons this meant a lot of apps/libraries stopped working, complaining of mismatched dylibs. As a veteran Linux user, I had seen something similar many many times with mismatched .so’s, and knew the standard approach was to just reinstall everything, because with an OS version upgrade, the standard Mac OS libraries probably changed versions/ABIs causing linkage errors/other inconsistencies. Typically that meant just doing brew upgrade and stepping out for a coffee.

Python still complains

A few python things remained persistently problematic, which is why I had these scripts handy ready to blow out various parts of the environment:

#!/bin/bash

pushd ~/Library/Caches/pip && rm -rf http wheels && popd
pushd /usr/local/lib/python3.7/site-packages && rm -rf `find . -name __pycache__` && popd

#!/bin/sh

rm -rf dist/ *.egg-info/ build/ venv/ .pytest_cache/ .mypy_cache/
rm -rf `find . -name __pycache__`

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

Despite all that, I still remained flummoxed by the fact that one service I was working on locally that was using snowflake-connector-python was still quitting unexpectedly with Abort trap: 6.

What is Abort trap: 6?

Mac uses this definition of trap; this message appears when the abort() function in the C standard library is called. It is akin to SIGABRT. There are various reasons to do an abort() over doing a standard application shutdown, but typically it’s more controlled than a segfault, but deeper into library/system internals and thus out of control of the developer. An abort() in the Linux kernel for instance will cause a kernel panic.

Getting Detailed Information

Weirdly enough, when abort() is called in an application in Mac OS X, a very helpful dialog titled “$program quit unexpectedly” pops up, and it’s only through there that you can see detailed error messages, despite the fact that the application may have been called on the command line.

Either way, clicking on “Report” allowed me to dig in more into the messages to see what was the problem.

Crashed Thread:        2

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Application Specific Information:
/usr/lib/libcrypto.dylib
abort() called
Invalid dylib load. Clients should not load the unversioned libcrypto dylib as it does not have a stable ABI.

Huh. That’s odd.

Initial Googling

It turns out that I wasn’t the only one encountering this error; many other folks had encountered this, and despite being shown the proper fix, some folks had posted a workaround that consisted of overwriting the library with a symlink to a pinned version. I decided that ultimately that was pretty tacky and not a good long-term solution, since inevitably there was no consistent version to link to, folks would eventually run into the problem again.

I went to go report this as a bug to snowflake, but discovered, in fact, someone had already beat me to it:

Someone traced the potential cause to something in the oscrypto library loading openssl, causing it to crash, but that’s where my trail ended, since no one could figure out a workaround to that, especially since oscrypto was created as a split from asn1crypto.

Since by this time I had realized that there was no easy good fix, and that this was actively blocking my work, I decided that I might as well investigate how to fix this once and for all.

Tracing the source of the bug

The first step to fixing was of course, to identify the offending line of code in the dependency libraries that would trigger it. I use PyCharm as my development IDE at work, and some of its great features include both a way to click through to see original code definitions, and a really powerful graphical debugger that is very similar in usability to IntelliJ’s debugger.

However the debugger is not useful if I can’t figure out where to set possible breakpoints.

I tried tracing through from snowflake.connector.connect() but soon got bogged down since it runs through the requests library as well. I took a step away for lunch/coffee and when I came back, I realized based on the hint above that it was probably a problem in oscrypto, I looked for where oscrypto was referenced in the snowflake library and added two breakpoints; one where it was imported, and one where the function was being used.

Cool, let’s try running the debugger and see if dig further.

Woah that’s weird! It crashed on the import?

Digging into the import

After verifying that it was that one import line (by adding a third breakpoint and seeing if the next import succeeded at all), I decided to dig in further to see how that import was causing problems.

I then stumbled upon how the libcrypto libraries were being determined on Mac, and how they were being loaded, in a file called _libcrypto_cffi.py:

libcrypto_path = _backend_config().get('libcrypto_path')
if libcrypto_path is None:
    libcrypto_path = find_library('crypto')
if not libcrypto_path:
    raise LibraryNotFoundError('The library libcrypto could not be found')

try:
    vffi = FFI()
    vffi.cdef("const char *SSLeay_version(int type);")
    version_string = vffi.string(vffi.dlopen(libcrypto_path).SSLeay_version(0)).decode('utf-8')
except (AttributeError):
    vffi = FFI()
    vffi.cdef("const char *OpenSSL_version(int type);")
    version_string = vffi.string(vffi.dlopen(libcrypto_path).OpenSSL_version(0)).decode('utf-8')

is_libressl = 'LibreSSL' in version_string

Aha! This might be where we’re running into problems! Let’s try the debugger and see if we can isolate the crash:

Bingo! We isolated the location of the problem!

Filing and fixing the bug

Now that I found where the issue was, I decided to file an issue with the oscrypto project on github. But filing the bug probably was not going to mean it was going to get fixed overnight, so I decided I better investigate to see if I can find make a fix.

Since this seemed to be a Catalina-specific issue, and Apple recommended that we pin to specific versions of the libcrypto dylibs, it seemed like the obvious fix would be to do just that. So I proposed the following change:

# if we are on catalina, we want to strongly version libcrypto since unversioned libcrypto has a non-stable ABI
if sys.platform == 'darwin' and platform.mac_ver()[0].startswith('10.15') and \
        libcrypto_path.endswith('libcrypto.dylib'):
    # libcrypto.42.dylib is in libressl-2.6 which as a OpenSSL 1.0.1-compatible API
    libcrypto_path = libcrypto_path.replace('libcrypto.dylib', 'libcrypto.42.dylib')

and

# if we are on catalina, we want to strongly version libssl since unversioned libcrypto has a non-stable ABI
if sys.platform == 'darwin' and platform.mac_ver()[0].startswith('10.15') and libssl_path.endswith('libssl.dylib'):
    # libssl.44.dylib is in libressl-2.6 which as a OpenSSL 1.0.1-compatible API
    libssl_path = libssl_path.replace('libssl.dylib', 'libssl.44.dylib')

Why 42 for crypto vs 44 for ssl? As per this discussion:

$ for i in `ls libcrypto.*.dylib`; do echo -n $i:; strings $i | grep libressl | head -n1 | cut -d'/' -f9; echo; done
...
libcrypto.42.dylib:libressl-2.6
...

$ for i in `ls libssl.*.dylib`; do echo -n $i:; strings $i | grep libressl | head -n1 | cut -d'/' -f9; echo; done
...
libssl.44.dylib:libressl-2.6
...

We didn’t have a problem with libssl yet, but better safe than sorry!

Testing the fix

This was a little bit tricky, due to the fact that this was in a library pulled as a dependency. So to test this, I cloned a copy of the repo down, added my fix, and installed from that local copy by installing it as an egg:

pip install file:///Users/shh/Development/github/oscrypto#egg=oscrypto

Running the debugger again in fact confirms that the fix works!

Merge and release

As of this writing, I am happy to say that the fix was merged shortly after I made the PR, and released in v1.1.1 of oscrypto! Future users should no longer be running into this problem :).

Thank you for reading!

The typical (but not only) background is for folks to come straight from an undergraduate program. Often this is their first full-time job, or first full-time job in tech, so it is critical to set expectations and guidance from the very start:

Know your current level

What level were you hired at? This should be easy to figure out through your HR portal. Sometimes this level is public to the whole company, sometimes it’s not. There is nothing preventing you from discussing your level with others. Your level defines your salary band and official title, which may or may not be also present in said HR portal.

Knowing where you are in your salary band is a generally good indicator of what the company initially expects of you. Lower in the band probably means you’re early in the level; mid-point means you’re performing on average; high in the band means they probably expect you to level up soon.

Titles are superfluous within a company, but are important for gaining a different job outside of the company. Titles may also be affected with ladders/tiers, and so if there is a slight difference between your title and your coworker’s title, you are likely on different ladders. Asking HR what these mean is always a good idea.

Find your career ladder document

A mature company should have career ladder documents and expectations for each level clearly posted, e.g. like Square has. Once you know your level, it’s important to take a look at what the expectations are at your current level, and then what the expectations are at your next level. Many tech companies declare that they will promote you once you have demonstrated that you are performing at the next level. The way to meet those next level expectations is to then seek out opportunities that will allow you to demonstrate those skills.

Promotions aren’t automatic

I think the most intimidating thing for folks who’ve never gone through a promotion process before, is figuring out when the right time to ask for a promotion is. If you are conflict-averse but a hard worker, you may happen to have a really good manager who just files a promotion for you with very little work on your part. Unfortunately this may make it harder to figure out when the right time for a promotion is later on!

Unsurprisingly, different companies will have different paths for the employee who seeks a promotion. Some require the employee to put together the packet and present it to the manager who will forward it on. Others require the manager to put together the packet and forward it on. Yet even others might be extremely informal about the process. In all these cases, there’s generally a review panel, whether org-specific or company-wide, that will discuss review all the packets and approve who gets a promotion.

There are multiple reasons why, in a steady-state, you may not “automatically” get a promotion. Your manager may be overworked, distracted, new, dealing with other situations and is simply forgetting about how promotions are important. It may be your manager hasn’t seen enough evidence from you and doesn’t feel confident about your case. It may be there are other more critically needed promotion packets that demand their attention.

In any of these cases, even if your manager is fully supportive of your case for promotion, you should assume that your manager doesn’t have 100% visibility to your work. Doing some upfront work whether it be familiarizing yourself with the promotion process, making a hype doc, or otherwise documenting clear evidence that supports traits for level+1, will all make your manager extremely grateful during the promo period.

Speaking about visibility…

I mentioned that there may be multiple reasons why your manager doesn’t feel they have seen enough evidence. A large chunk of that is visibility of not just accomplishments, but also problems of the employee. A good manager wants to know about both wins and struggles from the employee so that they can figure out how to make the situation better/provide advice.

One of the analogies I’ve made to several managers about their work is that they’re effectively playing a Real-Time Strategy game, e.g. say Starcraft. Their day-to-day is to figure out where the logistical and management problems are, solve those, to ensure they have efficient output. They are less concerned about the fact that they are out of Vespene gas, and more concerned about why they are out of it, what is the rate of Vespene gas usage, why was it drained suddenly, and how can they ensure that rate of use is consistent so they don’t run out again. In a game it’s easy to get visibility because lots of things show rates and metrics; in the real world, they have to rely on their reports being effective communicators of potential problems before they become actual problems.

I should add also that whenever I’ve made this analogy, a lot of managers chuckle and strongly agree.

Speaking about problems…

Complaining about potential problems is okay; understanding, dissecting, and explaining how problems are affecting your work/timeline/time estimates is even better. The technique to becoming a senior engineer is not just solving harder/more complex problems; it’s being able to understand tradeoffs, potential risks to the project, and outlining possible solution avenues, ahead of time of roadblocks. This is something that only comes with practice and diligence. Coupled with effective communication, and now your manager feels confident enough to defer to you and be hands-off because you have control over the situation.

Becoming senior is becoming your own manager

There’s a reason why senior engineers often transition to full-time management, and it’s because of the assumption that they have been already doing this really well for their projects. (Whether they are actually ready to go into management is another question for another day). There is the implicit expectation that senior engineers are also good at project management, and this is often recorded as explicit line items in leveling documents, such as leading a team, facilitating discussion, estimating risk, being able to break down a large task into smaller tasks, etc. You will likely not succeed in a senior+ capacity if you do not take on rudimentary project management skills.

Write things down

It is hard to do things like, promotion evidence, project planning, root-cause analysis, trade-off analysis, etc. if things aren’t written down. The higher you go, the more complex problems become, and the easier it becomes to forget decisions made in Slack or verbally. Becoming an effective worker in tech really involves becoming an effective communicator, at the very least in a written medium. (There’s also a good reason to become an effective oral communicator, but I argue doing it in written form should be done first…. even though I’m personally not actually good at this). When things are written down, you spend less time rehashing decisions, and less time spinning wheels, and more time actually executing.

Starting out

It may seem extremely daunting to figure out how to get to senior+ when you have just graduated from school. Start simple and easy; comment on design documents and code review by asking questions (asking questions is a good exercise in forces the author to re-explain, and if you can’t understand what they’re writing, you’re probably not the only one); question assumptions, especially to dig out historical decisions, thinking, or libraries; trust your manager/team when they give you a larger-ish project, and remember that executing a project is never a singular effort, but a team one, so leverage your team. Finally feel free to propose crazy ideas; sometimes people haven’t literally thought about that option, because the older you get, the crustier you get.

Once you get into the flow of things, I guarantee things will start falling into place.

Final thoughts

Just about everyone feels terrified in their first year at a workplace. This is a normal feeling. It will go away as you build confidence. Take note of other new hires that are senior, and watch as they make the same mistakes as you. The best senior hires are those who are honest upfront about how they don’t know everything, but are willing to learn with you and listen to you.

Good luck!!! and onward!!!

Resources:

• CTFs:
  • cryptopals.org - Cryptography CTF, you don’t need a math background
  • microcorruption.com - Embedded Security CTF
  • squarectf.com - our yearly small CTF competition
• Conferences (don’t attend, just look for the talks on youtube, or read papers):
  • Industry:
    • BSidesSF
    • USENIX PEPR - Privacy Engineering, Practice and Respect
    • USENIX Enigma
    • USENIX SRECon
    • StrangeLoop
    • DefCon
    • Black Hat
  • Academic:
    • IEEE S&P
    • USENIX Security
    • SOUPS
• Podcasts or mailing lists
  • tl;dr sec
  • Security, Cryptography, Whatever
  • On The Metal
• Youtubers
  • Curious Marc
  • Lockpicking Lawyer
  • Ben Eater - build an 8-bit computer from scratch
  • 3blue1brown - animated math
  • jan misali video essays
• People:
  • Twitter has a lot of security/privacy professionals on it
    • (I have a list probably, it needs to be organized)
  • James Mickens deserves his own section:
    • This World of Ours - article
    • Not Even Close: The State of Computer Security (with slides) - video
    • Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible? A: Because Keynote Speakers Make Bad Life Decisions and Are Poor Role Models - video
  • Speaking of USENIX, I really enjoyed Alex Stamos’ keynote in 2019:
    • Tackling the Trust and Safety Crisis
• Wobsites:
  • /r/netsec wiki
  • PoC GTFO has a small collection of newsletters that do explain pretty well how to make and break things
  • Smashing The Stack For Fun And Profit Classic buffer overflow howto.
• Selected papers I enjoyed or think are foundational:
  • (there are more, I haven’t collected or organized them yet)
  • CrySP CS858 old paper reading list
  • “I’ve got nothing to hide” and other misunderstandings of privacy

This guide is to help you set up a YubiKey to sign git commits and also SSH into servers using key-based auth. It’s intended for software developers who want marginally more security and peace of mind while doing development.

If you want a more advanced/detailed howto, with references, checkout this guide instead (not mine).

Prerequisites

I assume you know the following:

• vaguely what PGP/GPG is, and some understanding of its web of trust model
• what a yubikey is
• what SSH keys are w.r.t. client authentication

I assume you have the following:

• a YubiKey 4 or better, that has never been configured with GPG
• a computer with a newish version of GPG 2
• servers that still allow RSA keys (some folks already are into ED25519 keys only)
• an existing GPG secring, ideally with a 2048 or 4096-bit RSA private key
• the command ykman (on mac, you can use homebrew and brew install ykman)

Resetting GPG on the YubiKey

(source 1) (source 2)

$ ykman openpgp reset
WARNING! This will delete all stored OpenPGP keys and data and restore factory settings? [y/N]: y
Resetting OpenPGP data, don't remove your YubiKey...
Success! All data has been cleared and default PINs are set.
PIN:         123456
Reset code:  NOT SET
Admin PIN:   12345678

$ gpg --card-edit

Reader ...........: Yubico Yubikey 4 OTP U2F CCID
Application ID ...:
Version ..........: 2.1
Manufacturer .....: Yubico
Serial number ....:
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: not forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 0 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

gpg/card> admin
Admin commands are allowed

gpg/card> passwd
gpg: OpenPGP card no.  detected

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 1
PIN changed.     

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 3
PIN changed.     

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? q

BACK UP YOUR GPG KEY

$ gpg --export-secret-keys 8A2BD353 > 8A2BD353.asc

Loading your GPG onto the YubiKey

(source 1)

$ gpg --edit-key 8A2BD353
gpg (GnuPG) 2.2.12; Copyright (C) 2018 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
sec  rsa4096/BDE438068A2BD353
     created: 2013-09-11  expires: never       usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/422DAD542FC162F9
     created: 2013-09-11  expires: never       usage: E
ssb  rsa4096/1EA455BF07A988D2
     created: 2019-07-23  expires: never       usage: A   
[ultimate] (1). Sarah Harvey <-@shh.sh>
[ultimate] (2)  Sarah Harvey <-------@cs.uwaterloo.ca>
[ultimate] (3)  Sarah Harvey <------------@gmail.com>
[ultimate] (4)  Sarah Harvey <-------@csclub.uwaterloo.ca>
[ultimate] (5)  [jpeg image of size 13232]
[ultimate] (6)  Sarah Harvey <---@squareup.com>

gpg> keytocard
Really move the primary key? (y/N) y
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1

gpg> key 1

sec  rsa4096/BDE438068A2BD353
     created: 2013-09-11  expires: never       usage: SC  
     trust: ultimate      validity: ultimate
ssb* rsa4096/422DAD542FC162F9
     created: 2013-09-11  expires: never       usage: E
ssb  rsa4096/1EA455BF07A988D2
     created: 2019-07-23  expires: never       usage: A   

gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2

gpg> key 1

gpg> key 2

sec  rsa4096/BDE438068A2BD353
     created: 2013-09-11  expires: never       usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/422DAD542FC162F9
     created: 2013-09-11  expires: never       usage: E
ssb* rsa4096/1EA455BF07A988D2
     created: 2019-07-23  expires: never       usage: A   

gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3

sec  rsa4096/BDE438068A2BD353
     created: 2013-09-11  expires: never       usage: SC  
     trust: ultimate      validity: ultimate
ssb  rsa4096/422DAD542FC162F9
     created: 2013-09-11  expires: never       usage: E
ssb* rsa4096/1EA455BF07A988D2
     created: 2019-07-23  expires: never       usage: A   

gpg> quit
Save changes? (y/N) y

Setting up Git with GPG-signed commits

(source 1) (source 2)

$ git config --global commit.gpgsign true

For good measure, kill gpg-agent and restart your terminal:

$ killall gpg-agent # forces gpg-agent to restart

Setting up SSH with GPG

(source 1) (source 2)

Add the following to ~/.bash_profile (mac-specific):

GPG_AGENT_SOCKET="$HOME/.gnupg/S.gpg-agent.ssh"
if [ ! -S $GPG_AGENT_SOCKET ]; then
  gpg-agent --use-standard-socket --daemon >/dev/null 2>&1
  export GPG_TTY=$(tty)
fi
unset SSH_AGENT_PID
export SSH_AUTH_SOCK=$GPG_AGENT_SOCKET

Add the following to ~/.gnupg/gpg-agent.conf:

default-cache-ttl 600
max-cache-ttl 7200
enable-ssh-support

For good measure, kill gpg-agent and restart your terminal:

$ killall gpg-agent

Extract your new SSH pubkey from your yubikey:

$ ssh-add -L > ~/.ssh/id_yubikey.pub

Add the contents of that pubkey to your remote server’s ~/.ssh/authorized_keys.

Note that even just 1MB of text data has the potential to generate about 300MB of nonsparse vectors, so saving the vectors directly here doesn’t work for very large text datasets. It’s more recommended to save the actual model in sparse mode (which I will save for another snippet/guide).

from nltk import bigrams
from sklearn.linear_model import LogisticRegression
import json


def vectorize_line(header, line):
    unfiltered = ['{}{}'.format(i[0], i[1]) for i in list(bigrams(line.strip()))]
    return [unfiltered.count(h) for h in header]


def vectorize_files(true_file, false_file):
    header_set = []

    true_list = []
    with open(true_file, 'r') as fp:
        for line in fp:
            true_list.append(['{}{}'.format(i[0], i[1]) for i in list(bigrams(line.strip()))])
            header_set.extend([i for i in true_list[-1]])
            header_set = list(set(header_set))

    false_list = []
    with open(false_file, 'r') as fp:
        for line in fp:
            false_list.append(['{}{}'.format(i[0], i[1]) for i in list(bigrams(line.strip()))])
            header_set.extend([i for i in false_list[-1]])
            header_set = list(set(header_set))

    header = list(header_set)

    true_vector = []
    for e in true_list:
        true_vector.append([e.count(h) for h in header])

    false_vector = []
    for e in false_list:
        false_vector.append([e.count(h) for h in header])

    X = true_vector + false_vector
    y = [1] * len(true_vector) + [0] * len(false_vector)
    return header, X, y


def load(vectors_file):
    with open(vectors_file, 'r') as fp:
        return json.load(fp)


def save(vectors_file, header, X, y):
    with open(vectors_file, 'w') as fp:
        json.dump({'h': header, 'X': X, 'y': y}, fp)


def run():
    #header, X, y = vectorize_files('emails.txt', 'not-emails.txt')
    #save('vectors.json', header, X, y)

    vectors = load('vectors.json')
    h = vectors['h']
    X = vectors['X']
    y = vectors['y']
    clf = LogisticRegression(random_state=0, solver='lbfgs', multi_class='multinomial').fit(X, y)
    print(clf.predict([vectorize_line(h, 'noreply@gmail.com')]))


if __name__ == '__main__':
    run()

Shortcut command installation

gem install grb
gem install gpr
gem install gap

Checking out a repo

git clone <repo>
cd <repo>

Updating main

git fetch origin main

Creating a new branch

grb new <user>/<project>/<feature/ticket/whatever you want>

__

git add <files>

OR

gap

OR

git add --patch

git commit -m <message>
git push

git scmp

OR

gpr

<follow instructions (add some reviewers) to create a pull request to main>

On commits and amending

In general, smaller commits (and PRs) are better. Descriptive commit messages are great; if you’re still working on something, it’s fine to go back and change the commit messages later.

git commit --amend

Rebasing from main on your branch

(do this before merging on stale PRs!)

git rebase -i origin main
git push -f

Rebasing to squash/amend/delete commits

You can squash, reorder, split, or even change the message of some commits. This is great if you had a bunch of nit commits that could all be squashed, or if you want to split a large commit into several smaller ones. The instructions are mostly straightforward…

git rebase -i HEAD~<number of commits>

Changing branches

git checkout <branch>

Holding onto temp changes without explicit commits

git stash

Note that this will stash any file changes, including ones that are unstaged. However it will not stash any untracked files.

Branch history

git log

Copying a commit from elsewhere

This is known as cherry-picking.

git cherry-pick <commit>