user avatar
Mosquito Capital
@MosquitoCapital
SRE, hype skeptic. Sometimes write at mosquito.capital
Cape Town
Joined October 2022
394
Following
10.1K
Followers
  • Pinned
    user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 15, 2023
    A year ago this week, Twitter's new owner fired a bunch of people. Together, we made some guesses on what would (or wouldn't) happen in the first year after the mass employee culling/exodus. Let's look at how we did! 1/6
    user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 20, 2022
    PREDICTIONS THREAD: Twitter Doomposting Edition "Do you think Twitter will die this week?" A lot of people have asked me that in the last 48 hours. And my answer is a pretty hard *no*, for many reasons. It's not impossible, but it's very unlikely. With that said...
    32K
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    I've seen a lot of people asking "why does everyone think Twitter is doomed?" As an SRE and sysadmin with 10+ years of industry experience, I wanted to write up a few scenarios that are real threats to the integrity of the bird site over the coming weeks.
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    1) Random hard drive fills up. You have no idea how common it is for a single hosed box to cause cascading failures across systems, even well-engineered fault-tolerant ones with active maintenance. Where's the box? What's filling it up? Who will figure that out?
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    4) Bad code push takes the site down *in a way that also fucks up the ability to push new code*. This is the absolute nightmare scenario for teams like mine. When something like this happens, it's all hands on deck. Without deep systems understanding, you might never get it back.
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    3) Bad code push takes the site down. Preventing this was my day job, and I can tell you that it's one of the scariest scenarios for any SRE team, much less a completely understaffed and burnt-out one.
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    7) Someone, say, entirely hypothetically, @wongmjane, finds a critical security vulnerability in your prod iOS app. You need to fast-track a fix, *stat*. You have a team of experts who know how to navigate Apple's Kafkaesque bureaucracy for app updates, right? I sure hope you do.
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    For context, I have seen some variant of every one of these problems pose a serious threat to a billion-user application. I've even caused a couple of the more technical ones. I've been involved with triaging or fixing even more.
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    5) Mystery SEV. Suddenly, the site goes dark. The dashboard is red. Everything seems fucked. There's no indication why. You need to call in the big guns. Teams with names that end in Foundation. Who are they? How do you call them?
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    2) Physical issue with the network takes down a DC. I gather Twitter is primarily on-prem, and I've seen what happens when a tree knocks out a critical fiber line during a big news event.
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    6) Database is fucked. It's a big one. Everything is on fire. Who's the expert for this one?
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    8) Someone notices that it's possible to read anyone else's DMs by loading up a particular URL. This is a SEV1, massive, all-hands-on-deck, critical issue. You need people who understand deeply how your privacy abstractions work, and how to fix them.
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Jun 17, 2023
    Replying to @peachlybeloved
    The SF Gay Men's Chorus photo from 1993. The men in white are the surviving original members. Each man in black is standing in for someone lost to AIDS :(
    178K
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    9) The site goes dark at 4am. The oncalls have no idea what's wrong. You *need* an IMOC (Incident Manager On Call) who knows who to wake up, why, and how. Someone who understands your systems, can synthesize information at lightning speed, and coordinate a recovery effort.
  • user avatar
    Mosquito Capital
    @MosquitoCapital
    Nov 18, 2022
    Replying to @MosquitoCapital
    12) Oh shit. You reverted one of them. Now every locked account's tweets are visible to everyone. People might literally get murdered with machetes over their posts. That's not a hypothetical. It's now 9pm. The site is fucked. Who are you going to call?