Author: Memorymancer

An attempt to put down in words another axis which has been bothering me, while also conveying next to zero information.

A grossly oversimplified model of a day in the life of a researcher is that you have enough compute for N experiments, each of which will take T time. Additionally, you are asked to produce updates every interval of U time. I am aware that for fixed N and T, you can usually run a single experiment for time somewhere in [T/N, T], but I will be ignoring this fact for now.

During my PhD life was simple. Roughly speaking, I had enough compute to run 1 experiment at a time, which took a week. I had weekly meetings with my advisor, but he would frequently have admin duties and other things, so on average I probably met him once every two weeks. He didn’t ever ask for updates between meetings. This means, N = 1, T = 1 week, U = 2 weeks. 

As a “typical” week, you could imagine that Sunday I code up something to run and launch it in the evening. Monday morning I wake up and something has gone wrong so I fix it and relaunch it Monday afternoon. Monday evening I check and everything is going well. Tuesday morning, I do some preliminary analysis. Wednesday, I finish up the analysis scripts and just wait on the experiment to complete and come up with 2 new ideas for what to do next. Thursday I get lunch with a friend and get another idea. Friday, I plan out a flow chart of what I will do depending on how the experiment goes and analyze the partial results. Saturday I take off and maybe read a few cool papers my friend sent me. Sunday I restart the cycle.

There are multiple ways to interpret the above. One is wow! I’m working all the time! But another is wow! I was doing absolutely no work! Like look carefully at the above — there’s like no work being done! During my PhD, I leaned toward the former interpretation. Nowadays, I lean towards the latter. In hindsight, while I did some work, for the most part, I got lunch with friends, went to the gym, pulled silly pranks, and just generally had a lot of fun. Life was not perfect. I had my own struggles. But it is also true that the PhD was probably the “happiest” time of my life so far, whatever that means. And this is coming from someone who literally slept at the office.

Jumping back to the present, I have previously talked at length about the “external” pressures in my life now. There is also another axis of pressure which I probably won’t be writing about. But the third is this “internal” axis of pressure. You can imagine some values of N, T, and U that would make life quite difficult. The one line summary of my “internal” struggles is that I am overclocked.

I can already hear the jeers. Skill issue! And the answer is yes! It is 100% a skill issue. That being said — how does help me in the slightest? Since I joined (and speaking more broadly, since I left my PhD), I have significantly improved my clock speed. Unfortunately (or maybe fortunately), N, T, and U are not exactly static quantities either, and furthermore it’s not even clear that I am improving faster than the numbers are changing.

A part of me wants to moan and complain that if I had told my previous self that I would be SUCCESSFULLY CLOCKING AT N’, T’, and U’ (the numbers where I am able to handle) in July 2025, I would be absolutely amazed. But what does that matter? By old standards, N, T and U are insane. But take a look around. Everything is insane now.

In a different world, my course of action would be clear. I would call up several people, lay out exactly the situation I was in, including both the precise numbers and the exact things I was doing, and get their advice on a very low level about what do do. I have no doubt that I would glean 10+ things to improve on which would help things a lot. Unfortunately, this is not possible in the current world.

So what to do? I think there are three options (nonexclusive).

1. Talk to my coworkers about this in more depth. I have obviously done this to some degree, but I can do it more.
2. Think really hard myself about the above and what to do.
3. Ask an LLM for advice.

There are a lot of secondary effects of the above. A very simple example that I have already observed is that because of the pain of overclocking and falling short, it’s very tempting to choose {easy, unimportant} experiments over {hard, important ones}. If you think about this with System 2, you can often override the instinct. But when things are already overclocked, sometimes you just make a gut decision and it’s very, very, very easy to do this. I have done this many times.

People always say the important thing is asking the right questions, but I actually think that in multiple axes of my life, I am already asking the right questions. I simply also need the answers too. In any case, it is my privilege to live in this era. Sure, there is agony in having to face the mirror and see clear as daylight that you fall short and furthermore where. As a PhD student you can pretend to yourself that you are “doing your best” whatever that means. Of course, I wasn’t. But I could lie to myself. Now I can’t even do that. It is agony too to have to suffer largely alone when I know that many could and would help if I could but ask. But this is 2025. So I better figure it out quick!

Coda

When I first arrived here I got dinner with [redacted]. In the middle of eating, he apologized before suddenly pulling out his laptop to check on his [redacted size] experiment. At the time, I thought it perfectly reasonable (and still do)! It was an enormous experiment, and this stuff really matters! Fast forward 1.5 years, I launch experiments of that size basically everyday and so (probably) does he. Times have changed. Things are truly not the same as they once were.

A one month follow up.

A few days after I wrote Asteroid Defense, I got hit by a surprise asteroid attack. It was very intense. Nevertheless, in an era of agony, I remember that day with great pleasure. Ironic right? You see, I had taken that Monday off — by sheer luck — to sprint like a madman on repairing my asteroid defense station. That surprise attack would have absolutely decimated me just a few days earlier. But as things stood, I was left without a scratch. And so that day felt like catching the last chopper out of Saigon.

And so things are much better now. They are not perfect. In fact, by objective standards they are still quite dire, just not quite as dire as they were before. Because there has been at least one day where I felt like I could stand defiant against the Fates and whisper you were too late.

The truth is I have spent far too much time agonizing over my mistakes, my bad strokes of luck. When you gamble, you sometimes lose. This is part of the game. There is no such thing as a straight shot to victory in this era. You must learn to survive the drawdowns or you will not survive.

And my eyes, which I am slowly learning to trust more and more, tell me that serious meteor showers lie ahead. This time though, they will not be a surprise. The first small clumps of snow have begun to slide loose off of the mountaintop. It is now not a matter of if, but instead when the avalanche comes. I do not know when, where, which direction. All I can say is that it is clear that the center cannot hold in its current state. And so things will break sometime, somewhere, somehow. And unlike the meteors that hit in early 2025 where I can protest (to who?) that I did not see them coming, I believe that even my repaired systems will be taxed to the brink in the next few months. 

This means there are two things that are important to do.

1. The asteroid defense system is repaired. It has never been stronger in my life. It’s going to need to get a whole lot stronger. I have some ideas for how, and I’ll think of more as I implement them the first batch.
2. Not unrelated to the above, it’s time to start preparing some reserves. This is not a “get to it when you get to it situation.” I cannot currently afford a serious drawdown, and I think I might need it.

Here’s a simple change. I already ignore almost all news that doesn’t directly impact me. That isn’t enough. For many reasonable definitions of impact, I wake up to multiple pieces of news that directly impact me every week. The new rule is that unless I intend to change my NEXT F***ING ACTIONS, it doesn’t count as directly affecting me. So if I feel affected, I will ask myself what do you intend to change because of it? If the answer is nothing, then I will ignore it. Backtesting this on the last six months suggests it would have saved me a lot of heartache at very little cost. I think it also is one of the first steps in wresting control back into my own life.

Finally, it’s clear that the most important diff I need to make in my life is that I need to be happy. How is unclear though. At work every morning, I plan out the most important things to do for the day and at the top of the list for the last month has been a “P0: Be Happy.”

There are two reasons. One is that life is just painful when you are not happy. But the second is that there are domino effects on other parts of my life. The second reason alone necessitates appearing happy alone, but a better way is to genuinely appreciate the privilege of living in this era, this life, this time. It’s what I dreamed of. Really.

I imagine a conversation with my old self, as a child. I’m telling him every agonizing detail of my life, everything going wrong. He seems to brighten up with every word I say, no matter how bad it is. Even his tears seem to be tears of joy. I imagine myself asking him: This life kind of sucks. Why are you so happy I’m living it. And he would say: NO! Have you forgotten?

For as a child, I dreamed to live a life like this, and maybe I should have dreamed of something else, but there is no doubt in my mind then or now that this life I have now is both exactly what I dreamed of and better than the life I had then. Of course, I dreamed of winning too, but that is a separate thing entirely. This is the journey I chose — I would have cried tears of joy to have undergone it.

And there is a secret third reason. It’s been a while since I felt like my old self. Much of my power is gated behind it. I’m excited to get that power back. And I think in the coming months I will. For I will need it.

As many of you know, the last six months have been incredibly agonizing for me. I remain proud of many things. On the macro side, I think everything went basically right. If you gave me (right now) a chance to undo any macro decision I made in the last six months, I would undo exactly 0 of them. On the other hand, if you look at the micro day-to-day life—the one that I actually have to live through—my f***ing goodness.

I’m going to analogize my life as a planetary asteroid defense system. Asteroids (events that rock me—heh) hit at pseudo-random intervals. These are not truly random, but they are not predictable by any power that I possess, though occasionally I get a bit of warning.

The first warnings started arriving in January. The system was stressed, but I wouldn’t even say it was overloaded. By February, multiple substations had gone down. By March, I started truly panicking. By April, I had taken critical damage. Using a concrete metric, this meant I felt like screaming multiple times a day, and I spent the majority of my energy trying to avoid only the most serious of mistakes. I personally find it miraculous that I made 0 irreversibly bad decisions during this time, and indeed managed to squeeze out a few good ones too. May was tremendously better, mostly because I decided enough was enough. But also because the pressure let up a bit too.

Here we are in June. I have improved considerably. To put a number on it, roughly 80% of my normal subsystems are offline, but the core 20% is more or less running. Using the classic 80/20 rule, this means that I’m at roughly 80% of normal capacity, which feels about right.

1. Protect the Defense Station First

Historically, I would often let asteroids hit me rather than other points on the planet because my recovery rate greatly exceeded the asteroid hit rate, and healing the defense station (me) was often far easier than healing someone else’s damage (mostly outside my control). In that era, this was probably correct. In this era, it is certainly not. There’s a saying about how when the oxygen masks drop on an airplane, you need to put yours on first—irrespective of selfishness, you won’t be useful to anyone if you’re passed out from lack of oxygen.

In the last six months, I’d estimate that 30% of all the trouble was some variation of seeing people around me, many of whom I cared tremendously about, going through negative spirals. Ironically, this includes myself. The feeling of helplessness, being unable to save everyone (including myself), and being forced to ration out who I could help, was overwhelming. And because I did not ration perfectly (or in fact, even rationally), every part of my life suffered. I would frequently get angry at someone else for messing up because the energy I spent on them made it so I could not support someone else I cared about. Frankly, a much of the issue was that I had some illusion that I could be a heroic white knight—an illusion I of course knew was false but still pretended—but now it’s irrevocably shattered. Pathetic now that I write it out honestly, but true.

Yes, of course, AI progress is partly to blame, but I don’t think it’s fair to attribute it all to this, as at minimum, it’s all second-order. The effect roughly goes: AI progress -> lots of money, power and opportunity to be seized -> existing equilibria shattered. And of course you can see what happens from there.

This trouble is a bit different than my past bouts of unhappiness in that it isn’t because I massively f***ed up in the normal sense, it’s because I need a better asteroid defense system. This is still a f*** up! There is a saying you cannot save everyone, and usually, the emphasis is on “everyone.” But in 2025, the truth is that YOU cannot save everyone. Everyone can be saved, but not by YOU.

The immediate action item to the above is that you need to let some asteroids crash onto the planet, in important regions too. Saying no is hard for me, even harder if I’m unsure. That’s part of the pain. Sometimes you say no and wish you said yes, but you can’t live life that way. You must pay, one way or another.

2. Be Patient

One big issue with recoveries like this is that enough of my subsystems are down that I find myself short-circuited even when planning how to bring them back up. This leads to negative spirals that have to be caught and dealt with, slowing down the healing process. Ordinarily, this would be fine, but not while I’m actively defend against a meteor shower. A similar spiral occurs with missed opportunities. I will see an opportunity, instinctively reach out to grab it, and then fail because a subsystem is broken. This makes me furious at the thing that is responsible for breaking the subsystem (which is often myself).

I think there is a lesson here, which is that you should be really, really, really, really careful about damaging yourself to the point where the recovery systems short out, but that one is kind of obvious. And the second lesson is to make more robust recovery systems, but that’s for a later time. But the third is to be patient. If you’re playing a video game, you will often get a quest that says you need to obtain XYZ item, but then you discover that before that you need to uncover a rune at ABC location but before that you need to slay a dragon at IJK and to do that you need a certain piece of armor that is only unlocked at DEF level and so on. On one hand, that could be viewed as frustrating, but honestly that’s just the game. This is just life. In order to recover my substations, I need to carefully fix them one by one and some substations will require fixing others first.

I recall when, as a young child, I asked my old Go teacher once where I made a mistake in this game. He said — where to begin? I asked him to point out every single flaw in any of the moves I made (I was probably not even 1 dan at the time). He responded — if you want me to be frank, I can find an error in almost every move you make. And he was right! With Go AI’s you can actually check this nowadays. And yea, almost every move I make is wrong, even now, let alone then. There is no point in obsessing over every minute mistake. Some if it doubtless comes from growing up in an environment where people attributed a single small slip to a deep character flaw. And so I internalize each mistake as a problem with myself. If you repeatedly do the wrong thing over and over again upon reflection, that is an important flaw to correct. If you slip once and it’s not a critical error, I need to learn to forgive myself. It really doesn’t matter that much. It’s 2025. Perfection is not the standard.

So be patient. And do not tally. 

Coda

[redacted] occasionally says that this period of time is a brief glimpse of what it might feel like to live during the singularity. With respect to the particular events above, while obviously not unrelated to what is going on in AI, I think it would be a stretch to put too much of the blame on rapid AI takeoff. But in another sense — what does it matter? The feeling of living in the singularity is one in which you find it very difficult to adapt to the increasing pace of change and this I certainly felt. At much lower stakes, at much slower speeds, and (knocking on wood that I don’t jinx it) for only a brief, agonizing month did it feel truly almost out of control. But it was a glimpse. With the real thing, I don’t think that’s how it plays out. And if I live to see those days, I hope I’m ready for it.

When everyone disagrees with me, I find it much easier to stick my head out for something I believe in. When everyone agrees with me, I find myself feeling nervous. This is because the “energy” boost I get from being a contrarian is gone.

Take my recent project, for instance. When everyone opposed me, I was extremely headstrong in arguing in favor and trying to gain support. Survival instinct, really. I remembered those days, when I earnestly thought to myself that the main thing stopping me was getting the right resources. In a bout of good fortune, the winds have suddenly shifted, largely due to factors completely outside my control. Now, I find myself with the full support of everyone that matters, and nevertheless I find myself exceedingly nervous. There are no more excuses I can make to myself. Oh sure, I can always ask for another 10x GPUs, or better infra, or a more experienced team. But I know in my heart of hearts, that what I have been given is likely enough, if placed in the hands of someone who knows how to execute.

But can I execute? Here I am, staring down the gun, the clock ticking, feeling unsure of my own abilities. 

I suppose it’s always this way. There comes a moment in every journey, big or small, where the preparation is done, and execution is all that remains. Despite what the stories tell us, this part is the hardest. The Miami Heat fighting tooth and nail to get to the 2023 NBA finals. Shackleton, setting off from Elephant Island, for one last shot at survival.

In such moments, there isn’t really much to it really. All you can do is play.

27 lessons, mostly for my past self, though perhaps they may also be of use to you. Too often you learn the secrets of how to easily beat a level only after you have beaten that level. Ah well, that’s life.

1. Tempo matters. In games like Magic the Gathering or Hearthstone, a fundamental notion is that of tempo, which roughly captures the notion of how long you have to “setup” before you must actively begin winning the game. In a fast meta, games end in a few turns. In slower, metas, it becomes possible to play more expensive cards without losing on turn 3. In life, as in such games, you must get the tempo right. Go the right speed, whatever it may be. Even if that means throwing away gorgeous opportunities that simply aren’t good for the current tempo.
2. Outcomes are correlated. The rising tide lifts all boats (in your area). If I had to name a single thing that Chinese-American culture gets most wrong, it is this. Many of our previous generation believed at their core a philosophy of “there are a certain number of slots for success that you have to fight for.” I think this is basically wrong even for college admissions, which is all that the culture seems to actually care about. However, in large part, this is incorrect in America. In fact, it’s not even all that clear to me if it is actually true in China anymore either, but I digress. It is essential to unlearn this lesson as an American. Because the truth is: the more friends you get ahead, the more likely you are to get ahead yourself.
3. Consensus != Truth. Consensus is easy to find. Just go around and talk. Consensus will find you. However, one must not mistake consensus for truth, as much as she may appear to be. Many make this mistake and nowhere more than in San Francisco.
4. Be extra nice to admins. They can make your life trivial. They can make your life hell. They are often very vengeful. Even when they f*** with you, react with utmost charity and grace. Many people are very kind. I’ve had my a** saved more times than I can count by an admin bending the rules for me.
5. Hype sometimes wins. Some things can hit escape velocity before the law of gravity brings them down to Earth. It would be nice to believe the Warren Buffet approach to life where you only build upon things that are stable and fundamentally sound, but I think he’s incorrect. You may correctly observe that something is fake and just being hyped up. Does that mean it won’t succeed? I’m still young, but as far as my eyes can tell, that answer is a big fat NO.
6. People saying negative things about you behind your back matters a lot less than I thought. There is nothing you can do about it. There is nothing you should do about it. Good to keep a pulse about how much and who, but don’t freak out. If there is a wound to fear, it is that nobody talks about you at all.
7. With effort, there is nothing you cannot learn. This applies to things that you have natural affinity for (e.g. Go), as well as things that you have natural disaffinity for (e.g. stretching). It might be too late (or just impossible genetically) for you to reach the top 0.1% of any discipline. But hitting the top 10% is doable for almost any domain if you optimize. And hitting the top 1% is also doable for many domains that you aren’t totally not cut out for. This is easy to say but difficult to absorb on a fundamental level.
8. Double down on your winners, don’t fret about your losers. The counterpoint of the previous maxim is that you should focus on the things you do best — not because you cannot do other things, but because you might as well use whatever comparative advantage you have. I’ve probably spent far more time and effort into running than lifting, and yet I am far better at lifting than I am running. Sharp with a few weaknesses is much better than mediocre all around. It’s a power law world.
9. Shoot and Miss More. When you play games with power law scaling, obviously you cannot play optimally. No one can. The best thing you can do is take a lot of good shots on goal.
10. Beware the Regret Minimization Demon. If you don’t live with the reality that you didn’t choose optimally with the benefit of hindsight, you are probably shooting down the counterfactuals, either in your head or worse yet, in reality. I take more mental damage than most, it seems, regarding this. Things never go perfect in the counterfactuals either, I must constantly remind myself. Keep shooting. Keep missing. And never ever tell yourself that you could have never hit that shot anyways when you clearly could, just because it dulls the pain of missing. Because before you know it, you’ll stop shooting to spare yourself the regret of having tried.
11. Advice (and change) often must be changed in sets. Rejected name: advice is not convex. Let’s make this simple. Chocolate is delightful; avocado is too. However, chocolate avocado is not. More generally, a lot of advice MUST be bundled with other advice and people often neglect to say this. Take something in this piece: “shoot and miss more”. Do not take this advice in a vacuum. It may not end up well for you in the same way that “be extra nice to admins” will. The latter piece of advice is probably almost always positive value. The former piece of advice is far more important, but tricky to get right. I don’t know the precise conditions. If I did, I would fix it in myself first. 
12. Don’t listen to your parents. As a child, I was chided by everyone older than me that they regretted not listening to their parents more in their youth. And so I kept listening to my own parents long after I suspected that I should stop. This was a big mistake. So here’s a lone voice calling out to the void. While it is certainly true that the average kid listens to their parents too little, it is also occasionally true that a kid listens to his parents far too much.
13. Listen to the wisdom of the elders. Yet if there’s one thing I overcorrected for in my early twenties, it is that I not only stopped listening to my parents, I largely disregarded the advice of anyone who was not my age. This was a mistake. There are many things that can basically only be learned by experience. I’m still finding my way, but I’m much better at discerning when to listen to youth and when to listen to experience.
14. Only someone who knows you exceptionally well can give you reliably good dating advice. Everything else is a shotgun. I find it very difficult to convey this in a few sentences, despite how much I’ve thought about it and how important it is. The main lesson is that almost all rules / advice about dating is wrong. This is one example where I wish I had listened to the elders who told me so, but I did not understand until I heard enough stories that more or less forced me to believe. Though what is right I still cannot say! To my early twenties self! As painful as it was at the time, twenty-seven year old [memorymancer] is sitting here grinning at you. Thanks for all the lessons!
15. Mammon is not easy to understand. Money is both more and less important than I thought it was at eighteen. The Millions Don’t Matter was the one-liner of what I believed. If I had to summarize, I think I got the direction correct, but the details wrong. Better than the other way around of course, but the details matter too. I’m but 27. I will be taught many more lessons by Mammon before it’s all said and done.
16. Tread cautiously when speaking of money. Many people hold very strong feelings about money and for good reason too. The annual expenditures of people I know on housing range from O(1k) to O(100K). And I’m restricting to people I know in just four cities: NYC, Boston, SF, and London. Sadly, I don’t actually seem to have as many friends outside of those four cities as I would have hoped. What is cheap to one person might be extraordinarily expensive to another. Regarding my current apartment, it’s legitimately unclear whether it’s cheap or expensive. Depends on who you ask, I suppose.
17. Money is fungible. Saving 5 dollars on housing saves you the same amount of money as 5 dollars on clothing. Same for spending. This is one of those lessons that sounds inane but is actually quite profound. Once you internalize it, many obvious fixes present themselves. If you think you believe it, ask yourself how your spending patterns differ from the people around you. If the answer is: not very, then you have not internalized it (or you live among highly optimized peers). Nevertheless, to follow the previous lesson, I will not say anything further.
18. Use objective metrics if possible, especially when trying to measure system failure. I have a checklist of six things to measure my health. They are: 1) [redacted] 2) how much I squat 3) what time I sleep / wake 4) how many calories consumed in last 24 hours 5) whether I can touch my toes 6) when was the last time you played video games. I am very rarely at 6/6, but it’s easy to tell what’s wrong and how to fix it. Each of these has objective measurements and very simple fixes. Just do them. It’s very difficult to measure system failure using non objective metrics.
19. Routines are underrated. And it wouldn’t do to say my parents were wrong on everything. Here’s one example. My mother has been telling me for two decades on the importance of routines and I have failed to listen and failed even more to implement what I’ve heard. But they matter. They ironically don’t matter for many of the things that I was told they mattered for. But they matter for getting in sustained reps towards whatever you wish to achieve. Want to join the 1000 lb club? Lift every other day. Want to get better at coding? Write code everyday. Routines matter.
20. Hobbies can be dangerous. If one is not careful, they can easily consume your life. This applies doubly to hobbies that people often praise in my circles (skiing, climbing, traveling, fine dining, etc.), for the social gradient pushes you towards addiction. Whether this is your wish is up to you. But it is not what I wish (right now at least). 
21. Plan, don’t dream. Dreams are things you hope for. Goals are things you plan for. It isn’t a goal until you have a step by step process of how you intend to get there. It may not be the way you actually get there (and probably isn’t), but it’s a start. Until you have a concrete plan of attack, it’s a pipe dream, not a goal.
22. FOMO is very dangerous. [redacted] is far better than me about this. I certainly feel FOMO quite hard, although the primary effect on me thus far seems to be primarily an energy drain rather than poor decision making. It goes back to Jensen’s famous quip about high expectations. FOMO can only be felt by someone who, whether they admit it or not, believes that they deserve the world. Don’t do things because of FOMO. Do things because you believe in them.
23. Things Change. And that’s OK. I was reading my (unfinished) version of this for my 26th birthday and was quite intrigued to see that there is almost no overlap in what I’m thinking about. At this point, basically everything on that list I both 1) internalized and 2) hadn’t thought about in what felt like years. After all, it was only like 8 months ago that I was bumbling around in Boston and decided to hard commit pivot away from RL/game theory and back into my original field of language modeling. What a world.
24. Rules Change. Learn and relearn. I wanted to say something about rules in particular. Some of this is that rules always change with time. But perhaps more importantly, the rules change every time you “solve” a level and move to the next one. This applies everywhere: video games, dating, AI, work. “What got you here won’t get you there.” I spent (spend) too much time bemoaning how I wasn’t taught the rules of the game by my parents. The truth is: except for a select few, no one ever is. No use complaining. Just get to learning.
25. Stories are Important. Rothfuss tells us that“It’s as if everyone tells a story about themselves inside their heads. Always. All the time. That story makes us who we are. We build ourselves out of that story.” I thought understood this at 17, but I didn’t. Never forget it. You tell the story. So tell the story you want to tell.
26. Risk is mostly underrated. Certainly over the last year I should have taken more risk. But I guess the real advice does not sit in a vacuum, as I have just said. The real advice is: prepare yourself to be able to take as much risk as you can handle, and then take that risk when the opportunity comes.
27. Guard your spirit. There is one lesson above all else. Over the years I have seen far too many friends lose their spirit. Everyone frets about what they perceive they lack. Intelligence, social capital, height, looks, money: these are all of secondary importance — at best. What matters is your spirit. A small part of this comes from within, but most of it is given — by friends, family, your partner, words of the long dead and occasionally living. I am saddened to see this, but in my circles it is common to attack people and sap their spirit. Almost always these attacks are from those who were given much towards those who were given little. “NGMI.” “Just doesn’t have what it takes.” “You can tell who will succeed when you see them.” I often feel enraged at the thought. It’s not fair. But it’s also a terror that people will be right when they say it about me. So cherish what has been given to you and guard your spirit at all costs. Aside from a precious few whose light was taken from them far too early, there is time aplenty to run the full race — if you can somehow manage to keep running.

The last two months have been a rollercoaster. Here is a quick summary.

Life

• I’ve went on quite a few adventures. To tell the story of one of them like Kvothe likes to tell them: I scaled a 9 story building without a rope.
• My parents have decided to separate.
• [This update has been redacted because it is still in development. I note with some bemusement that a decade from now, it might end up the most important of anything on this list.]
• I decided to leave my PhD (technically a temporary pause, but we all know what that probably means). 5% chance I still complete my PhD.

Work

• I took an internship on the POET team. I flew out 1.5 weeks after signing my offer — before my background check had even cleared. I have been in the Bay Area since November.
• I recently decided to join DRAGON as a researcher.
• I decided not to help found BOOKER, a company that’s being started by a bunch of friends.
• I have become extremely “Olympiad-pilled.” I now believe that both IOI and IMO gold are within the reach of AI very soon and that deep reasoning capabilities in LLMs may be right around the corner. To be very concrete, I think there is a 50% chance AI gets an IMO gold medal within 2.5 years (very long right tail though). I note with some irony that the 2024 IMO will be held in the United Kingdom.
  • Like at Cana, I saved the best for last. My decision making over the last two months has largely revolved around this belief.
  • A side effect of this is that I no longer think that near term AGI is impossible. I nevertheless still think it is quite unlikely.

Of course, this is about as accurate of a summary of my life as “KKR buys RJR Nabisco” is an accurate summary of “Barbarians at the Gate.” The final destination is correct but you missed all the action in the middle. I am still processing everything (and likely will be processing everything for a while), but here is a quick list of assorted thoughts. 

• You cannot take your time to think through everything. In life, as it is in real time strategy games, choosing to think is an action in itself. This is the stronger version of the “not choosing is still a choice”. I do not know the proper balance. All I know is that the last few months, for the first time in my life, I repeatedly made decisions on wildly incomplete information under extreme time (and other) pressure. And while I am far, very far, from sure I made the right calls, I think I can say with fairly high confidence that I do know what the wrong call would have been: carefully taking the time to think through everything in full depth. I am also reasonably happy with the choices that I have made. I think that, as of today, they are the most likely to have been the right choices. Only time will tell what really is the case.
• [redacted] warned me — correctly — that the amount of slack available in industry for AI is at an all time low. Projects will get culled if you don’t show results within months at most, sometimes even less. Only in academia can you explore a topic for years without achieving anything. He was 100% correct. Ironically, this quip, designed to convince me to stay, is one of the reasons I decided to leave. I think industry takes it a bit too far, but some correction is in order for academia too. If you did not re-evaluate your life decisions as a result of Nov 30, 2022, then I am going to claim that you were incorrect. Too much changed on that day and the days that followed to ignore and continue on whatever path you were going, unless that path was somehow exactly correct.
• The “regret minimization” demon seems to be particularly strong these days. Even restricting to just 2023, I declined to be early at several relatively successful startups. I don’t regret that. There are three reasons. First and most importantly, one cannot regret such decisions. For if you do, you will go insane. Secondly, being early at a startup is a terrible, terrible, terrible deal from a risk/reward ratio. Truly awful. The only reason it might be good is the learning experience (which ironically might make the whole thing worth it). Third, I think the opportunities will come. But not joining BOOKER, I will confess there is a deep, deep fear that I will come to regret this decision. I suppose there is no escaping this. No matter what I chose, there would have remained a deep fear that I was choosing wrong. And in some sense, that is all I could hope for. For if you escape the regret minimization demon by sabotaging your own options, who are you really hurting?
  • Interlude: From a financial perspective, I think you want to be employee ~100 at a firm that gives you long exercise windows. Stay at the firms you think are doing well. Aggressively leave the firms that you think are doing poorly. You can hit 10 one-year cliffs in a decade of work. And that is assuming that somehow you struck dirt 9 times. At such companies, you have the ultimate inside information. 100 person startups are not large enough to conceal the important information. You should be able to get exceptionally strong signal about whether the company is doing well. [redacted] shared revenue information with employees while I was there, but even at ones that don’t — you can find it out. Worst case, count the number/size of your customers and do a back of the envelope calculation yourself. But really, all you need to look at is the vibe of the company to know. I think the risk is only slightly higher than that of working at a big company like Google, and your expected compensation, so long as you play your cards right and are not exceptionally unlucky, is roughly 10x. It’s in some sense the reverse of being an early employee, where you take arguable more risk than the founder and get substantially less equity (failed founders fare better than failed early employees).
• Money is one of the most ancient of powers. Every year I come to learn something about this old god that I did not understand before and look with complete bewilderment upon my past self, who thought that I understood him well. “I was here long before you were born, and I will remain long after you pass away” Mammon sneers at me. How could I have thought that I understood something so complex? This time, I learned a lesson that may be obvious to many of you. People are complex. Some people you ignore whatever they are telling you and watch where they put their money. Other people you entirely ignore where their money is and pay attention to what they are telling you. Most people you do something in the middle. In the Silicon Valley, people care almost as much about money as they do in New York finance circles. But people don’t admit it, especially to themselves. In New York, when one speaks about money, it is common knowledge what signals are being conveyed. But in San Francisco, it is not. Sometimes you can catch a glimpse of someone’s heart when they themselves do not, or perhaps more accurately will not, see it themselves. 
• I flip back and forth about whether the PhD was the right decision. I wasn’t even sure at the time, but you have to make choices regardless. If you had asked me a few months back, I would have told you that its was a terrible decision but I was staying because continuing (with +2 years already in) was better than quitting (and starting from scratch). The reverse of sunk cost fallacy, or so I would argue — you could call it “collecting the sunken benefits.” So I think it is especially ironic that at this moment in 2024, I am quitting, but now think the PhD was more or less the right decision (or at least reasonable?). The difference from six months ago is that I have found a soft landing. I suspect that it’s still far too early to tell though. Ask me again in a decade and I might know, and I emphasize the might.
• Research and product have opposite mindsets. I think this is the root of why so few teams (depending on your metric, perhaps literally 0) have solved both product and research. In product, you need speed, rapid iteration from user feedback, to change course on a moment’s notice from the faintest amounts of feedback. You can’t think your way through. You must act and re-evaluate at every step. You need the opposite of hardheaded conviction. In research, you will receive very little feedback until the end of each “cycle” of experiments. Of course, one exceptionally important skill is minimizing the length of time of that cycle and to fail fast, but no matter what, there are some things that you must see through in order to know whether they work or not. In research, you need to buy yourself as much runway as possible, prepare all you can, but nothing will work until the end — if you are lucky. You must have the conviction you are going on the right path with little to no confirmation that you are until you have completed the journey. Of course, I exaggerate, but not by too much. You can throw away ideas if you find reasons that they will not work within a week. But you cannot throw away ideas simply because they failed to succeed within a week — they almost never will, even if you are sitting on a gold mine. If you want a specific example, imagine the year is 2010 and you are trying out convolutional neural networks for image recognition. You give it one week and if you don’t get good results you are quitting for another hot topic. What are you finding? Absolutely f***ing nothing. You can say the same story for literally every important advancement in AI. I don’t think there is even a single exception (maaaaaaaybe adversarial examples). So perhaps a wiser version of me might have known that BOOKER would likely never have gone down that path that I hoped — the game was rigged from the start, even if none of the players knew that it was.

P.S. I was originally going to write a 2023 reflection, as is quite popular these days I hear. But I took a dump of everything that happened in 2023 and really all the interesting stuff was in the last two months. Here’s the rest of it.

• I pivoted my research off of multi-agent reinforcement learning and into LLMs and search, which is actually a lot less big of a jump than it might seem at first glance. Probably one of the smaller pivots in AI these days, actually. 
• I joined the 1000 lb club (separate day with straps though). I joined moments before my life blew up from the above, so I’m leaving a clean version with no straps + same day for 2024.
• I got a great fellowship that funds the rest of my PhD, along with a boatload of compute. I find this quite ironic.
• My advisor got promoted to [redacted].
• [redacted due to privacy reasons, but probably quite significant I would say.]
• I went clubbing for the first time in NYC. I also went clubbing for the second time in Las Vegas during the last two months, but that was not important enough to make the previous list. 
• I learned how to fold dumplings and got a lot better at cooking.
• I read 0 books in 2023, breaking an almost decade long streak of averaging over a book a month. What a year, right?

A wise man once told me that — contrary to popular belief that time is our most limited resource, most people actually run out of energy before they run out of time. As a boisterous young eighteen-year old, I did not believe him. But I think now I do.

Let’s begin the clock with COVID. During COVID, I was extremely plentiful on time, but very low on energy. I personally don’t blame long COVID, though perhaps it’s something to consider. I just blame plain old depression, my old foe. In 2021, I recovered significantly, but I would not consider myself recovered to pre-COVID levels until mid 2022 or so. Productivity increase from trough to mid-2022 / pre COVID was probably 10x. I truly did not do too much those two years other than write (and go on adventures I suppose). That’s the one thing that I actually do quite well when depressed / low on energy. If I can’t work, might as well have fun!

In the past year, things have gone quite well from a productivity / career standpoint. Well, I guess relative to the world, I have lost ground, but that is not at all because I have gone slower, but rather because the world seems to have hit the gas pedal like crazy. Often I worry that I’m falling behind the curve, but in truth, there is nothing much you can do about that. One can ask no more than to improve over yourself every day. There is still high variance in my daily productivity (working on this), but as a whole, I think I haven’t been more productive in my life (not saying too much heh). Nevertheless, in the last few months, I still feel as if I tire long before the hours falter. So here’s some reflection on trying to solve this, mainly on the idea of trading off time for energy.

Energy Sinks

• Mildly uncomfortable conversations. I lose an enormous amount of energy from this, and very disproportionately. My best suspicion is that my internal energy state looks more like an allergic reaction. I sense uncomfortableness and then I automatically prepare the “full response” for a total meltdown conversation. And then because <1% of these conversations become meltdowns, it’s a huge overreaction and I’m completely drained for no reason. One can quite naturally trace the origins of this reaction, but once again, this is another one of those habits which has outlived its usefulness.
  • Fixes: I’m going to reduce the number of mildly uncomfortable conversations by making more Irish exits. Somewhat separately, I have noticed I lack the skill to gracefully exit conversations. Something to work on. Secondly, I do think it’s important to protect against this energy drain. Running from uncomfortable conversations is not a bad first step (I currently have very minimal fear that me — of all people me! — will run away from too many uncomfortable conversations), but a second step is to build up defenses against energy drain. The next time I’m in an uncomfortable conversation that for whatever reason I am not evading, I will work on expending the proper amount of energy, not too low, but importantly, not too high either.

• Devastation in the World. This is quite related to the above, but different enough that it deserves its own point. I find myself quite demoralized by bad events in the world. Our generation comes of age at the dusk of the Pax Americana. For many years, we were shielded from the terrors of the world due to no fact other than our citizenship / location of residence. This shield is still here, but it’s clearly weakening by the day. In many senses, feeling bad about the world and doing nothing is a strict loss. And for many of these things, there is nothing we can do, but live as best we can, for there are powers beyond any of us at work
  • Fixes: Read less news. Think less about things that have nothing to do with you and that you cannot change. And perhaps most importantly, understand — at the gut level — that this world is far from perfect, that many things are still very ugly, that there is much work to be done to reach the light.

• LiChess / Instagram / Video Games. These are time drains, but that’s not the important part. They are energy drains. I have never once felt excited to go debug my code after playing a few LiChess games. Never. I notice I’m slightly mentally tired, and then LiChess just drains the rest of the energy away. Instagram and video games are even worse. Instagram has some slightly additional negative effects, but at the same time, there are actually some gains I’ve had, so I don’t think I *currently* regret using Instagram overall. On the marginal level though, I suspect too much additional instagram use is definitely going to be negative.
  • Fix (?): I’m not sure what the fix here is except to reduce the time. Ideally I’d like to replace it with something, but it isn’t that clear that there is an activity that fits. Reading does not work — it occupies a different zone entirely in the energy curve. The first things that come to mind are: TV, cooking, or yoga. I’m not particularly confident that these are good ideas / sustainable.

• Heavy Days at the Gym. I have been going to the gym a lot more recently, and have learned that many, many, many things that are often repeated about the gym are quite incorrect. In any case, most of those are for another time. What I have noticed though, is that heavy days at the gym are absolutely draining in terms of energy. Many a day has come where I have left the gym at 8PM and failed to do anything before going to sleep at 2AM. I was so tired that I could not even muster up the discipline needed to sleep on time (separate topic).
  • Fix: You do need heavy days to progress — one cannot improve very fast without occasionally going at least close to your limits. But I will be quite careful of how I expend energy in such cases.

Energy Sources

The high level point is that I need to protect my energy before it gets too low, or else often times I lack the critical amount of energy to engage in any of these tradeoffs.

• Adventures. Adventures are the primary way I trade-off time and energy. They are usually enormous boosts in energy at the price of only moderate costs of time. I should do this more. And instead of thinking about them like a luxury cost: I should think about them as a converter between two resources. 

• Traveling. Traveling is often a source of stress for many people. I think at this point though, I have enough experience that it’s positive. If I lived in London, I would tell myself to just book a weekend trip to a random place in Europe every single time I felt bored on Thursday. This is substantially less plausible in Boston. But I think the analogue is to just book a train ticket to NYC whenever I feel like it. Again, trade off time for energy. This one isn’t quite as efficient as adventures, but then again, what is?

• Eating regularly. A bad habit I have from antiquity is skipping meals. I won’t say I have gained nothing from this because I have. But this is one of those habits that may have been quite useful at some point but has overstayed its welcome. Sometimes the answer for why I have no energy is simply because I haven’t eaten a good meal in 24 hours. This is an easy fix. Goal: eat two good meals a day, at relatively consistent hours. 

• Light day at the gym. I previously stated that heavy days at the gym are an enormous energy sink (only time energy drops to ~0 frankly). In contrast, a light day at the gym is a very good energy boost. A light day is the same as a heavy day, except you only do ~80% of the weight. I’m actually now at the level that I think this might be good anyways. Of course, to optimize your gains, you want to do mostly heavy days with an occasional light day. But maybe it’s actually right for me at this point to do mostly light days, with an occasional heavy day. In any case, my previous attitude was that light days were a waste of time. But when considering energy, it’s a different story altogether. After finishing up my current goals, I will significantly reduce the number of heavy days at the gym.
  
• Sleep. They say sleep is essential and I’m sure it is. Nevertheless, I have personally found an inverse correlation between how much I sleep and how much energy I have. There is some indication that this is not unbelievable, even if it isn’t the most common occurrence. I’m going to try the recommendation in the link. I’m going to try sleeping 7 hours and gradually increasing the time. We’ll see if this is better or worse. Waking up too early is an energy drain. It doesn’t happen too much though, so I’m not so worried. But there is an enormous correlation between waking up at a reasonable hour and feeling good. It’s both a cause and an effect as fas as I can tell. 

• Hiking + Outdoors. Growing up I did not like the outdoors. I have now realized that this is primarily because I did not like going outdoors with my parents. I quite like the outdoors. Living in Utah with [redacted] is frankly, one of the best memories I have —— all time. Harvard has a wonderful outdoors club, which I am going to go to more. Bought hiking shoes last week!

Miscellaneous Thoughts

• Way out of my depth, but anecdotally I have heard that some people become much more productive after having children (obviously this is not true for everyone). Here’s a wild guess: maybe they were extremely energy bottlenecked but had a lot of extra time (even if they didn’t realize it). When they had children, they get a huge energy boost (as children often do) and this outweighs the time sink. 

• A related point. [redacted] and I have long discussed whether it’s worth it to spend time playing Go. I think I have >50% of becoming US National Champion in Go if I spent 1K total hours studying. That’s not very much time. But the energy cost is high. If there is some low energy way of studying and improving, I would love to hear suggestions. My current intended strategy is to grind endgames against the AI. This is not a generally good strategy — it’s adapted to my strengths and weaknesses. Overall, it’s probably still right to do game reviews with AI + life & death practice problems. 

The price we paid was the price men have always paid for achieving a paradise in this life. We went soft, we lost our edge.

Paul Atreides

My father has always chided me for my weakness to the cold. Whereas I was born and raised under the California sunshine, he spent his youth amidst the brutal winters of Beijing, Michigan and Canada. The man does not fear the cold.

The ancients braved the harshness of nature, a mistress fickle enough to change from freezing to scorching—sometimes over a single day. The modern man lives in controlled environments and argues over the proper definition of room temperature. While I am glad that few now die from exposure to the elements, there is also something lost, some antifragility we no longer possess, when the modern world coddles us from the way our ancestors used to live. It almost feels like modernity is giving us the physical equivalent of a drug that restricts our emotions. No more spur-of-the-moment murders executed in a fit of rage, no more soul-crushing depressive episodes triggered by the loss of a loved one, no more contagious outbreaks of fear when we hear about the latest atrocity via social media. But something is lost all the same.

Last year, I had the choice of wintering in either Hawaii or Boston. I chose Boston, in no small part to reinitiate a quest to imitate my father’s strengths. Many years ago, I failed on a similar quest. I stood outside one evening in a t-shirt and shorts for as long as I could bear. There was no increment, no progression, no learning curve. One day I was bundled up in a Patagonia; the next I faced the chilly winter air alone. I quit after only a few minutes and caught a cold a few days later. This time around I resolved to be smarter. My weapon of choice: cold showers.

If you trust the Internet, cold showers are nothing short of modern-day magic [1]. They build mental strength, ward off sickness and depression, lull you to sleep, improve your stoicism—even help you become more alpha. You won’t see too many other things praised by both Reddit and the tech elite [1]. Whether or not these rumors are true I cannot say. My life is too volatile for me to notice any small changes: the effect is washed out in all the noise. I do not really put much trust in advice read on the Internet anyways [2] and personally have not noticed any major changes in my life that I can confidently attribute to cold showers. Except for cold resistance that is.

Some of you may know that I recently took a plunge into the Charles. There were a few sleights of hand there that made the activity much safer than it might appear, but the biggest one is that I had trained extensively beforehand. Nowadays, I regularly take near-freezing showers without breaking a sweat. The Charles was colder (and longer) than my usual routine, but not by all that much. I knew I could do it.

If you want to try this for yourself, I would not recommend starting by turning the temperature all the way down. Influenced by [redacted], I am now warming up to the idea that improvement does not have be difficult. Pain is not the unit of effort. Your goal is not to improve in the first few days; it’s just to get started. Push too hard at the beginning, and you may feel a lingering dread that causes you to hesitate when you try again. You don’t want that. The failure mode for these kinds of things is that you lose interest in them, not that you aren’t “strong” enough to achieve your goals [3]. Make it really, really easy at the beginning—so easy that you cannot possibly fail. Only after you have established a routine should you begin to push.

My procedure was quite simple. I started with lukewarm showers while I was still in Texas. Over time, I turned the knob down slowly until it hit the lowest setting. In Austin, this was only around 70F or so. In Boston, it’s 45F (I measured it). Done incrementally, it’s honestly kind of … easy. I rarely ever forced it. There was no schedule, no goal for how long I stayed in the shower, not even any sort of obligation to take one every day. If I felt a little bit sick or otherwise uninterested in feeling cold that day, I just take a regular hot shower. No big deal and plus, risking any extra chance of illness in the modern era is probably unwise. Not every day brought improvement. Somedays I even felt like I took a few steps backwards. But slowly, I would notice that it would get easier and easier, until I could bear the cold for longer than I thought possible.

I am very happy with my training. I’m not quite ready to run a half-marathon barefoot in the snow yet, but now I can see how such a thing might be possible with sufficient training. Last November, I set out towards the Columbia River bundled in multiple layers and still had to turn back early because I felt unpleasantly cold. Now, I run along (and sometimes in) the Charles in the middle of winter wearing a t-shirt and shorts, and I feel great. Improvement doesn’t get much starker than that.

[1]: Gamestonk!! is the other thing that comes to mind

[2] I recall a certain story I heard in my youth, reminding me to view with a grain of salt any “secrets to success” spoken by a person who was not successful. I relate it below for your enjoyment.

Avi was enjoying a smoke one afternoon outside his apartment when a middle aged man interrupted him.

Zvi: Nice cigar you got there.

Avi: Eh? Oh this little lady. Used to smoke those $10 pieces of s*** from Malaysia but after I retired I said, f*** it. Life is short. Might as well enjoy the good stuff while I still can eh?

Zvi: And just how often do you get to enjoy your cigars? 

Avi: One or two a day, ever since I got back from Vietnam. Never miss a day, not even that time they had to dig some leftover shrapnel out of my knee back in ’98. 

Zvi: Oh no way, you were there too?  That’s a long time. Nixon pulled me out in what? ’69? He came to—

Avi: That son of a b**** left me there until 1973! 

Zvi: Still almost half a century. Man, that’s like what, a quarter million you spent on cigars over the years? You could have bought a Lamborghini with all that money!

Avi: *glares at Zvi*

Avi: Do you smoke?

Zvi: No

Avi: Then where’s your f***ing Lambo?

[3]: The most common reason I hear people try to take cold showers is improving “willpower”.To be honest, I’m not sure I really buy this line of thought. The whole point of my personal training with cold showers is that it’s only hard at the beginning, and if you take my incremental approach it’s not even hard then. In fact, you can almost say that this is the entire point: train until cold showers are no longer “cold.”

People often ask me why I studied economics as an undergrad if I planned to go into AI research. The full answer is long and complex, with a healthy dosage of improvisation and questionable decisions along the way. But here is a short answer.

If you complain that I did not spend enough time studying computer science, economics should not be your primary target. After all, I spent the majority of freshman year reading philosophy, history, and classical literature via a residential “Great Books” program. Economics is obviously not directly relevant to AI, but surely it’s closer than studying Rousseau, Plato, or Confucius! Because majors minimally constrain coursework at my university (and I was able to waive a decent number of electives with math/CS classes and a term abroad), I likely spent as much time studying the “Great Books” in college as I did on all my history and policy Economics classes combined.

This dilly-dallying did not cut into machine learning, which I studied as deeply, if not more deeply than most of my fellow classmates who were interested in AI research. What I did not study was the other parts of computer science. I have almost no understanding of how operating systems, databases, compilers, networks, or programming languages work. Not that I think that they are unimportant. On the contrary, I’ve always been curious to know what magic occurs between hitting compile and seeing the results of my code scribblings. I studied economics (and philosophy/literature/history) as opposed to more traditional topics in computer science because I think is they are understudied and undervalued by people in computer science. We computer scientists often snub our noses at other fields, believing that our field contains the most important secrets of the world. This arrogance pervades all areas of academia but has crippled computer science more severely than most (don’t even get me started about mathematics though). I tend to disagree.

My impression is that taking a single introductory class made me more knowledgeable in game theory than 99+% of AI researchers. I still can’t quite wrap my head around this fact, as game theory has extremely important applications in robustness, exploitability, adversarial environments, reinforcement learning, and even generative models. It feels almost too *free*, but then again, research is nowhere close to an efficient market. A unique background means that I know things that no one else in my cohort does. Furthermore, it ensures that my shortcomings are precisely the things that everyone else knows best. In a competitive environment, this would doom me. But in a collaborative environment? I like to think I’m not altogether terrible at working with other people, and as a result, my strength is that I can pair up with almost anyone and augment our combined strength to far beyond what normal collaborations can achieve, as our strengths are complementary instead of just duplicated. I need someone to cover my weaknesses but luckily my weaknesses are precisely what everyone else knows exactly how to do.

This arrangement does not work if everyone comes from a unique background (what would unique even mean in that case?), but that is not in any danger of happening in computer science. Until then, I’ve got my own little monopoly.