Pinned
It always amazes me how this is not the first thing taught in a CS class. It always makes people have Eureka moments whenever I show this to people...
- I can't explain how amazing @karpathy's lectures are. Andrej's lectures are detailed enough that I could not only follow along, but write my own tensor processing + autograd engine in Java+Kotlin & C++ from scratch. And best of all, it's 2x faster than PyTorch! SIMD for the win!
- python is actually fast when you control like 80% of the code in your call stack, move 60% to C++, micro-benchmark overhead of various library functions you don't control and build overkill acceleration datastructures like you're in CS undergrad for things you would have nuked
- Replying to @tritloAnd yet, most of UI overhead is CPU-induced driven by bad abstractions. I can assure you your GPU can render this effect just fine and will result in close to zero ms of added input delay if implemented well.
- I'm fairly convinced RL will not get us to end-to-end implementation of huge projects. Codex still has zero smell of "anticipating the future". We will likely have to revisit pre-training on *long*-running agentic data before attempting RL again. And even if we do "get there"Codex is so good, and is going to get so amazing. I am having a hard time imagining what creating software at the end of 2026 is going to look like.
- So, neural networks are REALLY robust to errors, as it turns out. Like, I just discovered a bug in my kernel caching logic, where I was computing matmul with completely wrong strides, and it STILL LEARNED. Like... WHAT WAS COMPUTED HERE IS NOT A DERIVATIVE BY ANY MEANS
- It is done. LibreCUDA can now launch CUDA kernels without relying on the proprietary CUDA runtime / driver api. It does so by communicating directly with the hardware via the ioctl "rm api" and Nvidia's QMD MMIO command queue structure.
- Perks of writing your training code by rolling your own tensorlib: 900 fps on a 4090 @ batch size 32, pytorch impl reaches 1000 fps @ batch size 512 on a 8xH100 node And yet you can use Triton and Cutlass while shipping a 4 MiB executable while still supporting AMD.
- A 256x256 matrix memory at 2048 tokens of sequence length can achieve 98.6% retrieval accuracy. With random embeddings, accuracy would saturate at 67% and with optimized embeddigs at 80%. Combined with a 4-layer LSTM decoder and a 1-layer store gate, the accuracy increases to
- Replying to @tritloif there is a platform where you should care the least about sharing memory between the GPU and the CPU, it is Apple Silicon. Traditional iGPUs have to reserve an entire chunk of physical memory dedicated to the iGPU. On Apple Silicon, both CPU and GPU can coherently negotiate
- just so you guys know the bottleneck for getting data out of Minecraft is literally FFmpeg and I can guarantee this model is both slower and worse than actually being ingame. The timer speed can be increased, frame capture can happen in game at scaled rate, and the thing that
