Weight decay is truly evil. looks worse for the first 400k steps and then suddenly overtakes on the home stretch.. But I can't even be mad, we've been warned about exactly this https://t.co/bQYreV7MPh
