Aaron Defazio (@aaron

Aaron Defazio

1,157 posts

Aaron Defazio

@aaron_defazio

Research Scientist at Meta Superintelligence Labs working on optimization algorithms. Fundamental AI Research (FAIR) team

Manhattan, NY

Joined March 2021

Pinned
Aaron Defazio
@aaron_defazio
Aug 1, 2024
Schedule-Free Wins AlgoPerf Self-Tuning Track 🎉 I'm pleased to announce that Schedule-Free AdamW set a new SOTA for self-tuning training algorithms, besting AdamW and all other submissions by 8% overall. Try it out:
MLCommons
@MLCommons
Aug 1, 2024
@MLCommons #AlgoPerf results are in! 🏁 $50K prize competition yielded 28% faster neural net training with non-diagonal preconditioning beating Nesterov Adam. New SOTA for hyperparameter-free algorithms too! Full details in our blog. mlcommons.org/2024/08/mlc-al… #AIOptimization #AI
GitHub - facebookresearch/schedule_free: Schedule-Free Optimization in PyTorch
From github.com
124K
Aaron Defazio
@aaron_defazio
Apr 5, 2024
Schedule-Free Learning github.com/facebookresear… We have now open sourced the algorithm behind my series of mysterious plots. Each plot was either Schedule-free SGD or Adam, no other tricks!
516K
Aaron Defazio
@aaron_defazio
Mar 29, 2024
Update: more experimental results rolling in. Here it is against SGD with both the step-wise and cosine schedule (both baselines heavily tuned, no cheating) This is something special indeed!
280K
Aaron Defazio
@aaron_defazio
Jan 20, 2023
Learning Rate Free Adam? @konstmish and I have developed D-Adaptation, a method for automatically adjusting the learning rate during training that ACTUALLY WORKS IN PRACTICE on a broad range of deep learning problems. arxiv.org/abs/2301.07733 1/2
arxiv.org
Learning-Rate-Free Learning by D-Adaptation
D-Adaptation is an approach to automatically setting the learning rate which asymptotically achieves the optimal rate of convergence for minimizing convex Lipschitz functions, with no...
189K
Aaron Defazio
@aaron_defazio
Jun 5, 2025
Why do gradients increase near the end of training? Read the paper to find out! We also propose a simple fix to AdamW that keeps gradient norms better behaved throughout training. arxiv.org/abs/2506.02285
64K
Aaron Defazio
@aaron_defazio
May 28, 2024
Schedule-Free paper is up! arxiv.org/abs/2405.15682 Joint work with collaborators @alicey_ang @HarshMeh1a @konstmish @akhaledv2 @AshokCutkosky We have some strong small-scale experiments on Transformers, comparing to chinchilla-style cosine 10x reduction schedules.
288K
Aaron Defazio
@aaron_defazio
May 4, 2024
L1 regularization for sparse solutions - as usually taught - is actually terrible in practice! I’m always surprised how few people know this. To get good results, retrain with the sparsity pattern found from the initial L1 run, but without the regularizer. Works much better.
Tiago Peixoto
@tiagopeixoto
May 4, 2024
I was genuinely impressed with how bad L1 regularization really is... And it's used *everywhere* in ML as some sort of silver bullet.
217K
Aaron Defazio
@aaron_defazio
May 13, 2024
Hard at work on a NeurIPS submission
58K
Aaron Defazio
@aaron_defazio
Sep 12, 2024
The O1 release posts are unscientific — they don’t compare against previous SOTA from other labs, they don’t cite or even acknowledge previous work in the area of inference time compute. This is actively harmful to the research community, and bordering on disingenuous.
39K
Aaron Defazio
@aaron_defazio
Dec 15, 2024
Good insight. Training recovers from loss spikes because spikes occur in only a few latent dimensions.
44K
Aaron Defazio
@aaron_defazio
Mar 28, 2024
Cooking up something special! Can't wait to get a paper out so everyone can try it out. An optimizer with no extra overhead, no additional parameters. Stay tuned!
107K
Aaron Defazio
@aaron_defazio
Feb 3, 2025
The sudden loss drop when annealing the learning rate at the end of a WSD (warmup-stable-decay) schedule can be explained without relying on non-convexity or even smoothness, a new paper shows that it can be precisely predicted by theory in the convex, non-smooth setting! 1/2
44K
Aaron Defazio
@aaron_defazio
Nov 6, 2024
New 1.3 release of Schedule-Free! Includes - Improved bias correction for additional stability - Improved wrapper version of the optimizer - Some code modernization - Added a SGD reference version
GitHub - facebookresearch/schedule_free: Schedule-Free Optimization in PyTorch
From github.com
62K
Aaron Defazio
@aaron_defazio
Mar 29, 2023
Wait… OpenAI already DID wait 6 months before releasing GPT4! They wrote a whole white paper about it….
279K