Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature.

Felix Dangel

Angelo Porrello Pietro Buzzega Felix Dangel Thomas Sommariva Riccardo Salami Lorenzo Bonicelli Simone Calderara Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature. 2026 February abs/2602.17385 CoRR https://doi.org/10.48550/arXiv.2602.17385 db/journals/corr/corr2602.html#abs-2602-17385 streams/journals/corr

Weronika Ormaniec Felix Dangel Sidak Pal Singh What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis. 2025 ICLR https://openreview.net/forum?id=3ddi7Uss2A conf/iclr/2025 db/conf/iclr/iclr2025.html#OrmaniecDS25 Yu Xin Li Felix Dangel Derek Tam Colin Raffel Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator. 2025 ICML https://proceedings.mlr.press/v267/li25j.html https://openreview.net/forum?id=m3zrHhiCCj conf/icml/2025 db/conf/icml/icml2025.html#LiDTR25 Marvin F. da Silva Felix Dangel Sageev Oore Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It. 2025 ICML https://proceedings.mlr.press/v267/silva25a.html https://openreview.net/forum?id=5QAKPBVdFH conf/icml/2025 db/conf/icml/icml2025.html#SilvaDO25

Felix Dangel Runa Eschenhagen Weronika Ormaniec Andres Fernandez Lukas Tatzel Agustinus Kristiadi Position: Curvature Matrices Should Be Democratized via Linear Operators. 2025 January abs/2501.19183 CoRR https://doi.org/10.48550/arXiv.2501.19183 db/journals/corr/corr2501.html#abs-2501-19183 streams/journals/corr

Wu Lin Felix Dangel Runa Eschenhagen Juhan Bae Richard E. Turner Roger B. Grosse Spectral-factorized Positive-definite Curvature Learning for NN Training. 2025 February abs/2502.06268 CoRR https://doi.org/10.48550/arXiv.2502.06268 db/journals/corr/corr2502.html#abs-2502-06268 streams/journals/corr

Marvin F. da Silva Felix Dangel Sageev Oore Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It. 2025 May abs/2505.05409 CoRR https://doi.org/10.48550/arXiv.2505.05409 db/journals/corr/corr2505.html#abs-2505-05409 streams/journals/corr

Andrés Guzmán-Cordero Felix Dangel Gil Goldshlager Marius Zeinhofer Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization. 2025 May abs/2505.12149 CoRR https://doi.org/10.48550/arXiv.2505.12149 db/journals/corr/corr2505.html#abs-2505-12149 streams/journals/corr

Felix Dangel Tim Siebert Marius Zeinhofer Andrea Walther Collapsing Taylor Mode Automatic Differentiation. 2025 May abs/2505.13644 CoRR https://doi.org/10.48550/arXiv.2505.13644 db/journals/corr/corr2505.html#abs-2505-13644 streams/journals/corr

Felix Dangel Bálint Mucsányi Tobias Weber Runa Eschenhagen Kronecker-factored Approximate Curvature (KFAC) From Scratch. 2025 July abs/2507.05127 CoRR https://doi.org/10.48550/arXiv.2507.05127 db/journals/corr/corr2507.html#abs-2507-05127 streams/journals/corr

YuXin Li Felix Dangel Derek Tam Colin Raffel Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator. 2025 July abs/2507.18807 CoRR https://doi.org/10.48550/arXiv.2507.18807 db/journals/corr/corr2507.html#abs-2507-18807 streams/journals/corr

Wu Lin Scott C. Lowe Felix Dangel Runa Eschenhagen Zikun Xu Roger B. Grosse Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization. 2025 September abs/2509.03378 CoRR https://doi.org/10.48550/arXiv.2509.03378 db/journals/corr/corr2509.html#abs-2509-03378 streams/journals/corr

Andres Fernandez Felix Dangel Philipp Hennig Frank Schneider 0001 Sketching Low-Rank Plus Diagonal Matrices. 2025 September abs/2509.23587 CoRR https://doi.org/10.48550/arXiv.2509.23587 db/journals/corr/corr2509.html#abs-2509-23587 streams/journals/corr

Mohamed Elsayed 0003 Homayoon Farrahi Felix Dangel A. Rupam Mahmood Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning. 2024 ICML https://proceedings.mlr.press/v235/elsayed24a.html https://openreview.net/forum?id=yrFUJzcTsk conf/icml/2024 db/conf/icml/icml2024.html#0003FDM24 12448-12468 Wu Lin Felix Dangel Runa Eschenhagen Juhan Bae Richard E. Turner Alireza Makhzani Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective. 2024 ICML https://proceedings.mlr.press/v235/lin24e.html https://openreview.net/forum?id=vuMD71R20q conf/icml/2024 db/conf/icml/icml2024.html#LinDEBTM24 29949-29973 Wu Lin Felix Dangel Runa Eschenhagen Kirill Neklyudov Agustinus Kristiadi Richard E. Turner Alireza Makhzani Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC. 2024 ICML https://proceedings.mlr.press/v235/lin24f.html https://openreview.net/forum?id=Y2wRKE0Qor conf/icml/2024 db/conf/icml/icml2024.html#LinDENKTM24 29974-29991 Felix Dangel Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods. 2024 NeurIPS http://papers.nips.cc/paper_files/paper/2024/hash/af45937b353f539d0a518b3bb890a01d-Abstract-Conference.html conf/nips/2024 db/conf/nips/neurips2024.html#Dangel24 Felix Dangel Johannes Müller Marius Zeinhofer Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks. 2024 NeurIPS http://papers.nips.cc/paper_files/paper/2024/hash/3d27d607586984908900eaa8ce19c96c-Abstract-Conference.html conf/nips/2024 db/conf/nips/neurips2024.html#DangelMZ24

Wu Lin Felix Dangel Runa Eschenhagen Juhan Bae Richard E. Turner Alireza Makhzani Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective. 2024 abs/2402.03496 CoRR https://doi.org/10.48550/arXiv.2402.03496 db/journals/corr/corr2402.html#abs-2402-03496

Samarth Bhatia Felix Dangel Lowering PyTorch's Memory Consumption for Selective Differentiation. 2024 abs/2404.12406 CoRR https://doi.org/10.48550/arXiv.2404.12406 db/journals/corr/corr2404.html#abs-2404-12406

Felix Dangel Johannes Müller Marius Zeinhofer Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks. 2024 abs/2405.15603 CoRR https://doi.org/10.48550/arXiv.2405.15603 db/journals/corr/corr2405.html#abs-2405-15603

Mohamed Elsayed 0003 Homayoon Farrahi Felix Dangel A. Rupam Mahmood Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning. 2024 abs/2406.03276 CoRR https://doi.org/10.48550/arXiv.2406.03276 db/journals/corr/corr2406.html#abs-2406-03276

Weronika Ormaniec Felix Dangel Sidak Pal Singh What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis. 2024 abs/2410.10986 CoRR https://doi.org/10.48550/arXiv.2410.10986 db/journals/corr/corr2410.html#abs-2410-10986 streams/journals/corr

Felix Dangel Backpropagation Beyond the Gradient. 2023 University of Tübingen, Germany https://publikationen.uni-tuebingen.de/xmlui/handle/10900/136114/ https://d-nb.info/1280233206 http://hdl.handle.net/10900/136114 https://nbn-resolving.org/urn:nbn:de:bsz:21-dspace-1361147

Felix Dangel Lukas Tatzel Philipp Hennig ViViT: Curvature Access Through The Generalized Gauss-Newton's Low-Rank Structure. 2023 2023 Trans. Mach. Learn. Res. https://openreview.net/forum?id=DzJ7JfPXkE db/journals/tmlr/tmlr2023.html#DangelTH23

Agustinus Kristiadi Felix Dangel Philipp Hennig The Geometry of Neural Nets' Parameter Spaces Under Reparametrization. 2023 NeurIPS http://papers.nips.cc/paper_files/paper/2023/hash/395371f778ebd4854b88521100af30ad-Abstract-Conference.html conf/nips/2023 db/conf/nips/neurips2023.html#KristiadiDH23

Agustinus Kristiadi Felix Dangel Philipp Hennig The Geometry of Neural Nets' Parameter Spaces Under Reparametrization. 2023 abs/2302.07384 CoRR https://doi.org/10.48550/arXiv.2302.07384 db/journals/corr/corr2302.html#abs-2302-07384

Felix Dangel Convolutions Through the Lens of Tensor Networks. 2023 abs/2307.02275 CoRR https://doi.org/10.48550/arXiv.2307.02275 db/journals/corr/corr2307.html#abs-2307-02275

Jonathan Wenger Felix Dangel Agustinus Kristiadi On the Disconnect Between Theory and Practice of Overparametrized Neural Networks. 2023 abs/2310.00137 CoRR https://doi.org/10.48550/arXiv.2310.00137 db/journals/corr/corr2310.html#abs-2310-00137

Wu Lin Felix Dangel Runa Eschenhagen Kirill Neklyudov Agustinus Kristiadi Richard E. Turner Alireza Makhzani Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC for Large Neural Nets. 2023 abs/2312.05705 CoRR https://doi.org/10.48550/arXiv.2312.05705 db/journals/corr/corr2312.html#abs-2312-05705

Frank Schneider 0001 Felix Dangel Philipp Hennig Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks. 20825-20837 2021 NeurIPS https://proceedings.neurips.cc/paper/2021/hash/ae3539867aaeec609a4260c6feb725f4-Abstract.html conf/nips/2021 db/conf/nips/neurips2021.html#SchneiderDH21

Frank Schneider 0001 Felix Dangel Philipp Hennig Cockpit: A Practical Debugging Tool for Training Deep Neural Networks. 2021 abs/2102.06604 CoRR https://arxiv.org/abs/2102.06604 db/journals/corr/corr2102.html#abs-2102-06604

Felix Dangel Lukas Tatzel Philipp Hennig ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure. 2021 abs/2106.02624 CoRR https://arxiv.org/abs/2106.02624 db/journals/corr/corr2106.html#abs-2106-02624

Felix Dangel Stefan Harmeling Philipp Hennig Modular Block-diagonal Curvature Approximations for Feedforward Architectures. 799-808 2020 AISTATS http://proceedings.mlr.press/v108/dangel20a.html conf/aistats/2020 db/conf/aistats/aistats2020.html#DangelHH20 Felix Dangel Frederik Kunstner Philipp Hennig BackPACK: Packing more into Backprop. 2020 ICLR https://openreview.net/forum?id=BJlrF24twB conf/iclr/2020 db/conf/iclr/iclr2020.html#DangelKH20

Felix Dangel Philipp Hennig A Modular Approach to Block-diagonal Hessian Approximations for Second-order Optimization Methods. 2019 abs/1902.01813 CoRR http://arxiv.org/abs/1902.01813 db/journals/corr/corr1902.html#abs-1902-01813

Felix Dangel Frederik Kunstner Philipp Hennig BackPACK: Packing more into backprop. 2019 abs/1912.10985 CoRR http://arxiv.org/abs/1912.10985 db/journals/corr/corr1912.html#abs-1912-10985

Juhan Bae Samarth Bhatia Lorenzo Bonicelli Pietro Buzzega Simone Calderara Mohamed Elsayed 0003 Runa Eschenhagen Homayoon Farrahi Andres Fernandez Gil Goldshlager Roger B. Grosse Andrés Guzmán-Cordero Stefan Harmeling Philipp Hennig Agustinus Kristiadi Frederik Kunstner Yuxin LiYuXin LiYu Xin Li Wu Lin Scott C. Lowe A. Rupam Mahmood Alireza Makhzani Bálint Mucsányi Johannes Müller Kirill Neklyudov Sageev Oore Weronika Ormaniec Angelo Porrello Colin Raffel Riccardo Salami Frank Schneider 0001 Tim Siebert Marvin F. da Silva Sidak Pal Singh Thomas Sommariva Derek Tam Lukas Tatzel Richard E. Turner Andrea Walther Tobias Weber Jonathan Wenger Zikun Xu Marius Zeinhofer