Pinned
Our NeurIPS paper is published on arXiv.
In this paper, we propose a new optimizer ADOPT, which converges better than Adam in both theory and practice.
You can use ADOPT by just replacing one line in your code.
arxiv.org/abs/2411.02853
Shohei Taniguchi
1,453 posts
Shohei Taniguchi
@ishohei220
Joined March 2015
- NeurIPSใฎ่ซๆใarXivใซไธใใพใใ๏ผAdamใซ่ปฝๅพฎใชไฟฎๆญฃใๅ ใใใใจใง๏ผใใคใใฉใซไพๅญใใใซๅธธใซๅๆใไฟ่จผใงใใใใจใ็คบใใ่ซๆใงใ๏ผๆๆกๆณใฎADOPTใฏ๏ผใณใผใใ1่กๅคใใใฐใใใซไฝฟใใใฎใง๏ผใใฒไฝฟใฃใฆ่ฆใฆใใ ใใ๏ผOur NeurIPS paper is published on arXiv. In this paper, we propose a new optimizer ADOPT, which converges better than Adam in both theory and practice. You can use ADOPT by just replacing one line in your code. arxiv.org/abs/2411.02853
- slideshare.net/ShoheiTaniguchโฆ ใใกใ่ฌ็พฉในใฉใคใใๅ ฌ้ใใพใใใใกใ็ฌฌ5ๅใฎ่ฌๅธซใๆ ๅฝใใใใจใซใชใใพใใใๅผทๅๅญฆ็ฟใจ็ขบ็ๆจ่ซ (ใใคใบ็ตฑ่จ) ใฎ้ขไฟใPOMDPใซใใใๅผทๅๅญฆ็ฟๆๆณ (e.g., ไธ็ใขใใซ) ใซใคใใฆ่ฉฑใใพใใ่ๅณใใไบบใฏใใฒๅฟๅใใฆใใ ใใ๏ผ็ท ๅใฏๆฅๆๆฅใฎ23:59ใงใใ
- ไธป่ใNeurIPSใซ้ใใพใใใ Adamใฎๅๆๆงใฎ่งฃๆใจใใใใซๅบใฅใใขใซใดใชใบใ ใฎๆน่ฏใๆๆกใใ่ซๆใงใใ ๆๆกๆณใฎADOPTใฏๅน ๅบใใฟในใฏใงAdamใใใใๆง่ฝใ็คบใใLLMใฎ่จ็ทดใงใใญในใฎในใใคใฏใๆใใใใใใจใ็ขบ่ชใใฆใใพใใ ่ฟใใใกใซarXivใซใไธใใไบๅฎใงใใๅฝ็ ็ฉถๅฎคใฎ่ซๆใNeurIPS 2024ใซ๏ผไปถๆก้ฒใใใพใใใ weblab.t.u-tokyo.ac.jp/%e5%bd%93%e7%aโฆ
- ๅ่ซใจใจใใซๅใ็ตใใงใใPyTorchใจPixyzใซใใDeepMindใฎGQNใฎๅ็พๅฎ่ฃ ใๅ ฌ้ใใพใใใ่่ ใฎEslamiใใใใ็ดๆฅใในใฆใฎใใคใใฉใๆใใฆใใใ ใใใฎใงใ่ซๆใซๅฟ ๅฎใชๅฎ่ฃ ใซใชใฃใฆใใพใใ่ๅณใฎใใๆนใฏใฉใใใ(ใชใGPU4ๆใฏๅฟ ้ ใงใ) github.com/iShohei220/torโฆ github.com/masa-su/pixyzoโฆ
- OMRON SINIC XใงใฎใคใณใฟใผใณใฎๆๆใICRA2020ใซ้ใใพใใใๆๆ็ฉไฝใ็ฐๅขใจๆฅ่งฆใใ้ใฎๅนพไฝ็ใชๅถ็ดใ็จใใฆ็ฒๅญใใฃใซใฟใๆดๆฐใใใใจใงใใญใใใใใณใๅ ใฎ็ฉไฝๅงฟๅขใๆจๅฎใใ็ ็ฉถใงใใใใคใบรใญใใใๅถๅพกใซ่ๅณใใใไบบใฏใใฒ่ชญใใงใฟใฆใใ ใใใผ
- PFNใฎใคใณใฟใผใณ้ใฃใใผ
- ๅผใฉใใใใชใผใใณใฝใผในใฎLLMใๅบใพใใ ็พๆ็นใงๆฅๆฌ่ชๅฏพๅฟใฎใชใผใใณใฝใผในใงๆๅคง่ฆๆจกใชใฎใงใใใใใไฝฟใฃใฆ้ใใงใใ ใใ100ๅใใฉใกใผใฟใตใคใบใปๆฅ่ฑ2ใถๅฝ่ชๅฏพๅฟใฎๅคง่ฆๆจก่จ่ชใขใใซโWeblab-10Bโใใชใผใใณใฝใผในใงๅ ฌ้ใใพใใใใชใผใใณใฝใผในใฎๆฅๆฌ่ชๅคง่ฆๆจก่จ่ชใขใใซใงๆ้ซๆฐดๆบใงใใ๏ผJGLUE่ฉไพกใงใฎๅฎ็ธพ๏ผ2023ๅนด8ๆ16ๆฅๆ็น๏ผ๏ผbit.ly/47zPTJiReaders added contextใใฎใใฌในใชใชใผในใฏใOpen Source Initiativeใๅฎ็พฉใใไธ่ฌ็ใช "Open Source" ใจใฏ็ฐใชใๅฎ็พฉใงใใชใผใใณใฝใผในใใจ่กจ็คบใใใฆใใใใจใซๆณจๆใๅฟ ่ฆใงใใ opensource.org/faq/#commercial 2023ๅนด8ๆ19ๆฅ็พๅจใWeblab-10BใฏHugging FaceไธใซใใใฆCC BY-NC 4.0ใฉใคใปใณในใงๅ ฌ้ใใใฆใใพใใ huggingface.co/matsuo-lab/webโฆ huggingface.co/matsuo-lab/webโฆ CC BY-NC 4.0ใฉใคใปใณในใฏๅ็จๅฉ็จใ็ฆใใใใฎใงใ OSI Approved Licenses ใฎใชในใใซใฏๆฒ่ผใใใฆใใพใใใ opensource.org/licenses/
- ไธป่่ซๆใNeurIPS 2022ใซๆกๆใใใพใใ VAEใฎใใใชใจใณใณใผใใไฝฟใฃใๆจ่ซใMCMCใซใๅใๅ ฅใใใใจใใ็ ็ฉถใงใ ใใใไฝฟใฃใๆฐใใๆทฑๅฑค็ๆใขใใซ Langevin autoencoderใๆๆกใใพใใOur paper โLangevin Autoencoders for Learning Deep Latent Variable Modelsโ has been accepted at NeurIPS 2022๐ We proposed a novel framework of deep generative models named the Langevin autoencoder (LAE). Brief summary in the thread below. arxiv.org/abs/2209.07036
- ๅฅจๅฑใใใพใใๅฝ็ ็ฉถๅฎคไฟฎๅฃซ๏ผๅนดใฎ่ฐทๅฃๅฐๅนณใใใฎ็บ่กจใใ2019ๅนดๅบฆไบบๅทฅ็ฅ่ฝๅญฆไผ ๅ จๅฝๅคงไผๅญฆ็ๅฅจๅฑ่ณใซ้ธใฐใใพใใใ weblab.t.u-tokyo.ac.jp/%e5%bd%93%e7%aโฆ
- ใใกใ็ฌฌ5ๅใฎ่ฌๅธซใๆ ๅฝใใใใจใซใชใใพใใใๅผทๅๅญฆ็ฟใจ็ขบ็ๆจ่ซ (ใใคใบ็ตฑ่จ) ใฎ้ขไฟใPOMDPใซใใใๅผทๅๅญฆ็ฟๆๆณ (e.g., ไธ็ใขใใซ) ใซใคใใฆ่ฉฑใใพใใ่ๅณใใไบบใฏใใฒๅฟๅใใฆใใ ใใ๏ผ็ท ๅใฏๆฅๆๆฅใฎ23:59ใงใใใๅญฆ็้ๅฎ๏ผ็ญๆ่ฌๅบง็ฌฌ1ๅผพใๅผทๅๅญฆ็ฟ่ฌๅบงใฎๅ้้ๅง๏ผ8/11ใใๅ จ6ๅใฎ่ฌๅบงใงใใๅผทๅๅญฆ็ฟใฎๅบ็คใใใsim2realใๆจกๅฃๅญฆ็ฟใControl as Inferenceใไธ็ใขใใซใชใฉใใซใใผใใพใใๆทฑๅฑคๅญฆ็ฟใฎๅบ็คใ็่งฃใใฆใใๅญฆ็ใใใฏใใฒใๅฟๅใ๏ผ๏ผ7ๆ26ๆฅ 23:59็ท ใๅใ๏ผ deeplearning.jp/reinforcement_โฆ
- **Update on the ADOPT optimizer** To address several reports that ADOPT sometimes gets unstable, a minor modification has been made to the algorithm. We observe that this modification greatly improves stability in many cases.Our NeurIPS paper is published on arXiv. In this paper, we propose a new optimizer ADOPT, which converges better than Adam in both theory and practice. You can use ADOPT by just replacing one line in your code. arxiv.org/abs/2411.02853
- Our paper โLangevin Autoencoders for Learning Deep Latent Variable Modelsโ has been accepted at NeurIPS 2022๐ We proposed a novel framework of deep generative models named the Langevin autoencoder (LAE). Brief summary in the thread below. arxiv.org/abs/2209.07036
- ๅนฃ็ M2ใฎๅค็ฐใใใ็ด่ฟ1ๅนดใงICLRใICMLใNeurIPSใซใในใฆ็ญ้ ญ่่ ใง่ซๆ้ใใฆใใฆใใพใใงใใใๆขๅญใฎๆทฑๅฑคๅผทๅๅญฆ็ฟใฎใขใซใดใชใบใ ใๅ้กใใ่ซๆใNeurIPS2021ใซๆกๆใใใพใใ๏ผๆพๅฐพ็ ใ@Tdash_Kozใใใ @shaneguMLใใใฎๅ ฑๅ็ ็ฉถใฎๆๆใงใ arxiv.org/abs/2103.17258
