Pinned
Today we’re excited to release Muse Spark. It’s our first end-to-end test of the new stacks we’ve built at MSL, and a true testament to this incredible team. We’re eager to learn from your feedback!
Yang Song
242 posts
Yang Song
@DrYangSong
Research Principal of MSL
Joined July 2014
- Excited to share our paper on a different approach to generative modeling. We can estimate gradients of the data distribution and sample with Langevin dynamics. No adversarial method and no approximation for tractable training. Record-breaking inception score of 8.91 on CIFAR-10.
GIF - OpenAI is nothing without its people.
- Happy to announce our new work on score-based generative modeling: high quality samples, exact log-likelihoods, and controllable generation, all available through score matching and Stochastic Differential Equations (SDEs)! Paper: arxiv.org/abs/2011.13456
- Excited to share our paper on accelerating feedforward computations in ML — such as evaluating a DenseNet or sampling from autoregressive models — via parallel computing. Speedup factors are around 1.2–33 under various conditions and computation models.
GIF - Checkout my new blog post on generative modeling by score matching and score-based models. I introduce the intuition behind these methods, their pros and cons, and also discuss the close connection to diffusion probabilistic models. yang-song.github.io/blog/2021/scor…
- Applications change, but the principles are enduring. After a year's hard work led by @JCJesseLai, we are really excited to share this deep, systematic dive into the mathematical principles of diffusion models. This is a monograph we always wished we had.Tired to go back to the original papers again and again? Our monograph: a systematic and fundamental recipe you can rely on! 📘 We’re excited to release 《The Principles of Diffusion Models》— with @DrYangSong, @gimdong58085414, @mittu1204, and @StefanoErmon. It traces the core
- Do not estimate the probability density of data. Instead, estimate its gradient! We provide improved techniques for training score-based generative models, enabling effortless generation of high resolution images. Comparable quality to GANs yet no need of adversarial training!
- We explored Jacobi iteration for accelerating sequential computation in a previous work (arxiv.org/abs/2002.03629), with success in PixelCNN decoding, DenseNet evaluation, and RNN training. It's gratifying to see that an improved method can now significantly speed up LLM decoding.Introduce lookahead decoding: - a parallel decoding algo to accelerate LLM inference - w/o the need for a draft model or a data store - linearly decreases # decoding steps relative to log(FLOPs) used per decoding step. Blog: lmsys.org/blog/2023-11-2… Code: github.com/hao-ai-lab/Loo…
GIFarxiv.orgAccelerating Feedforward Computation via Parallel Nonlinear...Feedforward computation, such as evaluating a neural network or sampling from an autoregressive model, is ubiquitous in machine learning. The sequential nature of feedforward computation, however,... - Our Strategic Explorations team @OpenAI is seeking hardcore researchers to develop fundamental methodologies for advancing image and text generation. We focus on exploratory research with long term impacts. Let’s connect and discuss at NeurIPS if you are interested in joining us!
- Thrilled to share that our paper "Score-Based Generative Modeling through Stochastic Differential Equations" has won an Outstanding Paper Award at ICLR 2021! Huge shoutouts to my awesome collaborators: @jaschasd @dpkingma @studentofml @StefanoErmon @poolio!We are thrilled to announce the #ICLR2021 Outstanding Paper Awards! Out of 860 excellent papers, the award committee identified 8 that are especially noteworthy: iclr-conf.medium.com/announcing-icl… Congratulations to the authors!! @shakir_za @iatitov @aliceoh @NailaMurray @katjahofmann
- This is the most educative post on diffusion models I’ve seen—perfect for beginners. Thanks @slaterstich for the amazing work!Diffusion Without Tears is our attempt to make the score-matching + SDE interpretation of diffusion geometrically intuitive. If you're interested in our upcoming interview with @DrYangSong, I recommend reading this first! Link below.
- Releasing our paper on MintNet! It's a new flow model built by replacing normal convolutions in ResNets with masked convolutions. It has exact likelihood, fast sampling with fixed-point iteration, and better performance than published results on MNIST, CIFAR-10 and small ImageNet
- Thrilled to share our latest work on consistency models! We simplified the math behind continuous-time consistency models, stabilized their training, and scaled them up to 1.5B parameters. We are now one step closer to real-time multimodal generation!Excited to share our latest research progress (joint work with @DrYangSong ): Consistency models can now scale stably to ImageNet 512x512 with up to 1.5B parameters using a simplified algorithm, and our 2-step samples closely approach the quality of diffusion models. See more
