Introducing Universal Simulator (UniSim), an interactive simulator of the real world.
Interactive website: universal-simulator.github.io
Paper: arxiv.org/abs/2310.06114
00:00
Sherry Yang
261 posts
Sherry Yang
@sherryyangML
Staff Research Scientist @GoogleDeepMind; Assistant Professor @NYU_Courant. Previously Post-Doc @Stanford, PhD @UCBerkeley, M.Eng. / B.S. @MIT.
Joined September 2015
- Interested in foundation models + RL? Keep an eye out for the 1st "Foundation Models for Decision Making" workshop at NeurIPS 2022: sites.google.com/view/fmdm-neur…. Call for submissions will soon follow. w. @du_yilun @jparkerholder @siddkaramcheti @IMordatch @shaneguML @ofirnachum
- Review paper on Foundation Models for Decision Making: arxiv.org/abs/2303.04129 Foundation models can characterize various components of decision making, such as states (S), behaviors (A), dynamics (T), task specifiers (R), through generative modeling or representation learning.
- Video generation can serve as world models and embodied planning tools, but they must be grounded in the physical world. Check out: VideoAgent for self-improving video generation using feedback from VLMs and action executions. Paper: arxiv.org/abs/2410.10076 Code:
- Checkout UniMat -- a unified representation of materials that enables scaling of diffusion models to millions of stable crystal structures. Website: unified-materials.github.io Paper: arxiv.org/abs/2311.09235
00:00 - Consider joining our team at Google DeepMind to work on foundation models for decision making, e.g., foundation model alignment, reasoning, planning, simulation, and optimization with foundation models.Our team (w/Dale, @daibond_alpha, @mengjiao_yang + others) at Google DeepMind is looking to hire. If you are interested in foundation models+decision making, and making real-world impact through Gemini and cloud solutions, please consider applying through boards.greenhouse.io/deepmind/jobs/…
- Video generation will revolutionize decision making in the physical world like how language models have changed the digital world. Interested in the implications of video generation models like UniSim and Sora? Check out our position paper:
- Happy to share that UniSim universal-simulator.github.io was selected for an Outstanding Paper Award at #ICLR2024. Check out the oral presentation today at 10:30am Oral 1B and poster on Wed at 4:30-6:30pm #87. Thanks to the award committee @eunsolc, @katjahofmann, @liu_mingyu,Announcing the #ICLR2024 Outstanding Paper Awards: blog.iclr.cc/2024/05/06/icl… Shoutout to the awards committee: @eunsolc, @katjahofmann, @liu_mingyu, @nanjiang_cs, @guennemann, @optiML, @tkipf, @CevherLIONS
- Source for this figure: arxiv.org/abs/2205.10816. Procedure Cloning is a simple but powerful idea: Teach the model not just what action to take but also the procedure for how to find this action. Original Tweet: x.com/mengjiao_yang/…My Bet: Strawberry is algorithm distillation/procedural cloning. Everyone right now is coming up with ways to distill System 2 into System 1, but that will always be limited. We need to train the model to run the algorithms, not just outputs (and post-train with RL of course).
- What does "Learn principles, not formulas. Understand, do not memorize” mean for autonomous agents? Chain of Thought Imitation with Procedure Cloning! ArXiv arxiv.org/abs/2205.10816 Code github.com/google-researc… Site sites.google.com/view/procedure… w/ Dale @pabbeel @ofirnachum
GIF - At #NeurIPS2024. I'll present generative-materials.github.io, and talk about generative simulators, world modeling, and video agent at the D3S3 (d3s3workshop.github.io), SSL (sslneurips2024.github.io), and Open-World Agents workshops. I'm recruiting PhD students this application cycle.
- Text-conditioned video generation can serve as universal policies (UniPi) and learn from sim, real, and web-scale videos. w/ @du_yilun, @hanjundai, @daibond_alpha, @ofirnachum, Josh, Dale, @pabbeel Paper: arxiv.org/abs/2302.00111 Web: universal-policy.github.io
- Checkout Generative Hierarchical Materials Search (GenMS) – a framework for generating crystal structures from natural language. Website: generative-materials.github.io Paper: arxiv.org/abs/2409.06762
00:00 - As video foundation models reach billions of parameters, how to adapt them to task-specific settings (e.g., animation, robotics) without access to the model weights becomes a pressing issue. We introduce Video Adapter: arxiv.org/abs/2306.01872 video-adapter.github.io
00:00
