Pinned
Realized I never posted this: I gave a talk at ReInvent this year about our multimodal-language-action model and the infra needed to train it.
link 👇
Jim Bohnslav
7,153 posts
Jim Bohnslav
@jbohnslav
post-training VLAs @zoox
- MFW I finally have to learn RL to train some reasoning models
- > be me, training vlms > use hf transformers because who wants to reimplement models if they don't have to > low MFU > `pytorch_profiler.py` > ViT takes 40X longer than the LLM > sus > .item() in the forward pass makes cudaStreamSynchronize every attention layer > MFW
GIF - Happy to present the first paper from my PhD! biorxiv: biorxiv.org/content/10.110… code: github.com/jbohnslav/deep… Install now using `pip install deepethogram`! (1/N)
- bytedance calling me GPU poor A model trained for 665,000 H100 hours is called "cost efficient", "moderate computational resources"
- I rarely do this, but this is an absolutely useless benchmark. The examples are not problems I care if my computer vision system can solve. I'm going to treat ZeroBench performance as an anti signal that someone made an overfit test taking VLM. I bet the next Phi will do great.Is computer vision “solved”? Not yet Current models score 0% on ZeroBench 🧵1/6
- Replying to @iScienceLuvrI haven't found any good model to write greentexts since
- SimDINO: simplifying DINO/ DINOv2. They use L2 loss on global <-> local features and a penalty on feature covariance to prevent collapse. With these additions, they can get rid of a lot of the bells and whistles in DINO/v2 training while improving representations (ImageNet-KNN).
- DiMA: distilling a VLM into a planner for driving. They use VAD as a baseline and pass tokens from its models into an LLM. The LLM is trained to do VQA, MAE, future prediction, and scene editing. You distill the LLM into planner transformer--you don't need the LLM for inference.
- What would the deepseek team accomplish with a month or two on xAI's H100 cluster?> $5.5M for Sonnet tier it's unsurprising that they're proud of it, but it sure feels like they're rubbing it in. «$100M runs, huh? 30.84M H100-hours on 405B, yeah? Half-witted Western hacks, your silicon is wasted on you, your thoughts wouldn't reduce loss of your own models»
- Super excited that DeepEthogram is published officially today in @eLife!! elifesciences.org/articles/63377 (1/N)
- Replying to @neeratandenIt's definitely not moral compunction. If they don't help Trump, it's because they think it's not in their interest, not against their morals
