First time I've opened up about the Lightning story, military, PhD, open-source and merging companies!
Excited to share the journey with everyone.
youtu.be/EmvrHnjAySg?si…
Most people don't know that Lightning Studios offer:
- free persistent storage
- free persistent environments
- unlimited background execution
- VSCode, PyCharm, (any IDE) integration
Set up your Studio environment once and reuse it again any time 🤯🤯
In this new 90 minute lecture, I show how to pretrain a 3B LLM from scratch. No edits. No detail skipped.
Companies want you to believe pretraining models is super hard and costly. With the right tools, it's not.
- We start by tuning the model on a cheap A10G.
- Then we scale
Excited to announce the release of bolts!
- linear + Logistic regression on TPUs/GPUs
- self-supervised learning
- RL
- GANs
- GPT
- Callbacks library
- Datamodules library
All powered by lightning and rigorously tested.
Don't waste more time implementing your own baselines...
Bolts is a new Deep Learning research and production toolbox from PyTorch Lightning. Iterate faster with pre-trained models, components, callbacks, and data sets, all modular, tested, and optimized for GPUs/TPUs.
Simply subclass, override, and train.
medium.com/pytorch/pytorc…
Excited to announce our new compiler - Thunder!
(built in collaboration with NVIDIA). 🤯 🤯
Thunder is a source to source compiler for PyTorch. It speeds up PyTorch models.
As an example, it speeds up llama 2 7B by 40%.
github.com/Lightning-AI/l…
Excited to announce the launch of GPT-42.
- Half the size of GPT-3 (100 billion parameters)
- runs on 775 watts a day (2000 calories)
- can do one-shot learning
- multi-modal
- does NOT require 9,000 GPUs
- 300,000 years worth of evolution research!
Inference API coming soon!
LLM? old news. New hotness will be LVMs (Large Vision Models).
This time, looks like @GoogleAI is finally staying ahead of @OpenAI.
How well does it work?
a 🧵-> 1/n
arxiv.org/pdf/2302.05442…
Excited to release our latest paper (with @kchonyc) which establishes a conceptual framework for characterizing contrastive learning methods (SimCLR, BYOL, CPC, AMDIM, Swav). (work done at @facebookai)
Btw.. this was the motivation for @PyTorchLightninbit.ly/yadimp