1/7 Spent the week-end with ControlNet, a new approach to have precise, fine-grained control over image generation with diffusion models. It's huge step forward and will change a number of industries. Here is an example.
Guido Appenzeller
3,591 posts
Guido Appenzeller
@appenz
At a16z investing in AI & Infra. 2x founder & CEO. CTO at Intel & VMware. CPO at Yubico. Tweets are my own.
- 🚨New Content: The Trillion Dollar AI Software Development Stack It will generate massive value, spawn hundreds of start-ups and has created the fastest growing companies in history. @stuffyokodraws and I did a deep-dive on market, start-ups and the evolving stack. ⬇️
- About 15% of all Stanford undergrads (of all majors) are learning how to build LLMs. Stats for @chrmanning's CS224N Natural Language Processing with Deep Learning below. That makes sense. LLM's are becoming a basic systems component like compute, networking and storage.
- 🔥 In a recent post Sequoia's @DavidCahn6 argues AI infra is overbuilt: - NVIDIA GPU revenue is $50b/y - This requires $200b in "AI revenue" - There is only $75b in "AI revenue" Thus there is a $125b hole I strongly disagree. AI Infra will be huge. Grab🍿and read on (🧵1/7).
- Replying to @appenz6/7 If you want to try this out, it's realtively easy to get running if you have a Windows PC with a high-end Graphics Card (12 GB RAM is ideal, but less works too). I used the Automatic1111 extension here.
- I am excited to announce that I am joining @Intel as CTO of the Data Platforms Group. Intel is not without challenges. But I think strength in CPUs, a diverse portfolio, massive scale and Navin’s great team will make it successful. And it's awesome to work with @PGelsinger again!
- 1/5 For companies doing Generative AI, finding enough GPUs is a difficult and expensive. We’ve seen companies spend 80%+ of total capital raised on compute resources. To help them @casado, @BornsteinMatt and I wrote a post how to navigate the cost of AI.
- 1/6 I fine-tuned a @bfl_ml's Flux.1-dev LoRA on myself. It fine tunes really well, easier than SDXL although not quite as easy as SD 1.5. Tuning was done on @replicate, the model hosted on @huggingface. Total cost is about $7 for 75 minutes on an A100. Full instructions below.
- Replying to @appenz2/7 The basic idea of control net is that your diffusion model works in tandem with a second model trained on a specific task. This is the sd15_scribble model. It helps Stable Diffusion 1.5 to interpret sketches, and turns my "sketch" of an Owl into an awesome drawing.
- Replying to @appenz7/7 Congrats to Lvmin Zhang and Maneesh Agrawala @magrawala at Stanford for developing this technology, the original research publication is here.
- 1/3 New benchmarks from @MLPerf , and they include the first good B200 numbers that I have seen. 11,264 tokens/s for Llama 2 70b is crazy, and about 3.7x the performance of the H100🤯 The bar for every AI silicon start-up out there just went up. Some thoughts below.
- 1/8 I created a picture of three robots posing to spell the letters "AI". A month ago creating this would have taken me days, this week-end it took me 15 minutes. Below an overview of ControlNet and OpenPose. Safe to say this will change image generation forever.
- Replying to @appenz5/7 And last but not least, ControlNet combines with fine-tuning via DreamBooth. I can fine-tune a model on myself (like in the James Bond picture), and then use it to render myself into a specific scene. Here is our daughter who wanted to be Black Widow.
- This "investor" currently has the #1 and #5 spot on GitHub's trending repos. I love working with this team.
