Andreas Steiner (@AndreasPSteiner) / X

Andreas Steiner

97 posts

Andreas Steiner

@AndreasPSteiner

Researching #ComputerVision at #GoogleDeepMind using JAX/Flax (github.com/google/flax). views are my own.

Zurich, Switzerland

Joined August 2021

Pinned
Andreas Steiner
@AndreasPSteiner
Jun 3
Gemma 4 12B in action: Object detection, function calling, voice command, segmentation, language switch, translation - all of this and much more without vision/audio encoders! (Inputs and outputs are real, but FC2 data shown as code, and generation speedified)
00:00
10K
Andreas Steiner
@AndreasPSteiner
Apr 14, 2022
LiT-tuned models combine powerful pre-trained image embeddings with a contrastively trained text encoder. Check out our blogpost goo.gle/3KJT6u1 and the 🔥 interactive demo ↓↓↓ #lit_demo google-research.github.io/vision_transfo…
GIF
Andreas Steiner
@AndreasPSteiner
Dec 5, 2024
🚀🚀PaliGemma 2 is our updated and improved PaliGemma release using the Gemma 2 models and providing new pre-trained checkpoints for the full cross product of {224px,448px,896px} resolutions and {3B,10B,28B} model sizes. 1/7
62K
Andreas Steiner
@AndreasPSteiner
Aug 31, 2022
Did you know how easy it is to create demos from JAX/Flax models that run in the browser? See our blogpost for 1. how to do the JAX to TF[.js] conversion 2. how to prototype simple demos in Colab 3. how to create a fully featured frontend app blog.tensorflow.org/2022/08/jax-on… #tensorflowjs
JAX on the Web with TensorFlow.js
From blog.tensorflow.org
Andreas Steiner
@AndreasPSteiner
Oct 4, 2021
"JAX: High-performance ML research" workshop at #AMLD2021 in Switzerland. I will be hosting this advanced in-person workshop with @avitaloliver on October 26th in Lausanne and explain how to build models with JAX/Flax. Still some tickets available at tinyurl.com/jax-amld21
Andreas Steiner
@AndreasPSteiner
May 27, 2024
PaliGemma-3B: Our latest open VLM that also does instance segmentation and object detection, with a great performance / compute trade-off. Research code / results: github.com/google-researc… Finetuning Colab: colab.research.google.com/github/google-… HF Demo: hf.co/spaces/google/…
Google DeepMind
@GoogleDeepMind
May 14, 2024
We’re introducing new additions to Gemma: our family of open models built with the same technology as Gemini. 🔘 PaliGemma: a powerful open vision-language model 🔘 Gemma 2: coming soon in various sizes, including 27 billion parameters → dpmd.ai/3QKEteK #GoogleIO
big_vision/big_vision/configs/proj/paligemma/README.md at main · google-research/big_vision
From github.com
13K
Andreas Steiner
@AndreasPSteiner
Jul 12, 2024
PaliGemma-3B paper is out – with 58 pages of exciting insights! arxiv.org/abs/2407.07726 As opposed to most other VLMs, you would finetune a bit for your usecase to make sure the model understands what you want – but that's so cheap it literally becomes a vertical bar in Figure 3
5.9K
Andreas Steiner
@AndreasPSteiner
Feb 19, 2025
Looking for a small or medium sized VLM? PaliGemma 2 spans more than 150x of compute! Not sure yet if you want to invest the time 🪄finetuning🪄 on your data? Give it a try with our ready-to-use "mix" checkpoints: 🤗 huggingface.co/blog/paligemma… 🎤 developers.googleblog.com/en/introducing…
8.7K
Andreas Steiner
@AndreasPSteiner
Feb 13, 2023
Scaling Vision Transformers to 22 billion parameters continues to improve ImageNet and OOD classification. And while ImageNet top1-accuracy seems to saturate short of 91% after fine-tuning, ObjectNet accuracy continues to increase, resulting in better effective robustness.
Mostafa Dehghani
@m__dehghani
Feb 13, 2023
Replying to @m__dehghani
This was a collaboration with an amazing group of people including @PiotrPadlewski, @_basilM, @m__dehghani, @JonathanHeek, @jmgilmer, @AndreasPSteiner, @MJLM3, @mcaron31, @ibomohsin, @RJenatton, @rikelhood, @mechcoder, @anuragarnab, @brainshawn, @giffmana, @mtschannen,...
9.7K
Andreas Steiner
@AndreasPSteiner
Oct 1, 2024
I'm at ECCV this week. Come see our PaliGemma demo on Thu 10:30 at the Google booth (or DM me to go for a good conference espresso!)
4.3K
Andreas Steiner
@AndreasPSteiner
May 4, 2022
Great codebase for writing nimble research code – I especially like the efforts for readability (short hackable code, staying clear of too many abstractions) and performance optimizations!
Alexander Kolesnikov
@__kolesnikov__
May 4, 2022
Let me introduce big_vision: an original home of ViT, MLP-Mixer, LiT and many more. These days I do all my research in this codebase: it is great for doing vision research with emphasis on large-scale pretraining and transfer. Highlights in 🧵 ⬇️ Link: github.com/google-researc…
Andreas Steiner
@AndreasPSteiner
Jul 14, 2023
NaViT lets you process more images with the same compute – with varying resolutions and aspect sizes! The performance gains translate to OOD datasets with extreme aspect ratios – with the a cropping tuned for OG square ViT, and even more so with a simple "no-prior" resize.
Mostafa Dehghani
@m__dehghani
Jul 13, 2023
1/ Excited to share "Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution". NaViT breaks away from the CNN-designed input and modeling pipeline, sets a new course for ViTs, and opens up exciting possibilities in their development. arxiv.org/abs/2307.06304
4.5K
Andreas Steiner
@AndreasPSteiner
Dec 5, 2024
Replying to @AndreasPSteiner
As the original PaliGemma, the pre-trained PaliGemma 2 models have segmentation and detection capabilities, and excel at OCR – which makes them extremely versatile for 🪄finetuning🪄. The original demo hf.co/spaces/big-vis… gives you an idea of the capabilities. 3/7
1K
Andreas Steiner
@AndreasPSteiner
Jun 16, 2023
Computer science that you can touch? If you're in ZRH, come and check out our nuru.nu Seli installation tonight at the D18 Art&Tech Show in Rote Fabrik, to experience the wonderful NCA work from @zzznah @eyvindn @RandazzoEttore @drmichaellevin
2K