Augustinas Malinauskas

#ResearchPaper0019 Why you should read 📘 How Well Does GPT-4o Understand Vision? 📍 By: Rahul Ramachandran, Ali Garjani, Roman Bachmann, Andrei Atanov, Oğuzhan Fatih Kar, Amir Zamir 📄 Published via EPFL | Read the paper: arXiv:2507.01955 🔍 What’s It About? This groundbreaking research from EPFL addresses a central question in AI development: Can multimodal foundation models like GPT-4o truly understand vision tasks as well as traditional vision models? The study evaluates popular models—GPT-4o, Gemini 2.0, Claude 3.5 Sonnet, Qwen2-VL, LLaMA 3.2, o4-mini, etc.—on six fundamental computer vision tasks: • Image Classification • Object Detection • Semantic Segmentation • Depth Prediction • Surface Normal Estimation • Visual Grouping Using a prompt-chaining technique, the authors convert complex vision tasks into formats compatible with text-based multimodal models, allow in direct comparison with specialized vision models. ⸻ 💡 How Will This Help Your AI Career? Whether you’re in research, product, or applied AI development, this paper: ✅ Reveals capabilities and limitations of multimodal LLMs in real-world vision tasks ✅ Introduces a reproducible evaluation framework that helps benchmark and compare models effectively ✅ Shows GPT-4o ranks #1 in 4 of 6 tasks, proving it’s a strong generalist despite being outperformed by vision-specialists in fine-grained geometry tasks ✅ Demonstrates where prompt design critically impacts results, offering insights for prompt engineers ✅ Validates prompt chaining as a smart workaround to evaluate non-trainable, closed models ✅ Highlights differences in semantic vs. geometric reasoning, a crucial area for product developers working on AR/VR, robotics, and autonomous systems ⸻ 🎯 Who Should Read This? • 🤖 AI/ML Engineers working on multimodal agents • 🧪 CV Researchers comparing specialist vs. generalist model capabilities • 🧠 AI Product Designers aiming to integrate vision-LM hybrids • 📝 Prompt Engineers and technical analysts who build with APIs • 🧑‍🎓 Students preparing for real-world AI challenges ⸻ 💬 Final Thoughts: This isn’t just another benchmark. It’s a wake-up call for the future of vision in multimodal AI. 📌 GPT-4o and its peers still have a long way to go on 3D tasks like depth estimation and surface normals—but this paper proves they’re already impressive semantic generalists. 📂 The authors also promise to open-source their evaluation toolkit, giving the AI community a unified framework to test vision in LLMs. 📥 Read it. Save it. Use it. 🌐 Project Site: fm-vision-evals.epfl.ch #GPT4o #ComputerVision #MultimodalAI #AIResearch #LLM #GenerativeAI #PromptEngineering #VisionTasks #MachineLearning #DeepLearning #OpenAI #Claude #Gemini #LLaMA #EPFL #arXiv

36

1 Comment

It's finally out! 🎉 🔎 Today we published our latest research article called "Beyond Reserves: Linking Blockchain Performance to Stablecoin Stability" in collaboration with TRON DAO. 🙌 🔎 Stablecoins are foundational to DeFi, trading, and payments, yet peg reliability remains under scrutiny. Periodic deviations continue to raise questions about the resilience of issuers and the full set of factors underpinning peg maintenance. 💡 Particula’s latest research moves beyond issuer reserves to evaluate how blockchain performance metrics - such as throughput, latency, and finality - relate to peg stability. The study examines eight major networks and focuses on Circle's USD Coin (USDC) and Tether.io's (USDT) over the period January 1, 2022 to July 31, 2025. Big thanks to our analysts Paweł Borowski and Jeeta Ann Chacko for doing all of the research and for writing such an insightful report! In a world where we have more and more AI generated content - I hope you all appreciate the effort we always put into our reports. 🤗 #stablecoins #tokenization #pegstability #USDCoin #tether #particula #tron Yong Yi T. Yeweon Park William Hess H.E. Justin Sun

70

7 Comments

New Short Course: Getting Structured LLM Output! Learn how to get structured outputs from your LLM applications in this course, built in partnership with DotTxt, and taught by Will Kurt, a Founding Engineer, and Cameron Pfiffer, Developer Relations Engineer. It's challenging for software to automatically parse through an LLM's freeform text outputs. Structured outputs—like JSON—solve this by converting natural language into consistent, clear, data that a machine can read and process. This course teaches you how to generate structured outputs while building several use cases, including a social media analysis agent. You’ll learn about structured outputs and efficient ways to generate outputs in your defined schema or format. You’ll begin by using structured output APIs, then use re-prompting libraries like “instructor” to generate structured output. Finally, you’ll learn how constrained decoding works; this is a very clever technique in which constraints are applied on each subsequent token generated, blocking any tokens that don’t fit your defined schema. In detail, you’ll: - Learn why structured outputs are important, how they allow for scalable software development, and the different approaches to generate them, including vendor-provided APIs, re-prompting libraries, and structured generation. - Build a simple social media agent using OpenAI’s structured output API, learn how to define a model's desired structured output using Pydantic, and perform basic programming with your outputs, such as importing structured data into a data frame using pandas. - Learn how to use the open-source library "instructor," which checks the structured output of the model and re-prompts the model until it validates the desired output, and explore the limitations of this approach. - Understand how structured generation by the “outlines” library works by modifying LLM logits, on a per-generated-token basis based on the desired format, to give a particular output structure. - Learn how regular expressions, which outlines works with, are represented as finite-state machines, and how they can be used to develop a range of structured outputs beyond JSON. By the end of this course, you’ll have broadened your knowledge of the approaches you can use to get structured outputs from your LLM applications. Please sign up here: https://lnkd.in/gpkNZvwQ

5,133

141 Comments

Reasoning LLMs Guide I know the timing is crazy, but we just published a guide on reasoning models. This is great for devs using models like o3 and Gemini 2.5 Pro. Also includes failure modes and limitations.

238

13 Comments

An interesting new paper on LLM-JEPA from Hai Huang, Yann LeCun, and Randall Balestriero. 💡 Previously I wrote about JEPA approach applied to videos (V-JEPA and V-JEPA 2) and time series (CHARM). Now the JEPA approach is finally applied to LLMs! This work bridges a major gap between AI for vision and language, offering a potential leap forward in how we train language models. Instead of just predicting the next word, LLM-JEPA teaches models to understand the underlying meaning by predicting abstract concepts (as JEPA approach does)—for instance, grasping the essence of a code snippet from its natural language description. The paper introduces a hybrid objective combining standard next-token prediction with a Joint Embedding Predictive Architecture (JEPA) loss, a technique highly successful in computer vision. The empirical results are compelling: LLM-JEPA consistently boosts performance, accelerates parameter-efficient fine-tuning (PEFT), and shows remarkable resistance to overfitting. This method doesn't just improve scores; it fundamentally creates more structured and transferable representations. While the current computational overhead is a challenge to address, this paper opens a promising new direction beyond traditional LLM training. 🚀 Review: https://lnkd.in/eC4Jte_r Paper: https://lnkd.in/erZJadb3 Code: https://lnkd.in/ethXT7sX

75

3 Comments

Switzerland just gave AI founders a cheat code. A fully open LLM built on public infrastructure. Announced by ETH Zurich and EPFL- and Open Source Not another closed API that can rug prices (You can try Apertus using link in comments) But why is this new LLM any different? Last week, I met a couple of brilliant founders working on high quality datasets for Multimodal AI With a huge focus on multilingual (Shoutout to Anata Ikauniece) And as we discussed I realised how much LLMs still struggle with Multilingual Training is English first and then translated to other language The kicker with the swiss LLM is multilingual. 1,000 to 1,500 languages, not English-only. That means fewer translation errors in the wild. And real access for non-English users. But not just that remmeber that Switzerland's LLM is open source too. License matters more than params. Apache 2.0 means real commercial freedom. But whats the real world impact? A friend I advise is working with a hospital lab that needed on-prem AI for compliance. Closed models failed their audits. Open weights passed with auditable data lineage. Economics change when you own the model. CapEx over OpEx - and predictable unit costs. Engineering changes too. You ship faster when your stack is inspectable. Ethics change the most. Governance is easier when datasets are transparent. Not biggest model - best incentives. Founders, this is the move. Prototype on APIs - graduate to open. AI markets loves speed and leverage. Open (not Open AI) gives both when you operationalize it. My rule going forward is simple. Rent for discovery - own for scale.  #MultilingualAI  #OpenSource

46

32 Comments

NLP Labs, Northeastern University & NiuTrans Research, Tong Xiao and Jingbo Zhu, "Foundations of Large Language Models” This book aims to outline the basic concepts of large language models and introduce the related techniques. As the title suggests, the book focuses more on the foundational aspects of large language models rather than providing comprehensive coverage of all cutting-edge methods. CONCURRENCE ——— #datasharing #dataportability #concurrence #dataaccess #M2MI #aiagents #agenticai #automatedM2Mintelligence #datainteroperability #FMI #globalsupplychain #crossboarderpayments #securefinancing #fraudprevention #tradefinance #openbanking #fintechdataplatform #tradematters #sustainability #digitaltrade #globaltrade #FinancialMarketInfrastructure #technology #LEI #digitalidentity #C4DTI #paperless #digitalidentities #datasharing #innovation #digitalisation #paperlesstrade #digitaltransformation   #digitaltradechampion #digitaltradeevangelist #openfinance #internationaltrade #digital #crossbordertrade #smartdata #cbdc #regtech #suptech #payments #innovation #banks #digitalrevenueimpact #risk #regulations #digital #m2mi #cybersecurity #risk #banking #embeddedfinance #openbanking #sustainablefinance #iot #money #bankofengland #digitaleconomy #fintech #multicurrency #privacy #artificialntelligence

25

𝐌𝐚𝐬𝐭𝐞𝐫𝐢𝐧𝐠 𝐑𝐀𝐆: 𝐓𝐡𝐞 𝐒𝐞𝐜𝐫𝐞𝐭 𝐒𝐤𝐢𝐥𝐥 𝐓𝐡𝐚𝐭 𝐖𝐢𝐥𝐥 𝐃𝐞𝐟𝐢𝐧𝐞 𝐭𝐡𝐞 𝐍𝐞𝐱𝐭 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 𝐨𝐟 𝐀𝐈 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬 If you’re serious about building a long-term career in 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞, there’s one skill that separates the AI enthusiasts from the AI engineers — 𝐑𝐀𝐆 (𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥-𝐀𝐮𝐠𝐦𝐞𝐧𝐭𝐞𝐝 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧). We’ve all heard about LLMs like GPT, Claude, Gemini, and Mistral. They’re incredible at reasoning, summarizing, and generating human-like text. But here’s the catch — 𝐭𝐡𝐞𝐲 𝐝𝐨𝐧’𝐭 “𝐤𝐧𝐨𝐰” 𝐲𝐨𝐮𝐫 𝐝𝐚𝐭𝐚. That’s where RAG changes the game. RAG combines the 𝐫𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠 𝐩𝐨𝐰𝐞𝐫 𝐨𝐟 𝐋𝐋𝐌𝐬 with the 𝐟𝐚𝐜𝐭𝐮𝐚𝐥 𝐚𝐜𝐜𝐮𝐫𝐚𝐜𝐲 𝐨𝐟 𝐩𝐫𝐢𝐯𝐚𝐭𝐞 𝐨𝐫 𝐞𝐧𝐭𝐞𝐫𝐩𝐫𝐢𝐬𝐞 𝐝𝐚𝐭𝐚, creating AI systems that can think, reference, and respond intelligently — just like an 𝐞𝐱𝐩𝐞𝐫𝐭 𝐡𝐮𝐦𝐚𝐧 𝐰𝐨𝐮𝐥𝐝. 𝐖𝐡𝐲 𝐑𝐀𝐆 𝐈𝐬 𝐚 𝐌𝐮𝐬𝐭-𝐋𝐞𝐚𝐫𝐧 𝐒𝐤𝐢𝐥𝐥 𝐢𝐧 𝟐𝟎𝟐𝟓/26 𝟏️ 𝐈𝐭 𝐛𝐫𝐢𝐝𝐠𝐞𝐬 𝐭𝐰𝐨 𝐰𝐨𝐫𝐥𝐝𝐬 — knowledge and intelligence. RAG connects structured and unstructured data (from PDFs, APIs, SQL, CRMs, etc.) directly to your LLM, enabling real-time, context-aware responses. 𝟐️ 𝐄𝐯𝐞𝐫𝐲 𝐥𝐞𝐚𝐝𝐢𝐧𝐠 𝐀𝐈 𝐜𝐨𝐦𝐩𝐚𝐧𝐲 𝐢𝐬 𝐮𝐬𝐢𝐧𝐠 𝐢𝐭. Microsoft Copilot, OpenAI ChatGPT Enterprise, Slack GPT, Notion AI, Box AI, and Salesforce Einstein all run on RAG-based architectures. 𝟑️ 𝐈𝐭’𝐬 𝐭𝐡𝐞 𝐟𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧 𝐟𝐨𝐫 𝐀𝐠𝐞𝐧𝐭𝐢𝐜 𝐀𝐈 𝐚𝐧𝐝 𝐋𝐋𝐌𝐎𝐩𝐬. RAG pipelines are the backbone of multi-agent systems, retrieval chains, and production-grade AI workflows. 4. 𝐘𝐨𝐮’𝐥𝐥 𝐦𝐚𝐬𝐭𝐞𝐫 𝐢𝐧-𝐝𝐞𝐦𝐚𝐧𝐝 𝐭𝐨𝐨𝐥𝐬 𝐚𝐧𝐝 𝐟𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤𝐬. Learn how to work with FAISS, Pinecone, Chroma, Milvus, Weaviate, LangChain, LangGraph, CrewAI, and LlamaIndex. 5 𝐈𝐭’𝐬 𝐜𝐨𝐬𝐭-𝐞𝐟𝐟𝐞𝐜𝐭𝐢𝐯𝐞 𝐚𝐧𝐝 𝐬𝐜𝐚𝐥𝐚𝐛𝐥𝐞. 🚀𝐓𝐡𝐞 𝐅𝐮𝐭𝐮𝐫𝐞 𝐨𝐟 𝐀𝐈 𝐁𝐞𝐥𝐨𝐧𝐠𝐬 𝐭𝐨 𝐑𝐀𝐆 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬 The next wave of AI innovation won’t just be about building smarter models — it’ll be about connecting models to the right knowledge, securely and efficiently. If you want to: ✅ Build 𝐩𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧-𝐫𝐞𝐚𝐝𝐲 𝐆𝐞𝐧𝐀𝐈 applications, ✅ Stand out in the AI job market, or ✅ 𝐅𝐮𝐭𝐮𝐫𝐞-𝐩𝐫𝐨𝐨𝐟 𝐲𝐨𝐮𝐫 𝐜𝐚𝐫𝐞𝐞𝐫 as an AI Engineer, You’re not just learning a technique 𝐲𝐨𝐮’𝐫𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐭𝐡𝐞 𝐢𝐧𝐟𝐫𝐚𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞 𝐨𝐟 𝐢𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐭 𝐬𝐲𝐬𝐭𝐞𝐦𝐬. 𝐁𝐨𝐧𝐮𝐬 𝐓𝐢𝐩: 𝐁𝐮𝐢𝐥𝐝 𝐚 𝐬𝐦𝐚𝐥𝐥 𝐑𝐀𝐆 𝐩𝐫𝐨𝐣𝐞𝐜𝐭 like a ChatGPT for your company documents or a customer support bot. It’s the fastest way to understand the architecture hands-on. #RetrievalAugmentedGeneration #RAG #GenerativeAI #LangChain #LLMOps #AIEngineering #AIForBusiness #MachineLearning #ArtificialIntelligence #DataScience #Chroma #LangGraph #CrewAI #LlamaIndex #OpenAI #MicrosoftCopilot #AITrends2025 #GenAI #PromptEngineering #LLMDevelopment #AIProductDesign #TechCareers #DeepLearning

8