Trainify

Watch your AI agent come to life

Comment

🚀 Trainify: Convert 2D image into 3D environments to train RL agents in the cloud

🌟 Project Overview

Trainify simplifies reinforcement learning by enabling users to create and train AI agents in 3D environments directly on the cloud. Leveraging cutting-edge 2D/3D computer vision and NLP, our platform bridges the gap between idea and execution, allowing you to generate complex environments with simple natural language prompts and image references.

💡 Inspiration

We were inspired by the increasing need for realistic 3D environments in various fields such as robotics training, game development, and simulation. Current solutions often require advanced technical skills and specialized knowledge. We wanted to democratize this process by leveraging the power of generative AI to make 3D environment creation accessible to everyone, regardless of their technical background.

🛠️ What It Does

Trainify offers a seamless workflow:

  • 📝 Creation Interface: Users can describe their desired environment using natural language prompts with their uploaded reference images for simplistic workflow
  • 🌐 3D Visualization: Segment objects of interest using the text prompt and turn images into 3D object representation where you can explore and modify in real time
  • ✏️ Editing Mode: Intuitive tools allow users to adjust object positions, properties, and relationships within the 3D space with our cloud-based collaborative editor
  • 🤖 Training Mode: Built-in reinforcement learning capabilities enable users to train AI agents to perform tasks within their custom environments.
  • 💬 AI Assistant: An integrated Gemini-powered chatbot provides context-aware assistance during every step of the process.

⚙️ How We Built It

Our platform uses a modern tech stack:

  • 🎨 Navigable 3D Frontend: React with styled-components for a responsive and intuitive UI and Three.js for real-time 3D visualization in the browser
  • 🔧 Cloud Native Deployment: Highly scalable cloud microservices deployed using GCP Cloud Run and efficient and customized compute units on the cloud for real time training using Docker and Kubernetes (GKE)
  • 🧠 Generative AI: Google's Gemini API for natural language understanding and generation, CLIPSeg for contrastive learning based image segmentation, Trellis for 3D gaussian splatting
    • Custom reinforcement learning framework built with PyBullet
  • 🗃️ Database & Storage: Firebase for user authentication and project management and Cloud Storage for highly accessible, available storage for large model files

🧩 Challenges We Ran Into

  • 🚧 Performance Optimization: Rendering complex 3D environments smoothly in the browser required significant optimization.
  • ☁️ Navigating GCP: Google Cloud Platform has a lot of complex features that take time to learn and setup
  • 🤝 Real-time Collaboration: Implementing collaborative features presented synchronization challenges.

🎉 Accomplishments That We're Proud Of

  • Created an intuitive interface accessible to non-technical users.
  • Integrating latest GenAI research from multiple fields like computer vision, agentic LLMs, and reinforcement learning.
  • Developed a novel approach for training reinforcement learning agents in user-generated environments directly from daily life scenarios
  • Built a scalable, cloud native architecture handling complex environments and simulations using Google Cloud Platform

📚 What We Learned

  • Importance of balancing automation and user control in AI-assisted creative tools.
  • Techniques for optimizing 3D rendering performance in browsers.
  • Effective communication strategies between frontend and backend components.
  • Methods to translate natural language descriptions into structured 3D scene representations.

🔮 What's Next for Trainify

  • 📦 Expanded Asset Library: Developing a more comprehensive library of pre-built objects and environments.
  • 👥 Enhanced Collaboration: Adding real-time collaboration features for teams like Google Docs!
  • 🧪 Integration with Industry Standard Toolkits: By extending native support to OpenUSD, we allow easy export and import between industry standard physical AI and robotics framework such as Nvidia Omniverse and Issac Sim, taking a step closer to training physical robots!

🎈 Try It Out

Experience the future of 3D environment creation with Trainify—where your imagination becomes reality with the power of AI!

Built With

Share this project:

Updates