Magic Mirror

We often find it difficult to train our robots in the spaces around us. Thus, we introduce a custom hardware integrated pipeline to create simulated training environments, a "mirror" of our reality.

Comment

Inspiration

As roboticists, we've all been here: You want to train a robot for a specific environment, but:

  1. Setting up training in real spaces is time-consuming
  2. Environmental constraints limit training options
  3. Data collection requires specialized tools
  4. Transferring skills between environments is challenging

Current solutions either require expensive hardware setups or suffer from the sim-to-real gap. The physical world is right there – we just needed a way to digitize it effectively! Enter Magic Mirror

What it does and how we build it

We've created an integrated pipeline that uses custom hardware to create perfect digital twins of physical spaces, enabling efficient robot training in virtual "mirrors" of reality.

What to Love About Magic Mirror

  1. Seamless Data Collection: Mobile robot platform gathers comprehensive environmental data
  2. Automatic Processing: SLAM technology creates dense point clouds converted to clean STL models
  3. User-Friendly Interface: Control robots and visualize/interact with point clouds and 3D models
  4. Efficient Training: Train robots via reinforcement learning in simulated environments Real-World Transfer: Skills learned in simulation transfer directly to physical spaces

No more struggling with environmental constraints. No more limited training scenarios. Just efficient robot learning in perfect digital replicas of your spaces!

Our Tech Stack

Here's what powers Magic Mirror:

  1. RealSense Cameras for high-quality spatial data collection
  2. ROS & RTAB-Map for SLAM and dense point cloud generation
  3. Custom Voxelization & Cleaning Algorithms for processing environmental data
  4. Tailnet for wireless connectivity via Flask
  5. Genesis RL Framework for training in simulated environments

Challenges we ran into

  1. Point Cloud Processing: Converting raw sensor data into clean, usable 3D models required sophisticated filtering and optimization.
  2. Wireless Connectivity: Ensuring stable, high-bandwidth connections for real-time data transfer between robots and processing systems.
  3. Designing, Manufacturing, Wiring, and controlling our hardware platform in the 24 hour time constraint with no preprepared electronics, sensors, or cad.

Accomplishments that we're proud of

  1. We were able to generate a streamlined and modern interface for the project
  2. Our point clouds could be efficiently converted into clean STL files
  3. We were able to train reinforcement learning models in the "mirrored" environment
  4. We were able to use ROS and RTAB to effectively localize and map our environment with the RealSense cameras

What's next for Magic Mirror

  1. Expanding Robot Compatibility: Support for more robot platforms beyond our initial implementations.
  2. Real-Time Environment Updates: Dynamic updating of simulated environments as physical spaces change.
  3. Multi-Robot Collaboration: Training robot teams to work together using our digital twin environments.
  4. Open-Source Release: Sharing our pipeline with the broader robotics community to accelerate research and development!

Built With

Share this project:

Updates