Overall Project
GIF

Note

The images are very limited — we posted quick photos due to time constraints.
Full details are included in the slideshow.

FireGuard: Enhancing Safety, Communication, and Navigation

Inspiration

As a past volunteer firefighter, one of our teammates experienced firsthand the challenges of limited visibility, disorientation, and broken communication in burning structures. Radios often fail due to noise, and teammates can’t always be located in real time. We wanted to create a tool that improves situational awareness, navigation, and communication for firefighters — ultimately saving lives.

What It Does

FireGuard is a helmet-mounted tactical command system that combines sensors, AI, and networking infrastructure into one platform.

Features and Benefits

Real-Time Object Detection (YOLOv5 on Camera + LiDAR) — Detects doors, furniture, appliances, and obstacles in real time. Camera and LiDAR data are fused for detection, with LiDAR point clouds providing rough estimations when vision is degraded in smoky conditions.
Semantic Room Classification (CV + LLM) — Converts CV-detected objects into natural room labels on-device using a lightweight DistilGPT-2 LLM for clearer communication (e.g., fridge + stove → "kitchen").
Hand Gesture Recognition (MediaPipe Hands) — Supports firefighter hand signals like "Stop," "Help," and "OK" for silent communication.
Voice-to-Text Transcription (Whisper) — Converts noisy radio speech into clear text and streams it into a shared chat system.
Team Communication (Chat + Pings) — Firefighters and commanders can leave pings with locations on maps, alongside text and Whisper-transcribed speech for coordinated awareness.
Inactivity Detection (IMU BNO055) — If a firefighter stops moving, the system pings their last known location to signal danger.
Object Avoidance Alerts (Radar Fusion) — Radar returns generate SAFE/WARNING/CRITICAL/EMERGENCY alerts, warning firefighters of nearby obstacles.
Radar + LiDAR + IMU Fusion for SLAM — Radar penetrates smoke while LiDAR captures local geometry; fused to enable SLAM mapping in zero visibility. Localization relies on IMU data, with drift reduced through self-correction using the fused point cloud.
Heads-Up Display (HUD) — Shows maps, detections, alerts, and communication overlays inside the helmet.
Command Center Dashboard (Next.js + React) — Tactical interface for commanders to monitor multiple firefighters.
Networking Infrastructure — Built a custom router with serial + WebSocket pipelines to transmit massive sensor streams (LiDAR, radar, camera, IMU, mic) into a command center dashboard. All helmets can participate in a mesh network, where nodes forward data to maintain connectivity in unreliable environments.

How We Built It

Hardware: Jetson TX2 (Ubuntu 18.04), ESP32/Arduino + I2S mics, RPLiDAR, 24 GHz FMCW radar, IMU sensors (BNO055), RGB camera.
Software/AI: YOLOv5 (object detection), MediaPipe Hands (gestures), Whisper (speech transcription), DistilGPT-2 (room classification).
Networking: Custom ad-hoc router, ports for streaming sensor data, with theoretical support for scalable mesh networking.
Frontend/Backend: Next.js + React dashboard, FastAPI + WebSockets for ingestion, fusion, and REST APIs.

Challenges We Ran Into

SLAM with IMU + LiDAR + Radar —The hardest challenge was drift and angular displacement, which made 2D mapping extremely difficult. We had to assume relatively stable, level head movement for the proof of concept. Correcting IMU + LiDAR point clouds for drift was computationally heavy, and the limited processing speed made real-time HUD display difficult. Sharing a global map over a mesh network also proved infeasible due to time contraints, so testing was restricted to a single helmet.
Router & Networking — Streaming LiDAR, radar, IMU, mic, and camera data over Arduino/Jetson pipelines proved extremely challenging due to bandwidth and synchronization. Setting up reliable serial communication and forwarding data over sockets made it difficult to maintain low-latency performance with such large data streams.
Jetson TX2 limitations — Only supported Ubuntu 18.04; dependency conflicts blocked ROS2 SLAM mapping and some CV models. Due to this, we abandoned the Jetson TX2 and instead ran all computation on the laptop, which also hosted the HUD display and command center website.
Hardware integration hurdles — Sparse documentation (500+ pages of radar specs) and poor tooling slowed progress.

Accomplishments We’re Proud Of

Built a working pipeline fusing radar, LiDAR, and IMU for SLAM in simulated smoky conditions, generating reliable maps and enabling object avoidance.
Prototyped semantic room mapping with sensor point clouds + deep learning.
Unified speech, gesture, and chat (with pings) into one communication system.
Designed a network architecture capable of scaling into a true mesh network.

What We Learned

Why localizing with IMUs in dynamic environments is so difficult.
The complexity of real-time networking for multi-sensor data.
Learned how semantic mapping with CV + LLMs, LiDAR point cloud object detection as a fallback, and self-correction through LiDAR–radar fusion complement each other, ensuring reliability when certain sensing modalities go offline.

What’s Next

Improving SLAM with more robust algorithms.
Expanding the mesh network for multi-helmet deployments.
Moving from a web-based HUD to a true helmet-mounted display.
Reducing hardware/networking complexity by consolidating into a single PCB and unified compute module.
Exploring microwave or alternative wireless signals to transmit mesh network data back to the control center.

FireGuard demonstrates how networking + hardware integration are just as critical as AI. By combining LiDAR, radar, IMU, and real-time data streaming into a mesh-capable infrastructure, we showed a path toward smarter, safer firefighting tools.