-
-
Picture of testing setup.
Inspiration
The main inspiration for our project comes from crisis situations, such as natural disasters and war zones. Search and rescue operations are crucial in these environments in order to save lives. An important part of an effective search and rescue operation is the ability to quickly and effectively assess a situation and decide how to proceed. However, these natural disasters can present dangers to the search and rescue teams, as the harsh weather conditions and unstable terrain can pose a risk to their safety. We wanted to create a system that can collect data in real time that the search and rescue team can use to assess the situation without risking their safety.
What it does
When the autonomous vehicle is deployed, it first assesses the crisis area by moving around while avoiding any obstacles. Simultaneously, it provides a live feed of the area to another device for the operators the view the surroundings remotely. This live feed provides annotations of key features such as humans and other objects using a YOLOv8 model. After surveying for a period of time, the system automatically switches into panoramic mode where panoramic images are stitched together using real-time image data. Once it completes a full circle, it will go back to moving around and showing the real time feed in surveying mode. The autonomous navigation and camera feed are run simultaneously via threading to improve speeds and performance. For object detection, we decided to use a YOLOv8 model to annotate what the camera sees in just a few milliseconds! A robust data transfer system using the socket library performs the server-client connection that allows for the transmission of arducam data.
How we built it
crisisvision is the culmination of over 20 hours of programming and testing. Some of the key technologies that made this project possible include threading, opencv, yolov8, and sockets. We performed several unit tests on the image processing and control functionalities and then integrated the two components together.
Challenges we ran into
The first challenge was setting up all the hardware. Our Raspberry Pi often crashed when we were logged into it due to the network blocking it from accessing anything on the web. Luckily, we were able to resolve it by using another network.
The second and biggest challenge was integrating the code and testing it on the Raspberry Pi. When we were working on the server-client connection to stream video, we were able to get it to work between Windows laptops. However, when we tested it on the Raspberry Pi, we ran into issues that had to be debugged and resolved. We tried a lot of different methods to fix the issues, and while we were able to resolve them, it was a huge setback for us.
The third challenge was optimizing the code. The working draft of the code did not have a very high frame rate. For our solution to be used in a real natural disaster event, the feed must be able to have a higher frame rate. Therefore, we had to optimize the code to reduce the number of computations.
Accomplishments that we're proud of
- Created a vehicle that could effectively navigate unknown environments autonomously.
- Created a real time feed with sub 20 millisecond annotations using the YOLOv8 model.
- Optimized the code using threads and image processing techniques which allowed for the arducam to output a video that was at a respectable frame rate (~5-10x faster FPS).
- Successfully worked as a team to integrate low-level software with high-level image processing tasks.
What we learned
One important lesson we learned was that version compatibility is very important between server and client devices when using Python sockets and more generally when developing embedded solutions. We also learned that having a general understanding of a processor's architecture can prove to be useful when trying to optimize software. Finally, understanding how to read the source code of complex libraries proved to be important in developing an efficient solution.
What's next for crisisvision
Our vision for this product moving forward is to continue pushing the boundaries of the program's speed while maintaining reliability. We hope to integrate our system with mechanical components that can assist in moving hazardous debris and objects. Some possibilities for optimization include using ARM SIMD operations if we continue to use the Raspberry Pi, potentially switching to the NVIDIA Jetson Nano for better AI inferencing, and migrating to C/C++ for better overall efficiency especially in image processing.
Log in or sign up for Devpost to join the conversation.