Our submission to Blueprint 2024
As cyclists, pedestrians, and drivers, we've experienced the nerve-wracking feeling of walking down rows of parked cars, hoping a passenger isn't about to open their door into us without looking. Especially during triathlon training, in which cycling is quick on curvy country roads, the danger or car doors can be immense. In a recent three-year span in Boston, there were 1700 bicycle accidents and incidents in the city, and a staggering 22% involved car doors. This 'dooring' behavior is both illegal and fatal, having killed many and injured hundreds more every year in the UK.
Our project combines the tools of Arduino, computer vision, and Python. The project begins with the main window, which uses a webcam (or, when on a car, a camera on the driver side mirror) to look at oncoming obstacles. It categorizes these obstacles by type (person, bike, car, etc), size, and position. Then, it applies an algorithm that determines if it is safe for the driver or driver-side passenger to open their door. If an object is over a certain size, determined experimentally by object type, the door will need to lock. Additionally, the door will be told to lock if an object is moving fast enough to become a hazard in the next few seconds. To actually lock the door, the Python program communicates with an Arduino board and a servo. The servo, which would be installed in the hinge of the car, is turned and locked shut when opening the door is unsafe, and set free to move at all other times.
We began with building the Arduino communication system; we used the arduino-python3 library and its provided .ino arduino file to do so. With the inbuilt servo support of the arduino-python3 library, we could communicate locking and unlocking the servo in our Visual Studio Code python project. The servo is locked and unlocked by attaching and detaching the servo in software. Then, we created a way to display a webcam feed on the user's display. We used TKinter, a Python GUI library, to create the frame for the feed, and used OpenCV to get the computer's webcam input. We then focused on creating a system to identify objects in the webcam stream. By using PyTorch and the pre-trained YOLOv5 image recognitions model, we ran object identification on CUDA hardware. We placed each OpenCV frame into PyTorch analysis, which generated an object type alongside coordinates and size. Note: Credit to Akash Ahnihotri (https://towardsdatascience.com/implementing-real-time-object-detection-system-using-pytorch-and-opencv-70bac41148f7) for a tutorial of setting up PyTorch and OpenCV. The final step is analysis and output. PyTorch gives its output data back to the program, which performs two analyses: static and dynamic. We built the static analysis to work solely on object size; if an object takes up a certain area of the screen based on its location (near the door) and object type (larger for cars, smaller for people), the door locks. The dynamic analysis works by finding the largest human object (bike or pedestrian) in the frame and seeing how its size changes between frames. If the object closes in on the camera too quickly, the door closes and locks. The final lock/unlock signal is then sent to the Arduino.
Our first challenge was getting the Arduino to communicate properly with our computer. After trying to use WiFi communication with an Arduino Giga, we decided on simpler wired communication with an Arduino Uno to avoid issues. In making the image recognition software, we tried to use TensorFlow but reached compatibility issues with Windows, prompting us to switch to PyTorch. Additionally, when we wanted to track images for our dynamic analysis, we began using the SORT library, which applies a Kalman filter to images to track individual objects. This was not functional, however, so we manually coded a system that tracks changes in the largest human object.