Safe Drive AI

HackDavis 2024

Mission: Reduce car related injuries and deaths

Use OpenCV, PyTorch, and Intel Developer Cloud to create and optimize Lane Detection, Driver Alertness Detection, Blind Spot Monitoring, and Proximity Sensing to make the cars on the road safer.

Our project for https://hackdavis.io/event 2024.

Parts of the whole system

Lane Detection

Use a forward facing camera and OpenCV to recognize lanes and alert the driver if they do not stay within their lane.

Driver Alertness Detection

Use a driver facing camera and OpenCV to detect if the driver is awake and paying attention to the road

Blind Spot Detection

Use PyTorch and Intel Developer Cloud Notebook to detect pedestrians walking infront and next to the car and alert the driver if they get too close. We were able to use Intel's ipex, PyTorch plugin, and bfloat16 to reduce the training time by 4 entire minutes. We used ipex from Intel AI and the Intel PyTorch plugin to leverage Intel AMX.

ipex example usage

import torch
import intel_extension_for_pytorch as ipex
from engine import train_one_epoch, evaluate
import datetime

# ...

params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(
    params,
    lr=0.005,
    momentum=0.9,
    weight_decay=0.0005
)

lr_scheduler = torch.optim.lr_scheduler.StepLR(
    optimizer,
    step_size=3,
    gamma=0.1
)

model = get_model_instance_segmentation(num_classes)

model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.float32)
model = model.to(device)

Optimization Experiment:

Start time: 2024-04-27 16:27:36.983459
End time: 2024-04-27 16:47:38.238154

datetime.timedelta(seconds=1201, microseconds=254695)

Start time: 2024-04-27 17:46:16.191538
End time: 2024-04-27 18:02:06.233684

datetime.timedelta(seconds=950, microseconds=42146)

Start time: 2024-04-27 17:02:22.302090
End time: 2024-04-27 17:18:17.402227

datetime.timedelta(seconds=955, microseconds=100137)

As you can see, intel ipex and bfloat16 saved us 4 entire minutes for this model. Imagine a much larger model and the type of time and compute cost savings that could be achived.

Intel LeaderBoard

We submitted a custom bfloat16 model to the intel leaderboard. Here is the model fine tuned from Gemma

2024-04-28_09-50

Use of LLMs and implications of hallucinations

We used LLMs to generate and extend data that we used to fine tune our text directions Gemma model. Hallucinations for directions in this case could cause people to go to places that do not exist. This could be fixed by using existing databases of locations (Like a mapping API) to insure that the LLM is always directing someone to a real place.

Proximity Alert

Use an Arduino and an ultrasonic distance sensor to alert the driver if they are too close to anything, including another car. We alert the driver by playing sounds and showing lights so they can take action.