DriveInsight

DriveInsight: a system driven by Computer Vision and AI to offer high-quality feedback for evaluating and improving driving. Perfect for all drivers, from beginners to professional.

Comment

Inspiration

Road safety has become an escalating concern in recent years. According to Transport Canada, the number of collisions and causalities has been rising in the past three years. Meanwhile, as AI technology grows exponentially, we identified a niche where we could leverage powerful AI to offer valuable advice and feedback to drivers of all levels, effectively promoting road safety.

What it does

Our system utilizes Computer Vision (CV) techniques and gyroscopes to collect meaningful data about driving performance, such as whether one does shoulder checks on turns or drives when drowsy. The data is then passed to the backend and analyzed by a Large Language Model (LLM). When users want to review their performance and ask for advice, they can ask the LLM to find relevant driving records and offer helpful insights.

Examples of potential use cases are supplementing driving lessons or exams for learners, encouraging concerned drivers to enforce good driving habits, providing objective evaluation of professional driving service quality, etc.

How we built it

Our system consists of
a data collection script (Python, React Native),

  • Runs a CV algorithm (utilizing roboflow models) and streams the output with the video and gyroscope (from mobile) data to the frontend web app

a frontend web app (React),

  • Receives and displays driving performance data
  • Provides an interface to review the driving records and interact with the LLM to get valuable insights Authenticates user logins with Auth0

a backend (Flask),

  • Connects to Google Gemini for LLM interactions
  • Transfers data and LLM outputs between the frontend and the database (Utilizing VectorSearch to extract relevant trip records as context for LLM to generate advice)

a database (MongoDB Atlas),

  • Saves and transfers metadata and analysis information of each driving trip
  • Configured to support VectorSearch

a cloud storage service (Google Cloud Storage)

  • Hosts driving videos which are media data of larger sizes

Challenges we ran into

  • Setting up web sockets to connect individual components for real-time data transfer
  • Configuring Auth0 to perform authentication correctly within the React app
  • Deciding how to store videos (Saving them to DB as BLOB vs Using paid cloud storage service)

Accomplishments that we're proud of

  • Built components that achieve their corrseponding functionalities (Idenify shoulder checks and closed eyes, interacting with LLM, querying database by vectors, etc.)
  • Overcame or worked around errors arisen from using libraries or SDKs

What we learned

  • Collaborating as a small team to deliver quickly
  • Utilizing web sockets for real-time data transfer
  • Utilizing vectorSearch in Mongodb Atlas to query documents
  • Utilizing Auth0 for authentication
  • Connecting various programs and technologies to construct the end-to-end point

What's next for DriveInsight

  • Consulting domain experts (drivers, driving examiners, etc.) for more driving habits to monitor
  • Fine-tuning specialized LLMs for even more robust and insightful responses
Share this project:

Updates