InsureSight

Where the claim output would be (spreadsheet), with your photos at the top in a scrollable view
The camera screen where you can take your claim pictures.

Inspiration

After the hurricane a parent's friend asked for help creating a list of all the things they thought they would have owned using AI since they were low on time and didn't have any pictures beforehand. Additionally, during Hurricane Helene our fridge's power went out and I heard that if you took a picture and listed all the items in your fridge, you could get a refund for the items you lost. If I had this tool, I could've done these tasks much faster and easier.

What it does

The user takes pictures of throughout their house of anything they would want to make an insurance claim of, the app recognizes all the objects shown in each picture and creates an insurance claim automatically containing all the objects the user took pictures of.

How we built it

Used react native to create a mobile app for iOS and android devices. Email template creation and excel spreadsheets in python. Pillow (a Python library for handling images) is used to convert images in different formats into data that the ML model could read. It is also used to draw the boxes where the ML model thinks the objects are.

Next, we import a pre-trained ML model that identifies objects in the image. The ML model identifies the objects in the image and saves the images found to a file.

Challenges we ran into

We struggled to get images to load back into the app after taking them. Running the app on teammates computers was difficult with nodejs. We ran into challenges with making an object identification model that could find objects. We had three different options for detecting objects: using an LLM, using a pre-trained object identification model, or using a search engine. We ultimately chose the object identification model. However, this only lets us detect 80 different objects.

We also ran into challenges prompting LLM models to output data in the format we needed it in. The LLM seemed unable to understand our instructions despite being given examples of how it should work. Excel Spreadsheet Updates: Being able to take the llm and incorporate it into the spreadsheet to auto update an excel spreadsheet. Wasn't able to connect the two together, to take in the information from the object detection system and update it within the excel spreadsheet.

Email Automation: Being able to generate new emails with an LLM and incorporate it with the spreadsheet, using tomail to control the email automation. Running into the problem that tomail does not incorporate attachments to the email. Using other methods for email automation arose many problems.

Connection Issues: Being able to connect the backend with the front end. Taking a Jupyter Notebook and implementing it into a python file so the LLM used can be accessed, and further used to access the notebook with email automation and spreadsheet updates.

Watson-AI Implementation towards the object detection system.

Accomplishments that we're proud of

The object detection system was a big hurdle to clear that we succeeded in. Camera component and saving the images to a folder in the user's camera roll was more convoluted than I had thought but I was able to get it to work. Set up spreadsheet and email automation.

What we learned

Learned how to use react navigator, and greatly deepened my react knowledge since I had never made a complete app from scratch before. Learned regex, how to use Jupyter, using flask, panda, and TensorFlow, nunpy, PyTorch, how to use watsonx models.

What's next for Hackathon 2024

Integration with an LLM, incorporating LLM into email automation, adding attachment of spreadsheet to email, integrating object detection with google lens, connecting the backend with the frontend for a complete app. App integration with the spreadsheet (Insurance claim).

How we collaborated between departments

We talked about career relation towards the application with an Aerospace Engineering major. For example, he mentioned how it could be used to created a spreadsheet or database of the inventory of the airplane hanger more easily before a hurricane or natural disaster in the case of an emergency.