Hat Or Not

Discover whether you should be wearing a hat or not. Our AI recommendation engine determines whether your outfit, environment and situation warrants wearing a hat.

Comment

Hat Or Not

Inspiration

  • Should you be wearing a hat or not?
  • Would your evening dress be improved by a fascinator?
  • Should your friend take off his silly bobble hat?
  • Would your deerstalker go with this lumberjack shirt?

Find out with Hat Or Not!

What it does

We have created an AI recommendation engine, which can take visual and environmental stimulus combined with context retrieved from external sources to make an informed decision and recommendation. The AI engine has many purposes, everything from automatically finding ingredients for a recipe, to recommending outfit choices, to providing tips for purchasing a car. Our implementation focused on solving the age-old question of whether or not one should wear a hat.

Hat etiquette has deep ancient traditions, dating back more than 5,000 years at least to when Egyptian Pharaohs were easily identifiable by their grand bejewelled golden headdresses. Today, etiquette remains an important and fundamental part of human interaction and hats are very much a statement of one’s purpose and one’s personality. The first application for our AI engine is to quantify this hat etiquette through intelligent determination of whether a subject’s hat or lack thereof is suitable for the outfit, season, time of day, expected weather/temperature and current fashion trends.

What we created

Our project is multi-faceted because we all wanted to branch out and learn and integrate new technologies.

1) A "Hat Or Not" mobile app which can take a picture of a person, identify their clothing, and determine whether they should wear a hat today or not.

2) A portable camera using an embedded microcontroller system (ESP32) which can take pictures of people and the environment and determine what people are wearing and determine if they need a "Hat Or Not".

3) An API for executing the ML models and handling requests from the portable camera and the mobile app. The mobile app and portable camera work by simply taking a picture and uploading via the API, then awaiting the response.

The API then delegates to a python program to run the ML on the image to 1) identify the types of clothing, 2) identify the colours of clothing 3) identify the properties of the person (i.e. male/female, etc.). The outputs of the machine learning models are combined with results of scraping fashionable data sources, such as magazines, websites and current catalogues allow us to determine what is a recommended suitable outfit to go with the items of clothing that were photographed. The data sources include fashionable hats and related outfits, including associated metadata such as whether the outfit is for summer, winter, rainy days, etc. and the same data is associated with the hats. We sourced our own data to train these models as hat-related models are hard to come by. The output from the API with the metadata looks like:

{
  "shouldWearHat": false,
  "isWearingHat": true,
  "clothingColour": "#1a2bf8",
  "clothingColourName": "Deep Blue",
  "hatMetadata": {
    "season": "winter",
    "type": "unisex white baseball cap",
    "usage": "sport"
  }
}

The above output is telling us:

  • the person is wearing a hat
  • the person should not wear a hat
  • metadata about the hat that is currently being worn, if a hat is being worn

4) ML models for determining clothing from pictures and recommending outfits. We trained four ML models, including the RESNET50 model which we trained on 5,000 different images for hat detection and 30,000 different images for clothing classification to generate embeddings. Using embeddings was ultimately critical to finding the best match. The colour detection model was trained on 865 different colours, and required colour correcting with histogram equalisation to fix inconsistent lighting and shading on photos taken in the real world.

5) A database for storing user history and tuning recommendations. We used a MongoDB instance within AWS for storing the user's history, including previous items of clothing, preferences, and environmental data. This is all used to tune recommendations. We take the known items of clothing to determine style and fashion preferences, and also use types of clothing and manufacturer information (such as environmental impact) and put all of this into the recommendations.

The database also provides a full history of the user's actions, allowing them to revisit previous recommendations.

How we built it

Our tech stack

Embedded:

  • ESP32 (WROOM-32D, dual-core)
  • ArduCAM OV2640 2MP Mini Plus
  • C++

Mobile App:

  • Kotlin/Java
  • Android Studio

API:

  • WebAssembly (JavaScript)
  • Suborbital (serverless framework)

Artificial Intelligence/Computer Vision:

  • Tensorflow
  • OpenCV
  • Keras
  • (with some help from Resnet models)

Database:

  • MongoDB (AWS)

Challenges we ran into

The project was incredibly challenging as we were very ambitious with trying to create so many tools on so many different platforms. Each team member took responsibility for a different section, so we had to coordinate a significant amount for testing, and try our best to ensure the relevant sections of our programs were ready when the others needed them.

We had a few specifically challenging events during this hackathon:

  • Tyler, who led the ESP32 development, managed to break the espressif/esp32 toolchain, such that he could no longer compile code for the board. The error related to using the incorrect number of parameters when calling std::swap which indicated an error well beyond the scope of our code. We never did figure out how this happened. We figured out this was due to the swap macro being redefined by the ArduCAM library. Once we edited and recompiled the library, the issue was resolved. Broken espressif/esp32 toolchain
  • Creating our own data sets since we are focusing on headwear and there are not very many available data sets relating to headwear identification so we had to create our own datasets to train models.
  • Broken data sets were a real problem, for the few datasets we were able they contained corrupt images and data and required that we re-started the training multiple times, often after already partially processing thousands of images.
  • Sending raw binary was difficult because most REST frameworks and clients expect JSON/string data, or even form-data for files. However, for the most efficient payloads we decided to transmit the images using raw binary, which required a fair bit of extra support to convert images to raw bytes and then figure out support for the sending/receiving frameworks.
  • Suborbital is too new, it is fully in beta and has limited documentation, especially for JavaScript (by default suborbital uses rust). Even the uuid JavaScript library could not be used because is has no support for crypto-RNG yet. Furthermore, the API interface is very limited and implementing headers and binary data was a completely mystery. However, we were determined to use it due to the potential for runtimes which are compatible with cloud and embedded systems.
  • We were really tired…
  • For some reason this DevPost took me hours to write, people kept distracting me… and the ice cream was awfully distracting too
  • Android does not have a native camera app so we had to create our own camera app in order to implement overlays/masking on the camera.

Accomplishments that we're proud of

We are very proud of the fact we delivered on such a complex and large project in only 36 hours. It sounded impossible when we started, it felt impossible that we had actually made it work when we finished.

The project was very large and it required every single person on the team not just pulling their weight, but committing to a very ambitious workload and timeline. The above list of challenges we had is large, but we didn't let anything stop us and always worked together to find good solutions.

With hard fought projects like these it can often strain the relationships of the team members, however, I believe we all thoroughly enjoyed the experience and had an amazing time together throughout the hackathon; and we have all gained friends throughout this hackathon.

What we learned

We learnt many new technologies, especially WebAssembly--and how to compile it, along with the suborbital serverless/embedded framework. We also learnt that ambition can pay off, and not to shy away from challenges.

Impact & what's next

We built a complex AI recommendation engine, not just a hat app. The software has many pratical and impactful applications, for example:

  • Ensuring workers on a construction site are all wearing hard hats, softy goggles, high visibility jackets, etc.
  • Ensuring people who are using an employee-only or school entrance are wearing the appropriate uniform.

Next steps is to use the recommendation engine for other purposes:

  • Automatically finding recipes and ordering ingredients for dishes, just tell the app what you want to eat for the week and the ingredients in the correct quantities will be shipped to you
  • Recommending vehicles based on lifestyle and location analysis
  • Many more!

Built With

Share this project:

Updates