Garbage Classification Model

Trash needs to be sorted better. How can we use computer vision and natural language processing to come up with a user interactive web app to promote easier recycling

Comment 1

Inspiration

We've all been there: ambitiously sitting down to dig into PAR's stir fry but 20 minutes in realizing that there was in fact no possible way to finish every morsel of what was stacked onto our black plates. So much food waste occurs in our dining halls on campus, but when you step back and think about how much food waste occurs across the nation's schools - that's a lot. And food waste is only one type of waste.

The world generates 2.01 billion tonnes of municipal solid waste annually, with the main waste categories being food/green, paper/cardboard, plastic, glass, and metal. Rapid urbanization, population growth, and economic development will push global waste generation to increase by 70% over the next 30 years. Thus, managing trash is critical to a sustainable future.

But how many people actually take the 5-10 minutes to research which type of waste their trash belongs into so we can manage waste better? That's where Germany, our inspiration, stepped in. In Germany, trash is strictly sorted into one of six bins: Blue (Paper and Cardboard), Yellow (Plastic and Soft Metal), Brown (Organic/Bio Waste), Black (Residual Waste), Red (Special Waste), and Glass (where glass is sorted into brown, white, or green glass). As the leading recycler of municipal waste, their comprehensive waste sorting system was something we digitally implemented so countries outside of Germany could easily follow the same model.

What it does

In our web app, an user can upload an image of their trash. Our computer vision model, which we trained on multiples datasets using convolutional networks can identify a variety of different objects. Then, our program maps the identified object to the specific bin classification it falls under, taking into account bin prioritization for objects falling under multiple categories (ex. red would rank higher than blue). As a fail-safe, the user can also manually type in their trash (our autospell check would ensure accuracy of input) and then it could be mapped to it's classification. Given that there are a ginormous number of words that can fall under each category, we used nltk as a part of nlp to generate a giant database of synonyms and hyponyms to accommodate a wider range of inputs. Then, our web app would display the correct bin that the user needs to drop their trash into.

How we built it

Computer Vision We need to be able to look at a disposed item and tell which of the six categories it belongs in. We developed an Image Classification Model that can sort objects to an extent: such as whether it's a fruit or vegetable, cardboard or plastic, type of electronic, etc. Most interestingly, Germany's trash management system sorts glass into three further categories: green glass, brown glass, and clear (white) glass. So I created an Image Segmentation Model too that segments an image and uses it average RGB value to figure out it's proper color.

Cloud Computing My computer didn't have enough storage for the sheer amount of training images to train the different models, so we ultilized AWS's S3 buckets to store and access our train, test, and validation data! This Cloud Computing solution ensured we could successfully train all the data we needed to given the time constraints of the Hackathon.

Natural Language Processing Main Idea: Utilizing computer vision to analyze a user-inputted image, we generate a predicted name for the object in the image. The specific NLP task we implemented was "named entity recognition", which identifies words or phrases as useful entities. Using this predicted name, we categorized it into the different bins used in the German trash sorting system. First, we imported our necessary packages. For this part of the project, we used import csv, fileinput, autocorrect, webcolors, numpy, math, collections, and nltk.

What is NLTK: NLTK (Natural Language Toolkit) is a popular platform that works with language data. Utilizing semantic reasoning libraries, we reached a logical conclusion based on predicted name to generate synonyms and hyponyms for a pre-created set of potential trash items under each bin category. We did this in order to create a more expansive and inclusive data set that can accomodate a wider range of potential user images.

Special Cases: One of our categories was glass, and in Germany, glass is categorized into green, brown, and white. Using webcolors, we converted given glass color into its rgb int values and cross-referenced these values to approximate which out of the three glass categories it would fit best in. Given that some objects could be present under multiple bins, we created an order of priority to properly sort these objects. And some objects, like paper, is a substring of many different trash items that fit into different bins (such as wax paper, toilet paper, paper towels, etc.). To overcome this issue, we prioritized the bin in which the inputted string exactly matched the data element. Furthermore, since some trash items could be contaminated with organic materials, we created a case where any object that was contaminated would automatically be sorted into residual waste. As a fail-safe in case the computer vision couldn't identify the object, we also created a functionality for the user to type their object. To account for misspellings, we also created an autocorrect function to correct user input.

Front-End Development (React.js, 3D-Modeling, and Flask) For our front-end, we created an interative web-app where users can enter pictures of their trash or a description of their trash, and the output will be the respective bin it belongs too. While our backend code currently isn't linked to this front-end, we plan to learn Flask and implement it to accomplish this task.

Challenges and Accomplishments - what we learned

Challenge #1: connecting our front end to our back end While we had a completed web app and a thoroughly functioning opencv/nlp model, our issue was trying to figure out how to integrate the two pieces together. We realized that using Flask was the best approach to tie our keras models into our web app, but unfortunately the learning curve was a little steep in our allotted time at the hackathon. With more time spent learning and executing our findings, we would be able to get this project full-stack!

Challenge #2/Accomplishment #1: using NLTK to improve our waste classification dataset One issue we were thinking about was that when the user inputs a word to describe their waste, there could be a massive amount of possibilities of what they could input for the same waste. Thus, we figured out that we needed to use NLTK to generate a giant database of synonyms and hyponyms for each element in our original waste classification csv file and write a new file combining our original and newly generated database. We were able to get this working, thus it was a more accurate model for our users to interact and sort their waste!

Accomplishment #2: Using AWS to train our opencv model To train our opencv model, our computer needed to handle massive datasets, but our computer didn't have enough storage for the sheer amount of training images to train the different models. Especially with a bin like brown (organic waste), there were so many possibilities of what compost could be. Thus we used AWS's S3 buckets to store and access these datasets. By using cloud computing, we were able to train and use a more accurate model within our allotted time at the hackathon.

What's next for Garbage Classification Model

  1. Using Flask to link our Backend with our FrontEnd
  2. Categorizing more items with our Image Classification Model(e.g. commonly disposed items like plastic utensils)
  3. Generating a more efficient algorithm to come up with more accurate synonyms and hyponyms for each element in our classification dataset, so we would cover almost all possible waste that could be thrown away.

Built With

Share this project:

Updates