GetRecyclued

Educate and Clue Kids In On Recycling using Object Detection Transformers plus Vision and Text LLMs!

Comment

Inspiration

GetRecyclued is inspired by my desire to "clue in" (educate) my kids about recycling. They're in elementary school and they haven't been taught about recycling at all, so at least where I live (Canada's biggest city) it's up to the parents to educate them. I think it's important for kids to be aware from a young age about environmental issues and how recycling benefits the environment. This Hack the Future hackathon provided me with the opportunity to create an AI app for that purpose and educate the next generation.

What it does

With GetRecyclued, kids (and adults) can snap a photo and upload it to the app to get "recyclued". The app detects the objects in the image using a machine learning transformer (Facebook's DETR ResNet-50 model) as well as a vision LLM (Llama 3.2 11B Vision). Using the bounding boxes returned by the DETR model, the app draws outlines (in blue) around the objects in the photo which can be recycled. How does the app know which of the objects can be recycled, if any? It uses a text LLM (Llama 3.1 405B) to take the named objects returned by the DETR and vision models and determine the recyclables. These are the objects which are both outlined and labelled in the "recyclued" image which then gets displayed. If there are no recyclables, the original image is displayed.

Next to the image, the names of all the recyclable objects are displayed in a word cloud. If none, then "no recyclables" is displayed. Both the image with the blue outlines and the word cloud can be downloaded and saved. Last but not least, the text LLM (Llama 3.1 405B as mentioned) is called upon again to generate an explanation for each recyclable describing the benefit to the environment of recycling it, with a summary of the overall positive impact for the planet. The summary is then read aloud by the app using text to speech. In this way, kids can learn about recycling through words, audio, and pictures.

What's next for GetRecyclued

For the next version of GetRecyclued ("G.R. MARK II") if I can raise enough funding, I want to build a robot version which can snap an image of a pile of stuff my wife complains about, scan it for the recyclables, pick those out to dump in the recycling bin, and on the way back tell my kids how recycling that stuff benefitted the Earth. Won't that be chill? Of course :)

Built With

  • gradio
  • llama-3.2-11b-vision-instruct
  • meta-llama-3.1-405b-instruct
  • object-detection
  • pillow
  • python
  • sambanova
  • text-to-speech
  • transformers
  • wordcloud
Share this project:

Updates