Mission Birthday

Mission Birthday is an easy-to-use food and resource finder for pregnant women and families. The goal is to ensure available food and supplies are provided for children at birth and every year after.

Comment

Mission Birthday is an easy-to-use food and resource finder for pregnant women and families. The goal is to ensure available food and supplies are provided for children at birth and every year after, regardless of individual circumstances.

Inspiration

We believe everyone should have food security and our project strives to make that easier for every family. We also want to help pregnant women find the resources and infant supplies they need to care for their baby from birth and through all their child’s birthdays. We thought it would be fun to create a way for people to easily find the food and supplies they need.

What it does

Mission Birthday has several parts that work together to create an effective resource finder. Organizations and groups with free food and supplies need to be able to easily add their information. The people looking for food and supplies need to be able to quickly search and find resources near them. Mission Birthday features include:

  • Resource Search: Zip code search for nearby food banks and resources
  • Hound Bot: A chat bot that prompts the user for their location and performs a search for nearby resources
  • Provider Resource Submit: A way for groups and organizations with free food or supplies to add their information to the database
  • Autofill Resource Information: Based on a flyer or document, autofill the Mission Birthday resource form

How we built it

Mission Birthday started with each team member working on part of the idea to create a proof of concept. These parts included:

  • Reading text in an image with Azure Computer Vision – Optical Character Recognition
  • Converting the text into a resource by using Azure Language Service – Named Entity Recognition. This let us pull out names, contact information, addresses, and even a list of products from a flyer.
  • Creating a chat bot that could get a user’s location and find nearby resources using Azure Bot and Bot Framework Composer.
  • Developing an Angular frontend website using Ionic and Capacitor so users could interact with each part of the project and take pictures with their camera from either desktop or mobile phone.
  • Creating a backend API for the website to communicate with the Azure Cognitive Services. This is written with ASP.NET Core Web API and uses a SQL Database with Entity Framework Core.

Once we had the initial parts working, we were able to add more functionality to each piece. We also designed and implemented the website UI so users could interact with our services. Then we published everything to Azure. We also added some sample flyers to the website so users can see how the autofill functionality works.

Challenges we ran into

Named Entity Recognition didn’t tag items as well as we expected; for example, street addresses had to be on their own line or the city would get missed. There were also some issues where a word is categorized differently than expected, like “Apples” is an “Organization” instead of a “Product”. Once entities were detected, converting many of them to a usable computer format is a whole other issue. For example, the service would correctly mark a string as a date range, but doesn’t do anything further to split up the range into individual date strings which could be parsed later.

Accomplishments that we're proud of

We were able to successfully integrate each team members’ part of the project with the website and everything worked as we planned!

Using Capacitor and Ionic with our angular site to create a Progressive Web App that can access devices like the camera regardless of running on Android, iOS, or desktop.

ASP.NET Core configuration made it easy to store API keys and secrets in user-secrets files during development (to prevent checking keys into source control). During production, we used Azure App Service to setup environment variables for the keys and database connection strings. ASP.NET Core and Azure helps protect our secret information.

What we learned

We learned a lot about Azure Cognitive Services:

  • How to create an Azure Bot and program it
  • What are Adaptive Cards and how to use them to display data from our API inside the chat bot
  • How to use Computer Vision and Language services from Azure Cognitive Services. We barely scratched the surface of available APIs that we’ll want to explore in the future to make our project even better.
  • From our first time using Ionic we learned a lot about how to interact with its theming engine and components.

What's next for Mission Birthday

First, we would like to use the Bing Search API to collect more data on food banks and related resources so our database has information on more areas. Then we would like to implement some of our ideas on how to make this project even more effective as a food and infant supplies resource finder.

  • Add a search by food or item. This would allow users to find a nearby resource with specific foods or supplies they need, like peanut butter or baby wipes.
  • Create a social media bot to help pregnant women or parents find resources just like our Hound Bot does on websites. This bot could also be used by organizations to advertise their current needs or upcoming special events. For example, a food bank that is running low on cereal can use the bot to let their followers know they need donations or to let people who use the service know that milk is in stock.
  • Packaging the hound bot Angular library as a standard web component so developers using any framework can utilize it on their websites.
  • Containerization of the various system components to allow for easier scaling.

Try It Out Instructions

Mission Birthday is available by visiting missionbirthday.azurewebsites.net.

Find a Resource

  1. Use the zip code search to quickly find existing resources. For best results use 97754, 97756, or 89503.
  2. The Resource Search Results page will display organizations and their contact information.
  3. Click the Mission Birthday title to go back home.

Add a Resource with Autofill

  1. Click the Add a Resource link on the home page.
  2. Click the Autofill from Sample Image button. Or, if you have a camera and a flyer handy, you can take a picture of it using the Autofill button.
  3. Select an example image. The website will process the image and fill out the resource form.
  4. Review the resource form and make any desired changes.
  5. Submit the form when satisfied. Validation errors will show if information is missing.
  6. Click Mission Birthday to go home and enter the zip code to see the new resource.

Find Resources with Hound Bot

  1. Click the Start Chat button on the home page. The Hound Bot will appear and introduce himself.
  2. Say “hello” or ask him to find food or supplies.
  3. He’ll probably ask for your zip code to get your location.
  4. After providing a zip code, he’ll tell you about the closest resource.

Built With

  • .net5
  • adaptive-cards
  • angular13
  • asp.net-core-web-api
  • azure
  • azure-bot
  • azure-cognitive-services
  • azure-computer-vision
  • azure-language-services
  • azure-web-app
  • bot-framework-composer
  • c#
  • ef-core
  • ionic6
Share this project:

Updates