Reel-It-Back

Converting Virtual Content to Actionable Experiences - Instagram Reels Mapping

Comment

Inspiration

Thinking about self-improvement, the first thing that comes to mind is growth. And when reflecting on situations where we’ve grown, it’s really from combining Theory with Action which lead to experiences. When we go through school and work, we combine the theory we learn with experience on the job to improve ourselves professionally. However, many tend to stagnate on their personal growth outside of work which results in weekends going unplanned leading to unfulfillment, the feeling of lack of productivity, and reduced experiences. But in a way, you can draw a parallel that Instagram is the theory that gives you the knowledge that you end up sending into group chats such as new activities, travel destinations, and even restaurants that are hidden gems in an unexplored neighbourhood. However, current Instagram features fail to bridge the gap to the action. What we’ve noticed through interviewing people across the globe is that actionable content quickly gets lost in the noise from other content being sent into a group chat. And even when you find the content, the next hurdle is aggregating the addresses by constantly pausing videos and manually extracting addresses from descriptions.

What It Does

Introducing Reel-It-Back, your group itinerary planner that brings what users see virtually into real-life memories. Our goal is to transform you from a passive consumer into an active participant in real-world activities. How Reel-It-Back works is that you invite a bot account into your group chat with friends and the rest of the output is all automatically populated onto our website.

The user will be presented with a map which geographically marks all the activities and spots, allowing you to easily zoom into the geographic area of interest (for example, your current city, a neighbouring town, or somewhere on the other side of the world). In addition, the specific Instagram Reels associated with the location are accessible, allowing you to revisit the original content. This solution also has built in confidentiality features by ignoring all texts in the chat and only filtering for lines with attachments (aka the Reels). By consolidating and organising all potential options in a single pane of glass, it greatly reduces the effort and activation energy of locating and planning new experiences, breaking unproductive streaks, and leading to personal growth through increased action in our personal lives.

How We Built It

Step 1: Automated Instagram Export Function - Because Instagram does not have an API to export data nor have there been any existing projects beyond liking posts or setting automatic DMs, our team needed to build an automation function from scratch. What this involved was trying Puppeteer for the first time and automating the steps of data exporting on a dummy Instagram account. With this we are able to convert the chat history of links into HTML code. We also used ChromeDriver for the first time to automate credential inputs to access Instagram and automate clicks to the export page. Through the automation we can download the Reels from the bots’ chats on a daily basis and add the net new content into a master database. This data would then be saved into Supabase.

Step 2: Converting HTML Code Into Categorizable Data - Once an HTML file is uploaded into our database, we have an automatic trigger to convert the HTML file to a Google Docs file. Within the Google Docs file, the data for each Instagram Reel will be outputted as a Description, Poster/Influencer, and Instagram Reel Link. To parse through the HTML files, we will use an LLM to look into characters/words and extract the address of the location in the Reel.

Step 3: Using AI to Classify The Type of Content - We used the Groq LLM to interpret the text and create tags associated to the video content (such as food, travel spot, poses, etc.). Groq will also search for the location or address from the description or the data pulled from Hugging Face’s video classification models.

Cloud Tech - Used AWS SAM framework to leverage Infrastructure-as-a-Code and serverless design. Uses DynamoDB and AWS Lambda. Provided the front-end website with computational power and storage which was used for our database. Also used Langchain and FastAPI to get a pipeline to process LLMs.

Step 4: Website Map Output - The data from the database will be mapped onto a Maps display with markers. By using Google Maps API, we are able to display markers of the locations fetched from the database along with the metadata/data associated with the locations. It will also add a preview of the Instagram Reel associated with the location based on the saved data.

Challenges We Ran Into

Our team is proud that we overcame a significant knowledge gap over the two days of working on the project.

We learned how to make automation functions from scratch as there were no developer APIs that Instagram and Meta provided that could export user-selected Instagram Reel URLs into data which includes the description, influencer/poster account name, Instagram Reel URL, and time stamp. This was a great opportunity to try automation using Selenium and Puppeteer for the first time to automate web tasks on Chrome. Within this aspect of the project, we learned how to structure the frequency of login and export requests to prevent being flagged for bot activity. It taught us the trade off of sacrificing speed for proper delivery of an output which was further tied to the urgency of our user’s needs. Instead of being in a dilemma about which to use, our business/user use case made it clear to optimize for speed with Puppeteer.

Once we had the proper data, we wanted to experiment with large language models to classify data to derive the type of content (food, travel, experiences), derive the geographical address of the mentioned locations, and create tags for future vector database capabilities. Our team used multiple classification models such as Hugging Face video-text-to-text for the first time in combination with the Groq LLM. By using these pre-existing models, we learned how it simplified information identification from multiple sources and triangulated commonalities. Use Langchain to also correct/make sure the audio is accurate.

Trying to make the Map Overlay on our web application is difficult as the Javascript Google Maps API is not interactive but it does allow us to create our an action item nor output our own data on top of our marker. There is also no active repository or files that are accessible for the output. However, our team got creative with the mentors and searched the WayBackMachine to find an older version of the page for the original action item files.

Accomplishments That We're Proud Of

What our team noticed is that many individuals don’t only use one social media app as they can span to TikTok, Facebook, Lemon8, XiaoHongShu, SnapChat, X, and BeReal. By creating a separate solution, it makes it difficult for these applications to copy it immediately as it would essentially be a centralized platform to recommend activities. It will be a content-based Google Maps that takes into account people’s locations to show where to go as Google Maps fails to recommend niche spots. We believe that we have an edge against Instagram and Google when it comes to developing the app. From conducting our own primary research and creating test outputs manually, we have gained traction on demo’ing the app from over 200 people. This is backed by the user viewing history on a sample output and surveying users. https://www.google.com/maps/d/u/0/viewer?mid=1F7toBFiw5CEN1UDlpbFrKgTunRVHM1Q&ll=36.47367805488249%2C128.62878071255915&z=3

With Instagram, the business is focused on expanding Instagram Reels into capturing more female users currently on TikTok which prioritizes growth and usage. This pushes Instagram to prioritze improvements to their discovery and search function to mimic or replace capabilities from TikTok. Our project would be counterintuitive towards their strategy by encouraging users to head offline to experience the content in real life. This step is also one degree further from Instagram’s current business as they monetize on ads for viewing content.

Looking at Google’s strategy and their annual report, their primary business driving growth is from Google search engine and YouTube ads. These business lines would most likely push higher product budgets at Google cannabilizing spend for Google Maps and restrict their abilities to push out an aggressive product feature. Also observing Google Maps’ growth over the past five years, the product has remained more stagnant with failed features such as the “For You Tab”.

What We Learned

Our biggest lesson came as a surprise. For our team dynamic, it was very unique having one business/commerce teammate that had very minimal technical knowledge and the rest as developers. Our team learned how to breakdown manual processes into even simpler tasks which then can be created by developers. We learned that there were significant benefits in having a diverse team where our business teammate focuses on idea generation storyboarding, and designing while working with each developer to ensure consistency in the final product.

When we first identified the problem with the social media usage, our team was very excited to understand the technical issues people encounter that result in a feeling of unproductiveness. Because of the team’s excitement, we trialed an unprecedented method of validating the problem by conducting primary research. Over the first night, our team conducted eleven interviews with young adults across the globe to observe their social outing plans, social media behaviours, and frustration they faced on social media platforms. This resulted in our discovery that many people had loads of executable experiences that gets “stuck in the chat”. Due to this discovery, we began to cater our idea to group chats on Instagram and reducing the friction of bringing their virtual desires into life.

This experience has been a massive step outside of everyone’s comfort zone as this hackathon was not only some of our first or second hackathon, but the diversity of each teammate’s background and unfamiliarity with each other pushed us to aim for ambitious goals.

What's Next for Reel-It-Back

Our team has many features we aim to complete in the near term which can be separated into key next steps accounting for school and co-op workload (assumes Week 0 is the hackathon weekend).

1. Mobile Application (Week 1 & 2) - Due to our team’s current expertise and the complexity of the backend output, our team focused on creating a web application that would contribute to the foundation of our mobile application. What we believe is that users will find the best experience on the mobile platform given the convenience of accessing their phone at any time and place, continuity in using their phone for social media, and push notifications to notify users of nearby locations related to their saved content.

2. Itinerary Recommendation & Initial User Testing (Week 2 & 3) - Our team aims to make the product incorporate more discovery features by using vectors to identify similar experiences near the user’s proximity. We would then anticipate rolling out the initial prototype to the eleven interviewees to get product feedback.

3. Language Translation (Week 3) - We plan to incorporate a translation API and/or an LLM to help reinterpret languages into the translated language of choice. This may also require us to use a speech-to-text language LLM to convert short videos where people are speaking in a different language.

4. Expansion to Content Not Related to Location (Week 4 & 5) - Based on surveying 23 young adults over the past two days, we noticed that people send cooking recipes, dance moves, photo poses, and outfit recommendations to the group chats and may not be properly showcased on our application.

5. Expansion to Adjacent Use Cases (Week 6) - Reel-It-Back can benefit influencers by assisting them in discovering less popular locations creating differentiated content or pinpoint hot spots where lots of content creators are also covering. Alternatively, it could be a strong management tool for businesses to identify their online publicity or social presence adding another long-term layer to the sustainability of the product.

Built With

Share this project:

Updates