TrackMyExpense

TrackMyExpense automates receipt scanning and expense categorization using AI agents, delivering instant, clear reports to simplify budgeting and boost financial clarity effortlessly.

Comment

Inspiration

Managing personal and business expenses can be tedious and prone to errors, especially when dealing with numerous receipts and invoices. I was inspired to simplify this process by using AI to automate receipt scanning, categorization, and insightful reporting. With the advancements in generative AI and agentic frameworks, like Google ADK, it seemed like the perfect time to create a solution that would save people time and provide them with clear visibility into their spending habits.

What it does

TrackMyExpense uses AI agents to process receipts and invoices, extracting key details such as merchant names, dates, itemized purchases, and totals. It automatically categorizes expenses into meaningful groups like groceries, travel, or entertainment and generates easy-to-understand reports and visualizations. The tool integrates seamlessly with a user-friendly front-end where users upload receipts, and the backend intelligently orchestrates multiple agents to parse, analyze, and summarize the data, empowering users to track their spending in real time. The "Reporting Agent" creates a concise summary report of the data analysis, which can be downloaded as a CSV file for further analysis.

How we built it

The project was built using Google’s Agent Development Kit, which allowed me to create modular AI agents specialized in OCR, categorization, summarization, and reporting. I developed a Streamlit frontend for rapid prototyping and an intuitive user experience. The backend is deployed on Google Cloud Run, ensuring scalability and ease of deployment. Key technologies include Python, FastAPI, Google’s Gemini language model, and Cloud Firestore for data persistence. Throughout development, I focused on building clear API workflows that connect the agents, enabling smooth data flow and processing.

Challenges we ran into

One of the biggest challenges was managing the complexity of passing large base64-encoded files through API calls and ensuring the AI agents could reliably extract structured data from varying receipt formats. Integrating multiple agents to work in a coordinated pipeline required careful design and debugging to handle asynchronous calls and function chaining. I managed to overcome issues with agent integration by setting up a Sequential Agent workflow.

Deploying and configuring cloud infrastructure to handle the end-to-end processing flow while maintaining responsive user interactions was another hurdle. I also encountered issues with artifact management and session handling that needed custom solutions.

Accomplishments that we're proud of

I successfully built a multi-agent AI pipeline that converts raw receipt images into meaningful expense reports with categorized line items and visual summaries. The seamless integration of front-end and back-end components delivered a user experience that is both powerful and accessible. My solution demonstrated strong adaptability to different receipt formats and reliably produced detailed spending insights. Finally, deploying the entire stack on Google Cloud showcases a scalable, cloud-native architecture leveraging cutting-edge AI.

What we learned

This project deepened my understanding of agent-based AI workflows and how to orchestrate multiple specialized agents for a real-world application. I gained practical experience with Google’s Agent Development Kit, its function calling mechanisms, and handling asynchronous processing in Python. I learned the importance of session and state management in conversational AI contexts. The experience also highlighted the challenges and best practices in securely handling file uploads, cloud deployments, and user-friendly UI design for AI-powered tools.

What's next for TrackMyExpense

Moving forward, I plan to enhance the app’s analytics capabilities by integrating trend detection and anomaly alerts to flag unusual spending patterns. I would like to add support for additional document types, such as invoices and expense reports, from various sources. Improving multi-user support with personalized dashboards and budgeting features is also on the roadmap. I aim to explore deeper integration with accounting software and mobile platforms to extend accessibility. Ultimately, the goal is to make TrackMyExpense the go-to AI assistant for effortless financial tracking and management.

Built With

Share this project:

Updates