Luminous

UI when a user started our extension
UI when a user starts speaking

Inspiration

The inspiration behind our innovative app stemmed from a close friend's experience, who is visually impaired. He faced significant challenges in understanding visual content during university lectures, as the images and diagrams used were not accessible to him. This struggle highlighted a broader issue: the need for inclusive technology that bridges the gap for individuals with visual impairments in educational and professional environments. Recognizing this, we set out to create an app that could transform the way visually impaired individuals interact with visual content. By allowing users to take a snapshot of any screen or picture and then ask questions about it using voice commands, our app leverages advanced AI to provide audible answers. This approach not only empowers our friend but also extends the benefits to a wider audience, fostering greater inclusivity and accessibility in learning and information sharing.

What it does

Our app revolutionizes learning by enabling users to capture screenshots from videos and inquire about the captured content through voice. When a screenshot is taken and a question posed, our software integrates seamlessly with OpenAI's ChatGPT-4-vision model for analysis. It generates a comprehensive textual response, which is promptly converted into audio using sophisticated text-to-speech technology. Audibly relayed back to the user, this feature delivers an interactive learning experience, enhancing comprehension and retention of educational materials.

How we built it

We developed a Chrome extension utilizing a blend of JavaScript, HTML, and CSS to create a user-friendly interface. Complementing this, we constructed a dedicated server tasked with processing both screenshots and user queries. This server operates on a Flask application framework, meticulously coded in Python to ensure efficient handling of requests. For the storage of all screenshots and audio responses, we have integrated Azure Storage, a robust and secure cloud storage solution, ensuring the safekeeping and accessibility of data.

Challenges we ran into

Efficient Data Management: Early on, we grappled with how to handle the influx of screenshots and audio files. Strategizing for efficient storage and retrieval became key.
Privacy and Security: The app deals with sensitive data, so we prioritized user privacy. We focused on implementing robust encryption and developed strict privacy protocols to build trust and secure data.
Real-Time Processing: Users expect quick feedback. Perfecting the balance between a high-performance backend and optimized algorithms was challenging. Our efforts here aimed to deliver a responsive and uninterrupted experience. ## Accomplishments that we're proud of
Seamlessly integrated advanced AI technology from OpenAI ChatGPT-4-vision model to interpret visual information and provide accurate, context-aware responses.
Successfully navigated and optimized text-to-speech technology to convert text responses into clear, natural-sounding audio, making information consumption more accessible.
Created an intuitive user interface for our Chrome extension that can be easily navigated by all users, regardless of their level of technical expertise.
Built a scalable architecture using Azure Storage that can handle growth and increased demand without compromising performance.
By doing this project, we can help more visually impaired students to learn online courses and connect them to higher quality online courses.

What we learned

Gained in-depth expertise in leveraging AI technology for solving accessibility issues.
Refined our skills in developing Chrome extensions, learning about the intricacies and best practices in browser extension development.
Deepened our understanding of cloud-based storage solutions and how to efficiently manage large sets of data.
Learned the nuances of real-time data processing and how to optimize for performance without sacrificing the user experience.
Understood the art of remote collaboration, leveraging various tools and methodologies to ensure effective communication and productivity among team members.

What's next for Luminous

Refining User Experience: We are committed to simplifying the app interface further, ensuring that users of all skill levels can navigate it with ease. This entails refining voice command detection, simplifying workflows, and incorporating user feedback to make Luminous more intuitive.
Boosting Data Privacy and Security: We understand that our users entrust us with their personal data. We will continue to advance our security measures, such as implementing end-to-end encryption for screenshots and audio files, and ensuring our compliance with international data protection regulations.
Enhancing Processing Efficiency: Real-time feedback is crucial for our users. We are dedicated to optimizing our server architecture to reduce response times, ensuring that the processing of screenshots and questions is as swift as possible.
Expanding Browser Compatibility: Our goal is to increase the accessibility of Luminous by making it available across various browser platforms. Successful listing in the Microsoft Edge extension store will mark the beginning of this expansion, with plans to extend our reach to Firefox, Safari, and other popular browsers.
Community Building and Awareness: Engagement with the visually impaired community and advocacy groups is a priority. We hope to build a strong community around Luminous and raise awareness about the importance of accessible technology in education and professional development.
Fostering Inclusivity in Technology: In the long term, we aim to broaden the scope of Luminous beyond education, to assist visually impaired individuals in various aspects of daily life. By continually refining the app, we envision creating a more inclusive digital world where visual content is accessible to everyone.