Inspiration
Our inspiration stemmed from the desire to bridge the gap between imagination and reality, turning the act of sketching a imaginary character into a dynamic and immersive experience. We envisioned a world where doodles could come to life, fostering creativity and engagement in a whole new dimension. Our goal is to transform art into a gateway for AI-driven learning, where each creation becomes an interactive educator in its own right.
What it does
Our app allows you to sketch a character on your iPad, and the app converts these drawings into dynamic 3D models that are placed before you using Augmented Reality. These models can then serve as educational tools, providing interactive and engaging content with a unique personality that you decide, blending entertainment with learning in a seamless AR experience.
How we built it
Frontend: We developed our app using Xcode and Swift, optimizing it for iOS devices to leverage the powerful iOS ARKit framework.
Backend: Our backend is predominantly built on Google Cloud Platform's VertexAI, leveraging the ImageTextModel for image description generation. This, combined with user-provided personality traits, forms the basis for creating a comprehensive personality for the conversational character. The personality acts as context for a TextGenerationModel, also hosted on VertexAI. Our entire system is orchestrated through Flask APIs and implemented in Python.
A pivotal decision in our architecture was the adoption of Redis as a database to store context for chatbots. Recognizing the challenges associated with sending large text files to Text Generation Models, we opted for Redis to store a dictionary. The value field within this dictionary holds the ongoing communication between the user and the chatbot. This value is continuously updated as the conversation progresses, ensuring that the AI model always has the latest context when generating responses.
The efficient and swift database operations provided by Redis have played a crucial role in optimizing our application's performance. This approach not only enhances speed but also minimizes delays, contributing to a seamless user experience.
Challenges we ran into
Embarking on this journey without prior experience in Xcode or Swift was a formidable challenge for our team. Navigating through the complexities of iOS ARKit and integrating AI elements presented a steep learning curve that needed to be overcome.
Accomplishments that we're proud of
Efficient 3D Model Rendering: Being able to use iOS ARKit and seamlessly integrating it into our app to render dynamic 3D models was a noteworthy accomplishment. This achievement not only adds a visually engaging element to the user experience but also demonstrates the grit and determination of the team to learn something new and execute it the best we can.
Seamless Redis Integration: The successful implementation of Redis for preserving chat context stands out as a significant accomplishment. This feature allows for more meaningful and continuous interactions with the AI, enhancing the overall conversational experience.
AI-Infused Learning Experience: Integrating Google Cloud Platform's VertexAI, including ImageTextModel and TextGenerationModel, has allowed us to create a unique learning experience. The fusion of AI-generated personalities and educational content within the AR environment provides users with an innovative and personalized learning journey.
What we learned
Adaptability is Key: Navigating unfamiliar territories such as Xcode, Swift, and ARKit demanded adaptability. Learning on the fly and embracing challenges allowed us to grow individually and as a team.
Integration Challenges: Integrating various components, from Google Cloud Platform's VertexAI to Redis, highlighted the importance of thoughtful architecture. Overcoming integration challenges enhanced our understanding of system design.
Collaborative Problem-Solving: The complex nature of our project necessitated collaborative problem-solving. Regular communication and brainstorming sessions facilitated innovative solutions to challenges.
What's next for KunstenAR
Animated / AI Generated 3D Models: Animating 3D Models is a complex idea that would be extremely helpful for content creation.
Expanded AI Capabilities: Continuously refining and expanding the capabilities of our AI models to provide even more dynamic and personalized interactions for users.
Community Engagement: Fostering a community around our app, encouraging users to share their creations, experiences, and educational insights. This collaborative environment will contribute to the ongoing evolution of our platform.
Log in or sign up for Devpost to join the conversation.