Journey

Figma design wireframes
The user is asking Hagird out to lunch. Quesitoning further about dragons and their existence.
Ron is asked by the user what is muggle is.
The user confronting Voldemort telling him, "I will defeat you!".
The story is Hunger Games. The user engages in conversation with Katniss.
The user is conversing with the main characters at the end of the Hunger Games novel. They ponder about the future of the story.

Inspiration

Our team is made of story lovers. For as long as we can remember, we have been seeking to consume and create stories wherever we go. As we discovered our vastly different reading methodologies due to dyslexia, varying native languages, and differentiated learning experiences, we sought to create a platform that allows all adventurers to immerse themselves in stories.

Journey was created from a desire to enhance users’ reading experiences from the very beginning: childhood. During our ideation process, our research indicated two major findings: 1) Children who were regularly read to showed significantly higher levels of reading comprehension in developing years. 2) Time spent reading decreases as children age because they expect to comprehend harder-to-read stories less. This is especially prevalent when children are transitioning from picture-books to novels.

As we contemplated ways to immerse children in worlds of literature, the idea of integrating AI was presented as a powerful catalyst for engagement. Educators could use this tool to bring stories to life and allow students to clarify their questions first-hand, thus transcending traditional reading methods and innovating the educational experience.

Calling from our personal experiences, our early ideas involved using AI generated text-to-speech to deliver literature to students struggling with dyslexia or English as a second language. However, after many rounds of idea and product development, we are proud to deliver much more than simple text-to-speech. We are proud to have implemented our personal stories into aiding others in developing their own stories as well.

What it does

Journey enables readers to interact live-time with book characters as they explore stories. Customizable conversations allow users to gain explicit insights regarding plot, settings, and even characters’ opinions of each other. Journey’s interactable characters are unlocked in time with their introduction in book—in other words, readers get to meet characters such as Ron Weasley at the same time as Harry Potter!

Journey’s mythical interface and speech-to-text features create an exciting visual and auditory experience. Watch as settings magically appear as you read, listen to Journey’s storyteller weave your favourite tales, and experience the beauty of a great book.

Journey will break down daunting chunks of text into bite-sized pieces and allow users to enjoy a comprehensive reading experience. By simply uploading a PDF book file, you can start your reading journey!

How we built it

Development started by fine turning AI prompts based of story context, who the user is roleplaying, and who the character is to perfectly match the kind of response we had in mind. We aimed to generate creative responses that conveyed information relevant up the point in the story the reader is at while mimicking emotions of the character the AI is portraying. An omniscient narrator is also available for questioning by the user. The narrator knows everything about the story and provides and unbiased an objective view on the story. This can be used for clarification or a brief summary of the story.

To gather the material, novels were obtained in pdf form and were parsed through and split into pages. The pages are displayed main application interface, providing the opportunity interactively parse through the pages in the list. The AI intelligently knows where the reader is when selecting a page based on its content; thus, only allowing the user to talk to a character that has been introduced up to that page.

Journey’s visual element was brought to life with the aid of Open AI’s Dall-E, an image generation tool. Each character and page background has a unique generation made by the AI through a simple API call. Each avatar is displayed next their character under the ‘Character Name’ section.

Challenges we ran into

Integrating each feature and tool we used into one cohesive product (TTS, character generation, string parsing, character responses, and background art generation)
Staying conscious of our budget while creating API requests to test our code
Preventing characters from spoiling later parts of the book or breaking the fourth wall when asking them questions
Training the AI to tailor responses to emotions based off characters in the book depending on who they’re interacting with
Matching the different TTS voices OpenAI offers depending on the character they’re narrating as
Staying hydrated and healthy while racing to meet the 36-hour deadline!

Accomplishments that we're proud of

This was Anthony Botticchio, Larry Han, and Claire Hu’s first ever hackathon and Claire’s first-time programming!
Elements in our Logo and UI/UX were 100% drawn by hand using vector art on Figma! We wanted to follow an RPG game style theme, so custom making everything was the only way to go.
One of our team members has personally faced learning disabilities, so creating a solution for firsthand problems was incredibly fulfilling. We believe this experience gave us a deeper understanding of the current pain points, allowing us to craft a tailored solution for the problem space we identified.

What we learned

Hackathons are not easy…one hour seems to go by in fifteen minutes!
OpenAI is extremely powerful, however tremendous training is needed to make it robust
Grouping designs in Figma is essential—otherwise your frames will get very messy very fast!

What's next for Journey

We hope to continue developing Journey and helping students learn and enjoy their reading experiences. As the problem space we chose to tackle is close to our hearts, we truly believe others struggling with similar experiences deserve the chance to explore wonderful stories in an easier fashion than we have.

Reading is instrumental in the development of children, and we hope that by continuing to develop Journey we can give a generation of children a love for reading!