Riffr

• 

  Demo

• 

• 

• 

💡Inspiration

we were really captivated by how this dedicated Youtuber Hugo Cornellier who took photos in front of a camera every day for 9 years, and ended up making a time-lapse collection of over 2500 photos, placed perfectly around himself, as you can see below 🔽

It turns out, this was a painstakingly slow and difficult process. Aligning along a specific face, resizing images, and compiling these to make a meaningful video. It took the creator 50+ hours just to make this!!

Being inspired by his pain, we decided to lend a hand and put out a tool that can automate all this configuration, detect and cycle between faces, and generate a video with it all!

Psst...With this app, a user doesn't even have to click photos repeatedly at the same place or in the same position - the ML model adjusts all sorts of photos together 🤭

⚙️What it does

It helps you make awesome, awesome montages! Okay, that's an oversimplification. Let me clarify how. You're allowed to choose up to 120 images(nice, right?), run the detection on them, and then edit photos with faces detected and size them up accordingly. Also, you choose a frame rate, that is, how fast the images go by, and you can play and pause the editor to check out a preview with those settings. When you're done, the tool compiles these cached images and their configs and wraps it all up together in a neat little video. Here are some other use cases for this app:

• A collection of burst-mode photos compiled together to give a motion effect.
• Creating a vacation or travel montage with different locations but the same poses.
• A fun little eye-catching flip through of some transitional photos, like childhood to adult or emotion to emotion(happy->sad->angry->surprised).
• Last but not least, for an art-themed purpose, show off your skills!😉

🛠️ How we built it

We built this hack using the Quasar framework, which is basically built on top of Vue.js. We first implemented a file picker, restricted it to load only images in the browser, then use a machine learning model to discard to photos with no faces detected. We did have to change the image caching technique, which crashed our development tab quite a lot of times, but was finally solved with correctly using Web Workers. Then, we used the Media Recorder Web API to encode these images captured from the readable streams from the canvas to compile an output video.

⚠️ Challenges we ran into

• We had not used the ml5.js library before, let alone their face-detection pre-trained model.
• We faced memory leaks and crashes 😖 when loading and manipulating a large number of images in the browser's context.
• We did not know how to create image caches and efficiently flip through images at high frame rates while simultaneously centering them.
• We struggled with capturing video streams from the HTML Canvas, and had never before used the native MediaRecorder to record and compile a video with Javascript.
• Our domain provider did not work and ate up a lot of time attempting to correct the errors - time that could have been used to add more features! 😢
• One of the most frustrating things to accomplish was to preserve image quality when creating a video from a readable stream with Javascript.

👩‍💼🧑‍💻 Accomplishments that we're proud of

• Developing end-to-end a simple web app that helps make artistic photo montages sooo much easier!
• Making the app work entirely in the frontend, from face detection with ML and compiling a video, all with just Javascript, so the privacy of the user is preserved!
• Learning so much and diving deep into Web APIs and using them like pros for an art-themed hack😎 (Although it did take some keyboard-banging to get there)
• Creating a web-optimized .webm format of video, along with screen dimensions' sized output in the browser.

⚓ What we learned

• How to use web workers, most importantly to load and cache images asynchronously on an entirely separate execution thread.
• Using the canvas for image manipulation, and using the ImageBitmap to switch images with lightning fast speed!
• Understanding how Machine Learning works in the browser, and how to use a face detection model to align images with bounding boxes returned.
• How to use HTML Canvas with Vue.js and capturing streaming chunks off of it to create a video blob!
• Correct usage of web-video encoders and how to set bitrates(we used the VP8 codec instead of VP9 because of this issue) to ensure high-quality recording of a video.

😏 What's next for Riffr

We have some feature ideas that we'd like to be added to the project for future versions:

• Color Picker that helps you choose a background color for the whitespace in the montage.
• Text and tint effects on the canvas that will make the montage more interesting.
• A shareable link to the compiled video, along with proper backend connectivity.
• Reordering images in the browser before editing them.
• An onboarding tour of the functionalities so you don't have to simply read usage instructions.
• Changing recording dimensions and choosing different export extensions like a .gif.

Built With

• appwrite
• canvas
• javascript
• quasar
• vue