UtterScripts

Use your iphone to go hands free to go hands free in the apple ecosystem to complete complex tasks, such as coding, messaging or even scraping data using llama3 + groq.

Comment

Inspiration

Ever had a great idea but you're middle of doing something else; or have you wished that you also need to be doing something else in parallel- going async would be awesome. Well, wait no longer, cause we hacked together a prototype that can setup your dev environment, share a link over slack, or create a web scraped list for you just through the power of voice.

What it does

With accessibility options enabled, on your iPhone just double tap at the back to activate a voice-based Shortcuts that takes in instructions and executes a series of sophisticated tasks in our Mac without ever hitting a single finger stroke. We have recored examples of scaffolding a NodeJS application and opening on VS code or sending a well wishing text on Mothers' Day.

How we built it

We used Shortcuts on iPhone to activate a voice input that tunneled into a fastAPI server on the Mac, which then executes the series of tasks using OpenInterpreter backed by Meta's LLama3-70b Instruct model (hosted on GroqCloud) to interpret the utterances into an elaborate series of AppleScript commands.

Challenges we ran into

Having LLama3 70b-Instruct was fast and efficient for most tasks, however, in tasks that require a longer CoT, need longer context length models. However, with the pending release of Llama3 with 400b+ parameters and potentially longer context lengths, we might be able to get away with not using closed-source models.

Accomplishments that we're proud of

Chained together the benefits of Apple ecosystem with OpenInterpreter powered by SOTA open-source model to create a seamless day-to-day user friendly tool.

What we learned

With the right prompting, we can uncover hidden gems in capabilities of Llama3. The safety guardrails built-in the model give us confidence in ensuring despite unforced or forced errors, we could accomplish those tasks with good intentions.

What's next for UtterScripts

1) Test with larger open-source Llama models pending release. 2) Work with newer iPhone and Apple Watch models with 'action buttons' to trigger tasks via voice.

Built With

Share this project:

Updates