Pulse Editor with AI Agent Assistant

Pulse Editor is a multimodal coding platform that lets users code interactively through typing, drawing, speaking, and more—enhanced by AI for a seamless, creative development experience.

Comment

Inspiration

The past two years have witnessed transformative advancements in AI, but the integration of these technologies into creative tools remains fragmented. We aim to bridge this gap by creating an all-in-one AI creation environment that leverages the potential of AI for coding, text, image, audio, and even video editing.

While some tools and platforms have attempted this, like OpenAI's 12 days release of ChatGPT suite, and many other specialized AI software, the open-source community is ripe for innovation to democratize these capabilities.

What it does

Pulse Editor is an innovative cross-platform editor enhanced with built-in AI assistants. It allows users to interact with code and creative projects through text, voice, and drawing gestures.

Key highlights:

  • Fully extensible and open-source.
  • Enables smart editing on all platforms.
  • Expands beyond coding to support diverse AI-assisted creative workflows, including text, image, audio, and video editing.

How we built it

To make the editing experience feel more interactive than a traditional editor like VSCode, I intend to make a new editor in the web and mobile environment that allows not only typing, but also drawing and speech.

The high level approach to solve this is as following (divided into 2 parts -- the MVP actionable items for this hackathon; and optional/future items not exactly the focus of this hackathon but can be added as bonus)

  1. MVP actionable items:
  2. make a single-file editor using Nextjs with responsive design for both desktop broswer and mobile broswer
  3. use HTML5 canvas API to allow drawing and cropping the region encircled by drawn lines
  4. extract information from the cropped region including: row_start, row_end, column_start, column_end, and text extracted using OCR
  5. feed the extracted information to "copilot" via in-context learning
  6. while drawing, allow speech-to-speech interaction with the "copilot" using OpenAI or ElevenLabs API

What's next for Chisel Editor with AI Agent Assistant

Looking ahead, Pulse Editor aims to evolve into a more versatile and robust platform. Our immediate focus includes:

  • Enabling Easy Extension of Views: Simplifying the process for developers to create and integrate custom views, empowering a thriving ecosystem of plugins and tools.
  • Enriching Native Support for Mobile and Desktop: Enhancing performance, UI/UX, and native features to deliver a seamless experience across all devices.
  • Allowing Cloud-Based AI Inference & Project Workspace: Providing secure and scalable cloud solutions for AI inference and project management, enabling creators to work anywhere, anytime.

Built With

  • llm-agent
  • nextjs
Share this project:

Updates