Code Whisperer

Can't type? No problem speak your code.

Comment

Speech Extension - VS Code Extension

Inspiration

As developers, we spend countless hours typing, debugging, and refactoring code. However, traditional coding methods can be limiting, especially for those with accessibility needs or those who want to enhance their workflow efficiency. Speech Extension was born from the idea of making coding more accessible, efficient, and hands-free. By integrating voice recognition into VS Code, developers can now speak their code, modify it, and even have it read back to them.

What it does

Speech Extension is a powerful VS Code extension that enables developers to code using voice commands. This tool enhances accessibility and productivity by integrating seamless speech-to-text and voice-driven code modifications.

Key Features:

  • Speak Code – Convert spoken words into code using speech-to-text technology.
  • Modify Code via Voice – Edit, delete, and manipulate code using natural language commands.
  • Text-to-Speech – Listen to your code being read aloud for better accessibility and review.
  • Hands-Free Coding – Reduce reliance on keyboard input and improve workflow efficiency.
  • Multi-Language Support (Coming Soon) – Expanding support to various programming languages.

How we built it

We leveraged the following technologies to bring Speech Extension to life:

  • VS Code API – To build the core extension functionalities.
  • OpenAI API – For speech recognition, transcription, and processing commands.
  • Faster Whisperer - Convert audio to text
  • Fast API - Send requests from extension to back-end server
  • Python – Backend processing and handling commands.
  • Node.js – Manages extension-related services.
  • Firebase – To track users and manage web services.
  • Tailwind CSS – For an elegant and user-friendly front-end experience.

Challenges we ran into

One of the biggest challenges we faced was optimizing the parsing speed to ensure real-time speech-to-code conversion. Efficiently handling voice input and transforming it into accurate, structured code required refining our parsing logic and improving response times.

Accomplishments that we're proud of

We are proud that our project is fully functional, allowing developers to speak, modify, and interact with their code using only their voice. Achieving seamless speech integration within VS Code was a significant milestone.

What we learned

Throughout the development process, we gained valuable insights into working with speech recognition APIs, optimizing real-time processing, and enhancing user experience within the VS Code environment. Additionally, we learned how to balance accuracy and speed when converting spoken commands into structured code.

What's next for Speech Extension

Our next steps include:

  • Publishing the extension to the VS Code Marketplace for wider accessibility.
  • Expanding language support beyond JavaScript, Python, and C++.
  • Adding customizable voice commands for a more personalized experience.
  • Enhancing AI-powered code assistance to make voice-based coding smarter and more intuitive.

Installation & Usage

Install from VS Code Marketplace

(Currently in progress – link will be added soon.)

Install Manually

  1. Clone the repository: bash git clone https://github.com/tasnimhny/speech-app.git cd speech-app
  2. Install dependencies: bash npm install
  3. Build the extension: bash npm run build
  4. Install backend dependencies: bash pip install -r requirements.txt
  5. Launch in VS Code:
    • Open the project in VS Code.
    • Press F5 to run the extension in a new VS Code window.

How to Use

  1. Activate Speech Mode:
    • Open the Command Palette (Ctrl + Shift + P) and select Start Speech Mode.
    • Alternatively, use a custom keyboard shortcut or a voice command (if configured).
  2. Speak Your Code:
    • Example: "Define a function called add that takes two arguments" → js function add(a, b) { return a + b; }
  3. Modify Code via Voice:
    • Example commands:
      • "Delete line 10" → Removes line 10.
      • "Change let to const on line 5" → Updates let to const on line 5.
      • "Read this function aloud" → Uses text-to-speech to read the function.
  4. Stop Speech Recognition:
    • Say "Stop listening" or use the Stop Speech Mode command.

Contributing

We welcome contributions from the developer community.

How to Contribute:

  1. Fork the repo.
  2. Create a feature branch: bash git checkout -b feature-branch
  3. Commit your changes: bash git commit -m "Add new feature"
  4. Push to the branch: bash git push origin feature-branch
  5. Open a pull request – we will review and merge it.

License

This project is open-source and available under the MIT License.

Try Speech Extension Today & Experience Hands-Free Coding

Stay tuned for updates and let us know your thoughts.

GitHub Repo: https://github.com/tasnimhny/speech-app

Built With

Share this project:

Updates