AI-Terminal

Speech to text in training
Speech to text predicting a word said in an audio file

Inspiration

We wanted to create a seamless interface that brings the power of AI directly to a user’s desktop, eliminating the friction between natural language queries and actual system operations.

What it does

AI-Terminal is a desktop application that listens to your plain-language commands—like “Create a file named demo.py here”—and executes them on the spot. It also answers questions, provides code suggestions, and streamlines routine tasks by leveraging an AI model trained on specialized data (codenamed “lily-cybersecurity”).

How we built it

We built the desktop interface using Electron with a React-based UI, enabling a familiar web-development workflow packaged in a native desktop environment. On the backend, we used Python to integrate and manage our custom-trained AI model. The model was fine-tuned on “lily-cybersecurity,” a dataset that helps ensure both contextual understanding and accurate command execution.

Challenges we ran into

Bridging Electron and Python: Integrating a responsive React frontend with a Python backend required careful API design and handling of asynchronous calls.
Custom Model Training: Achieving high accuracy with our “cyer lilly” fine-tuned model involved significant hyperparameter tuning and data preprocessing.
Security Concerns: Giving AI direct access to system commands demands robust permissions handling and user control.

Accomplishments that we're proud of

Successfully packaging a user-friendly desktop app that feels modern and intuitive.
Training and integrating an AI model that can handle both natural language Q&A and direct system operations.
Maintaining a responsive pipeline between Electron, React, and Python, ensuring minimal lag for real-time feedback.

What we learned

Cross-technology Integration: The project taught us how to synchronize web technologies (React) with desktop capabilities (Electron) and advanced backend logic (Python AI).
Model Fine-tuning: Hands-on experience with data curation and iterative training for a specialized AI model.
User Experience Matters: Even powerful capabilities need to be presented simply to drive adoption and trust.

What's next for AI-Terminal

Expanded Capabilities: We plan to add more complex operations like orchestrating multi-step deployments and integrations with other productivity tools.
Improved Security: Implement advanced permission layers and user control so they can define which operations the AI can and cannot perform.
Plugin Ecosystem: Allow third-party developers to create extensions that add domain-specific knowledge and functionality to the AI-Terminal.

Bonus

We built a speech-to-text AI from "scratch" which is somewhat functional for a certain set of single words, but we kept it on the side and did not end up using it.