SilentCare

sign in page

Inspiration

After certain surgeries or medical procedures, patients can temporarily lose the ability to speak. This can happen because of breathing tubes, throat surgeries, sedation, or stroke recovery. Even when patients are fully conscious, communicating basic needs becomes difficult. They may rely on pressing a call button, pointing, or writing on paper, which can be slow and frustrating.

We wanted to explore whether a simple computer vision system could help patients communicate basic needs using hand gestures. Instead of trying to build a full sign language translator, we focused on a small, practical scenario: helping temporarily non-verbal patients express urgent needs like water, help, pain, or bathroom assistance.

SilentCare was created as a prototype to show how gesture recognition and AI language generation could work together to make this kind of communication faster and easier.

What it does

SilentCare is an AI-powered gesture communication system for hospitals. Using only a standard bedside camera, SilentCare detects patient hand gestures and translates them into meaningful requests for nurses in real time. Patients perform simple gestures which are detected and translated into natural sentences such as:

🤟: “I am in pain and need a nurse.”

✋: “I need water.”

✊: “I need assistance.”

The system then:

Speaks the request aloud in the patient’s room
Sends the message instantly to a nurse dashboard
Automatically assigns an urgency level (High, Medium, Low)

This allows nurses to triage patient needs faster and ensures patients who cannot speak are still heard.

How we built it

SilentCare combines real-time computer vision with AI-powered language generation.

We used OpenCV to capture video from a webcam and process frames in real time. MediaPipe was used to detect hand landmarks and track finger positions. Based on those landmarks, we implemented a lightweight rule-based gesture classifier that determines which gesture the user is making.

Recognized gestures are stored in a short buffer and sent to the Gemini API, which converts the detected gesture sequence into a natural sentence that a patient might say to a nurse. This allows the system to transform simple gesture labels into clearer, more human-friendly messages.

Finally, the generated sentence is displayed in the interface and can optionally be spoken using text-to-speech.

Challenges we ran into

One of the biggest challenges was keeping the project scope realistic. A full real-time sign language translator was not feasible within a short hackathon timeframe, so we narrowed the problem to a specific use case: communication for temporarily non-verbal hospital patients.

Another challenge was gesture reliability. Real-time hand tracking can be sensitive to lighting conditions, camera angles, and hand positioning. To improve reliability, we limited the vocabulary to a small set of visually distinct gestures that could be detected more consistently.

We also had to carefully integrate Gemini in a meaningful way. Instead of using it for raw gesture detection, we used it to convert detected gesture sequences into natural language messages.

Accomplishments that we're proud of

We’re proud that we were able to turn a complex accessibility problem into a focused and working prototype. By narrowing the scope to a real hospital scenario, we created something that is easy to demonstrate and immediately understandable.

We’re also proud of the way we combined different technologies. MediaPipe handled the real-time gesture detection, while Gemini improved the output by transforming simple gesture labels into natural sentences.

Most importantly, we created a demo that highlights how computer vision and generative AI can work together to support communication in healthcare settings.

What we learned

This project taught us how powerful the combination of computer vision and generative AI can be when solving real-world problems.

We learned how to integrate AI models into real-time applications and how important it is to design technology that prioritizes simplicity, reliability, and accessibility, especially in healthcare environments.

We also learned how impactful even simple AI systems can be when they focus on solving real human challenges.