Hey Doc, Need a Hand?

Setup

Inspiration

In light of recent/ongoing COVID epidemic, it has become clear that our nursing staff is not only invaluable in the duties they perform, but also heavily understaffed. To help reduce their workload, we wanted to create a robot that could help perform the duties of a surgical assistant.

What it does

Our robot is able to react to voice commands and bring the proper tools to the doctor.

How we built it

We used a robotic arm kit and connected it to an Arduino that could use serial communication to receive signals from a laptop running our speech detection programs. So whenever someone said the name of the tool they want, the machine learning algorithms running on the laptop would process the audio, and send the signal to the robot arm to pick up the correct tool using the gripper of the robot arm.

Challenges we ran into

We ran into issues with interfacing our machine learning algorithms with the Arduino. Since the Arduino didn't have the computing power that we needed, we had to create an indirect way to use our speech processing by having the laptop communicate with the Arduino.
We also had problems powering our electromagnet. The Arduino can only output a maximum of 5V, which is not enough to make a powerful electromagnet. So we thought of using a relay circuit that can be controlled by the Arduino, and can allow a larger voltage and current to flow through the electromagnet. Unfortunately we could not get a very powerful electromagnet thus we decided to go with the gripper instead.

We also wanted to use a camera to track the doctor's hands, so the robot would be guided to their hands to know where to drop the tools. We managed to develop a way to track the hand moment and determine the direction in which the arm should move but we had difficulties mounting a camera(difficulties with ESP32 CAM implementation) to the arm.

Accomplishments that we're proud of

We're proud of our robot arm's ability to move as a response to voice commands, and being able to accurately and consistently move how we want it to. We are proud of our build in the short amount of time we had and believe that this technology has several applications and is scalable.

What we learned

We learned a lot about serial communication with the Arduino, using computer vision to track objects, and using speech processing to determine what commands are being spoken. We also learned a lot about the limitations of the Arduino and methods to bypass those limitations in other ways, like using a relay or using serial communication between our Arduino and a laptop.

What's next for Robotic Surgery Assistant

-We'd like to make the electromagnet on our robot stronger, so it could pick up heavier tools and faster. -For future versions of the robot, we'd want to be able to implement the hand tracking technology so that our robot could use computer vision to track the doctor's hand and know where to deliver the tools to. -We could also work on making our robot arm bigger and stronger so that it can pick up more and heavier tools even faster than it already could.

Another aspect that can be improved upon is using object recognition to identify the various surgical tools to that labelling of the tools is not required.