Countability

Project Name: Countability Elevator Pitch [200 characters]: Surgical tools left in a patients’ body can cause illness or death. Countability is a low-cost, accessible technology using AI and smartphones to track tools in the operating room to catch mistakes.

About the Project:

Inspiration

Last week, a woman from New Zealand discovered a dinner plate-sized surgical tool left behind in her abdomen after a C-section. Unfortunately, incidents like this are not uncommon; surgical tool retention is estimated to occur in 1 out of every 1000 surgeries. This rate is much higher in areas without consistent medical access, or in makeshift hospitals.

Existing mechanisms to counter medical device retention aren’t comprehensive. Surgeons currently use a ‘count board’, or a whiteboard within the operating room to track the number of tools used. However, as in the previously mentioned situation, this mechanism fails if a surgeon or nurse forgets to count a device. Some hospitals in the US are piloting RFID tracking mechanisms, however, these are too expensive (require high-resolution cameras, consume too much electricity, etc), especially for hospitals from rural areas and smaller community hospitals.

What Countability does

Countability uses computer vision and object tracking to keep a real-time inventory of surgical implements used in a procedure, and alerts surgeons when objects are unaccounted for. To do this, camera feed of sterilized and unsterilized (used) tools from smartphones is sent to the Countability webapp. There, the contents of the trays are analyzed using Yolo V8, a state-of-the-art multi-object detection model capable of differentiating various medical devices in the same tray.

During the surgery, surgeons can view which equipment is in use at a glance on a monitor in the OR. Members of the surgical team can also update the program of the surgery’s progression (started, in progress, ended), which will alert the team of relevant errors. For instance, if the surgery is in progress and is one sponge unaccounted for, the program will display a level one error. If the surgery was to end and the sponge is still unaccounted for, the error will escalate in severity.

After the surgery, the team can analyze statistics such as the number of each level of errors thrown during the surgery, timestamps for important events, and any errors that were overridden by the team through their personal login in the Countability webapp. This automation enhances surgical efficiency by eliminating the need for time-consuming manual counts, reduces the risk of human error, and ultimately improves patient outcomes.

How we built Countability

Data Collection and Annotation: We located specific datasets of annotated surgical instrument images. This data was the basis for training our AI model to recognize and track instruments accurately. Multiple different data analysis pipelines were tested including that of Google’s ML Vertex, Trainable Machine, Box Segmentation based on pixel distribution, and various other attempted methods. The process involved hours of data preparation and validation. We experimented with various layerings of machine learning algorithms eventually settling on Yolo v8 as it performed the best for our task.

User Interface & Backend:

The user interface is built with Next.js and is structured as three SPA applications running the camera software, alerts panel, and surgical technician panel. These communicate with a Python with FastAPI Server through WebSocket to transform a camera feed into meaningful alerts for surgeons. Alerts and surgical records are kept in an SQLite database to persist records and sync them across sessions.

Challenges we ran into

Here are some of the challenges we encountered and how we overcame them:

One of the initial hurdles was the complexity of collecting and annotating surgical instrument data. Surgical environments vary widely, and obtaining diverse and representative datasets was not easy as this is not a commonly trained dataset. We overcame this challenge by implementing a model that was trained on images of surgical tools in metallic trays and in different backgrounds to augment the model and diversify its output potential.

After that, the next big issue was model training. Developing a custom multi-object detection model that could consistently recognize surgical instruments in real-time was a formidable task. We surmounted this challenge through iterative training, fine-tuning, and many hours of work. Communication between clients was another major hurdle. CountAbility runs as a distributed network of devices, with the alert monitor, technician panel, and camera software communicating through WebSocket and HTTP. Syncing alerts across these devices proved difficult as it required resolving issues with one-to-many and many-to-one communication.

The final major challenge was real-time processing. Implementing this feature required optimization and resource management to ensure seamless performance.

Accomplishments that we're proud of

CountAbility's fully custom-trained multi-object detection model consistently achieves high levels of precision in identifying and locating surgical instruments in real-time. After initial setup, the tool tracking system can run fully autonomously without any input from the surgical team. In a high-stress operating room environment, small tasks like tool tracking can be extremely burdensome and fall through the cracks. Countability's capability to provide real-time information is impactful as it adds information to this high-stress environment that wouldn't be there otherwise.

CountAbility's user interface was designed with the high-pressure, fast-paced atmosphere of an operating room in mind. To minimize operational error, status indicators and identifiers use large color blocks that can be viewed from far away, and the system runs autonomously once set up for a procedure. No more taking off gloves or putting down tools to keep track of items.

Accessibility and Affordability: CountAbility's mission is to provide a cost-effective solution for all healthcare facilities, including smaller facilities such as military hospitals. We've worked diligently to ensure that our platform remains low cost and low resource requirement, making it accessible to a wide range of medical settings.

What we learned

Through the many challenges we overcame, we learned some valuable lessons and picked up some important skills. One big area of learning was the pipeline for training a custom object detection model. This process involved much more data preprocessing than we had originally anticipated, and the rest of the process was not smooth either. However, in succeeding to create the model, we can now implement any custom image segmentation or object detection model given a dataset. We also learned that the user-centered design is essential. The UI was built primarily through the perspective of the users such as the surgeons and nurses, which helped keep the user experience flawless and enjoyable while being simple and aesthetically pleasing. Our journey also reinforced the significance of high-quality, diverse datasets. Robust data collection and annotation are the bedrock of developing accurate AI models.

What's next for Countability

In the future, we hope to implement: Customization for Specialties: We recognize that different surgical specialties have unique requirements, such as unique surgical equipment that can be difficult to identify/seperate.

User research: We plan to reach out to healthcare providers, particularly in underserved regions/temporary hospitals, to see how best Countability could serve them in its current state and improvements we can make.

Regulatory Compliance: Since Countability handles healthcare information, we will stay vigilant in maintaining HIPAA compliance and we will work to increase safety. This ensures that CountAbility meets the highest standards of patient safety and data security.

Educational Initiatives: We plan to offer training programs and resources to ensure that healthcare professionals can harness the full potential of CountAbility in their surgical practice.