Kramhtron.ai

Goal

With Kramhtron, we aim to streamline data center operations for technicians working on the floor, who are actively diagnosing and addressing issues with hardware and connectivity in the data center environment. Large volumes of work mean that technicians need to work faster than ever, even with an ever-growing backlog of tasks to do. Ticket systems and communication become tedious, and inventory management and human error become additional friction that causes significant inefficiencies in the workflow of data center technicians. We support an agentic workflow to for these pain points.

We can make the lives of technicians and engineers easier - allowing them to spend more time on their most important work.

What it does

To achieve our goal, we take a number of approaches to increasing operational efficiency for data center technicians:

• All in one system for inventory management and stock replenishment
• Voice and video comms from engineering to DC techs
• Give technicians and engineers a voice agent with RAG access to service manuals for Dell servers
• Ticket creation and inventory management with voice agent

How we built it

The core of the application is built around Convex, which is a platform as a service built around a sync engine that provides serverless functions and a reactive database with to shuttle data across all connected clients in real time. We integrated Convex with Auth0 for authentication, ensuring that only the minimum number of required people have access to data in the system. The front-end of the application uses React with Tailwind along with the Convex SDK to interact with and view data in real-time.

Vision functionality uses NVIDIA Nemotron models which we are accessing through the OpenRouter API. The video feed comes from a physical.inc camera dev kit, which sends the video over Wi-Fi to a Python server that utilizes OpenCV to do some processing, and finally streams video frames into Convex for storage and real time broadcasting, from which frames are pulled to display in the front-end application.

The voice agent uses ElevenLabs integrated with the NVIDIA Nemotron models as well. Within ElevenLabs, we configured a RAG pipeline so that we can answer questions from DC techs and engineers about the hardware they're working with, all via simple conversational interactions - without having to dig through documentation on the floor.

Challenges we ran into

• We had to make some tweaks to the physical.inc SDK locally to get the cameras online

Accomplishments that we're proud of

• Video streaming and storage through Convex
• Implementing ReAct agentic workflow for adding details to tickets and triaging tickets created via voice agent
• Implementing mock video streaming workflow using Python and FFmpeg by streaming frames from video to Convex server

What's next for Krahmtron AI

• Detect asset tags in frame of technician's view to automatically surface relevant data such as past tickets and diagnostic history
• Use custom-trained models to help with triage and prioritization of tickets
• Add more documentation for more manufacturers, server racks, and other hardware to the RAG knowledge base for the voice agent

Built With

• convex
• nvidia-nematron
• opencv
• openrouter
• python
• react
• tailwind
• typescript