robotics arm
pick up an object
Wall-e and Eveeee <#

WALL-E: Hands-Free Robotic Arm Control for Paralyzed Patients

Inspiration

Today, 5.4 million Americans live with some form of paralysis or ALS. For many of them, the simplest tasks, picking up a glass of water, moving a phone closer, grabbing a book off a table, require another person's help, every single time.

We kept coming back to one question: what if you could move objects just by looking at them? How can we help people with muscle dysfunction and communication difficulties perform daily tasks?

Using an EEG device, a laptop, a camera, and a 3D-printing robotic arm, we help patients control the robotic arm to perform daily tasks by applying computer vision and signal processing to interpret brain activity.

What It Does

WALL-E is an assistive robotic arm system designed to restore object manipulation autonomy for paralyzed patients through three intuitive, non-invasive input modalities:

Input	Modality	Action
👁️ Gaze	Eye tracking	Select an object
🦷 Jaw clench	Muse 2 EEG	Trigger a grasp
🗣️ Head movement	Webcam head tracking	Reposition the arm

A central state machine fuses these signals to drive a 6-DOF xArm 1S through inverse kinematics — translating natural human intent into precise physical action, with no hands, voice, or caregiver required.

How We Built It

Wall-E is built on three tightly integrated systems working in concert.

For the robotic arm, we worked with a Hiwonder xArm 1S, debugging servo behavior, hand-measuring link lengths, and implementing inverse kinematics, forward kinematics, and trajectory planning from scratch to achieve reliable, repeatable end-effector positioning.

For brain-computer interfacing, we connected a Muse 2 EEG headband and decoded raw electrical brain signals in real time, using adaptive baseline calibration to classify jaw clench gestures as reliable, intentional control inputs, filtering out noise and false positives.

For computer vision, we built a gaze-based selection system using a single camera, turning the user's eye movement into a mouse cursor that highlights and selects physical objects in the real world, no hands required.

All three systems are fused by a central state machine that interprets user intent across modalities and translates it into precise physical action.

Hardware

Robotic arm: Hiwonder xArm 1S (6-DOF) - The robotic arm came as a kit with pre-installed servos. We assembled it from scratch and built our own inverse kinematics solver tailored to the arm's specific geometry.
Computing brain: MacBook running on GPU.
EEG device: Muse 2 headset
Vision: Inno Maker Camera

Software Pipeline

Object Selection The get_coordinate.py script uses an Inno Maker Camera to scan the scene and detect object coordinates. It integrates Depth Pro for depth estimation and YOLOv8 nano for real-time object detection. Gaze tracking is handled by Eye_track.py, which also maps the user's gaze to the arm's base rotation.

Grasp Trigger (Jaw Clench) muse_connection.py maintains the EEG stream from the Muse 2 headset. jaw_clench_detection.py processes the signal using an adaptive baseline to distinguish intentional clenches from noise.

Repositioning (Head Movement) head_tracker.py uses the built-in webcam to track head yaw, mapping it directly to servo 6 of the arm for intuitive repositioning. Patients can tilt their head left/right to reposition the arm.

Arm Control xarm_controller.py runs the central state machine, computing inverse kinematics, forward kinematics, trajectory planning, and other robotics functions to translate all input signals into coordinated arm movements.

Operational Workflow

Setup Phase → Calibration Phase (Eye + EEG) → Main Loop
                                                   ↓
                                     Gaze selects object
                                                   ↓
                                     Jaw clench triggers pick
                                                   ↓
                                     Head movement repositions
                                                   ↓
                                     Jaw clench triggers drop

Challenges We Faced

Solving Inverse Kinematics Arguably the hardest challenge we faced was solving the inverse kinematics of the robotic arm. Since the gripper operates independently, we treated the arm as a 5-DOF system. While forward kinematics was straightforward to derive using DH parameters, inverse kinematics proved significantly more difficult — we experimented with Newton-Raphson, BFGS, gradient descent, and Jacobian-based methods, but each struggled with singularities and failed to converge reliably across the full workspace. The best result ultimately came from Levenberg-Marquardt, which handled the nonlinear least squares formulation robustly and converged consistently across the arm's reachable configurations.

Coordinate Accuracy Raw depth estimates had consistent offsets. We derived empirical correction factors:

$$x' = \frac{x}{2.2} - 10, \quad y' = y + 5d, \quad z > 15 \text{ mm (threshold)}$$

where $d \in {-1, +1}$ represents the direction of movement.

Jaw Clench Sensitivity The adaptive baseline required careful tuning of the $k$-threshold and detection window (in ms) to reduce false positives without missing intentional clenches.

Input Coordination To prevent conflicting signals, EEG drop detection was gated: the system only arms for a drop trigger after the head has been centered for 500 ms.

Device Reliability Both the xArm and Muse 2 were prone to mid-session disconnects, requiring graceful reconnect logic.

Camera Initialization An OpenCV initialization bug required debugging during the setup phase before the vision pipeline could run reliably.

Accomplishments We're Proud Of

What we're most proud of is shipping a fully functional assistive device from scratch within a single hackathon. Integrating three independent control systems — EEG-based jaw clench detection, eye gaze selection, and head movement tracking — into a single coherent real-time pipeline was no small feat, and seeing them work in concert for the first time was a genuine milestone. We're equally proud of cracking the inverse kinematics problem: after working through Newton-Raphson, BFGS, gradient descent, and Jacobian-based approaches that all failed at singularities, landing on Levenberg-Marquardt and watching the arm reliably reach its target was one of those moments that made the late nights worth it.

What We Learned

At 4am, the arm decided it was done for the night — unfortunately, we weren't. We spent six hours debugging until 10am on submission day, which taught us more about robotics than any textbook ever could. Turns out hardware doesn't care about your deadlines. We also learned that the gap between "it works in theory" and "it works at 4am when everything is on fire" is much, much wider than expected. Next time, we'll sleep before the hackathon. Maybe.

What's Next for WALL-E

We love robotics and our vision is clear — evolve Wall-E from a hackathon prototype into a medical-grade assistive device. That means pursuing higher precision hardware, clinical-level safety standards, and rigorous real-world testing with actual patients. On the software side, we plan to expand the control modalities, improve IK robustness across edge cases, and build a caregiver monitoring interface. Longer term, we want to explore insurance compatibility and FDA regulatory pathways to make Wall-E a viable option for patients with ALS, spinal cord injuries, and locked-in syndrome — people who need it most.