PenGram AR

Inspiration

Training is large expense in enterprise. Particularly, in situation where inexperience operator needs help from an expert who is not next to him, the best way for instruction is through skype or phone call. However, it still can be ineffective, and it would be better if the experienced expert is right next to the operator to teach him side by side.

What it does

In our current hack, we can draw annotation in different colors in 3D. This annotation should be transmitted in real time to operator's view if we have networking work properly, and the annotations should appear next to operator's real object as it appear to annotations to the hologram that expert sees.

How I built it

Unity, Kinect Fusion, Python, Flask

Challenges I ran into

Networking is very hard to set up on hololens

Accomplishments that I'm proud of

We came up with a novel way to do hand-tracking with the Hololens, greatly improving the interactions possible with the device.

What I learned

We also ran VR demos for almost 8 hours, and led a workshop. It's hard to find time to get work done on the hack while juggling these other (really fun) events!