Manuel Meier, Paul Streli, Andreas Fender and Christian Holz
Sensing, Interaction & Perception Lab
Department of Computer Science, ETH Zürich

This is the research repository for the IEEE VR 2021 Paper: "TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing"

TapID enables direct touch input for Virtual Reality on physical surfaces to support tasks like typing, browsing, and object manipulation. Our wrist-worn devices detect taps, identify fingers, and relay events to the VR environment.

The dataset consists of IMU data from 18 participants collected using the TapID wristbands. Each data point represents a tap gesture performed with one of five fingers (thumb, index, middle, ring, or little finger) of each hand.

The data can be downloaded from here.

1. Clone this repository:

   git clone https://github.com/yourusername/TapID.git
   cd TapID

2. Install the required dependencies:

   pip install -r requirements.txt

3. Download the dataset from the link above and extract it to the data directory.

You can run the experiments using the command-line interface:

# Cross-participant experiment (default mode)
python run_experiment.py --mode crossparticipant --sensors 0 1 2 3

# Cross-block experiment
python run_experiment.py --mode crossblock --sensors 0 1 2 3

# Specify output directory and experiment name
python run_experiment.py --mode crossparticipant --sensors 0 1 2 3 --output_dir results --experiment_name my_experiment

Manuel Meier, Paul Streli, Andreas Fender, and Christian Holz. TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing. In Proceedings of Virtual Reality Conference 2021 (IEEE VR). DOI:https://doi.org/10.1109/VR50410.2021.00076

• TapID paper PDF
• TapID video
• TapID project page

@inproceedings{vr2021-TapID,
  author = {Meier, Manuel and Streli, Paul and Fender, Andreas and Holz, Christian},
  booktitle = {2021 IEEE Virtual Reality and 3D User Interfaces (VR)},
  title = {TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing},
  year = {2021},
  volume = {},
  number = {},
  pages = {519-528},
  doi = {https://doi.org/10.1109/VR50410.2021.00076}
}

Current Virtual Reality systems typically use cameras to capture user input from controllers or free-hand mid-air interaction. In this paper, we argue that this is a key impediment to productivity scenarios in VR, which require continued interaction over prolonged periods of time-a requirement that controller or free-hand input in mid-air does not satisfy. To address this challenge, we bring rapid touch interaction on surfaces to Virtual Reality-the input modality that users have grown used to on phones and tablets for continued use. We present TapID, a wrist-based inertial sensing system that complements headset-tracked hand poses to trigger input in VR. TapID embeds a pair of inertial sensors in a flexible strap, one at either side of the wrist; from the combination of registered signals, TapID reliably detects surface touch events and, more importantly, identifies the finger used for touch. We evaluated TapID in a series of user studies on event-detection accuracy (F1 = 0.997) and hand-agnostic finger-identification accuracy (within-user: F1 = 0.93; across users: F1 = 0.91 after 10 refinement taps and F1 = 0.87 without refinement) in a seated table scenario. We conclude with a series of applications that complement hand tracking with touch input and that are uniquely enabled by TapID, including UI control, rapid keyboard typing and piano playing, as well as surface gestures.

The dataset and code in this repository is for research purposes only. If you plan to use this for commercial purposes, please contact us. If you are interested in a collaboration with us around this topic, please also contact us.

THE PROGRAM IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY
WARRANTY. IT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE
QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE
DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR
CORRECTION.

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW THE AUTHOR WILL BE LIABLE TO YOU
FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL
DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT
NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES
SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH
ANY OTHER PROGRAMS), EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.