EMG from Combination Gestures with Ground-truth Joystick Labels

<p>Dataset of surface EMG recordings from 11 subjects performing single and combination gestures, from "**A Multi-label Classification Approach to Increase Expressivity of EMG-based Gesture Recognition**" by Niklas Smedemark-Margulies, Yunus Bicer, Elifnur Sunger, Stephanie Naufel, Tales Imbiriba, Eugene Tunik, Deniz Erdogmus, and Mathew Yarossi.</p> <p>For more details and example usage, see the following:</p> <ul> <li>Paper pdf -&nbsp;<a href="https://arxiv.org/pdf/2309.12217.pdf">https://arxiv.org/pdf/2309.12217.pdf</a></li> <li>Experiment code - <a href="https://github.com/neu-spiral/multi-label-emg">https://github.com/neu-spiral/multi-label-emg</a></li> </ul> <h1>Contents</h1> <p>Dataset of single and combination gestures from 11 subjects.&nbsp;<br>Subjects participated in 13 experimental blocks.<br>During each block, they followed visual prompts to perform gestures while also manipulating a joystick.<br>Surface EMG was recorded from 8 electrodes on the forearm; labels were recorded according to the current visual prompt and the current state of the joystick.</p> <p>Experiments included the following blocks:</p> <ul> <li>1 Calibration block</li> <li>6 Simultaneous-Pulse Combination blocks (3 without feedback, 3 with feedback)</li> <li>6 Hold-Pulse Combination blocks (3 without feedback, 3 with feedback)</li> </ul> <p>The contents of each block type were as follows:</p> <ul> <li>In the Calibration block, subjects performed 8 repetitions of each of the 4 direction gestures, 2 modifier gestures, and a resting pose.<br>Each Calibration trial provided 160 overlapping examples, for a total of: 8 repetitions x 7 gestures x 160 examples = 8960 examples.</li> <li>In Simultaneous-Pulse Combination blocks, subjects performed 8 trials of combination gestures, where both components were performed simultaneously.<br>Each Simultaneous-Pulse trial provided 240 overlapping examples, for a total of: 8 trials x 240 examples = 1920 examples.</li> <li>In Hold-Pulse Combination blocks, subjects performed 28 trials of combination gestures, where 1 gesture component was held while the other was pulsed.<br>Each Hold-Pulse trial provided 240 overlapping examples, for a total of: 28 trials x 240 examples = 6720 examples.</li> </ul> <p>A single data example (from any block) corresponds a window 250ms of EMG recorded at 1926Hz (built-in 20&ndash;450 Hz bandpass filtering applied).<br>A 50ms step size was used between each window; note that neighboring data examples are therefore overlapping.</p> <p>Feedback was provided as follows:</p> <ul> <li>In blocks with feedback, a model pre-trained on the Calibration data was used to give realtime visual feedback during the trial.</li> <li>In blocks without feedback, no model was used, and the visual prompt was the only source of information about the current gesture.</li> </ul> <p>For more details, see the paper.</p> <h1>Labels</h1> <p>Two types of labels are provided:&nbsp;</p> <ul> <li>joystick labels were recorded based on the position of the joystick, and are treated as ground-truth.</li> <li>visual labels were also recorded based on what prompt was currently being shown to the subject.</li> </ul> <p>For both joystick and visual labels, the following structure applies. Each gesture trial has a two-part label.</p> <p>The first label component describes the direction gesture, and takes values in {0, 1, 2, 3, 4}, with the following meaning:</p> <ul> <li>0 - "Up" (joystick pull)</li> <li>1 - "Down" (joystick push)</li> <li>2 - "Left" (joystick left)</li> <li>3 - "Right" (joystick right)</li> <li>4 - "NoDirection" (absence of a direction gesture; none of the above)</li> </ul> <p>The second label component describes the modifier gesture, and takes values in {0, 1, 2}, with the following meaning:</p> <ul> <li>0 - "Pinch" (joystick trigger button)</li> <li>1 - "Thumb" (joystick thumb button)</li> <li>2 - "NoModifier" (absence of a modifier gesture; none of the above)</li> </ul> <h2>Examples of Label Structure</h2> <p>Single gestures have labels like (0, 2) indicating ("Up", "NoModifier") or (4, 1) indicating ("NoDirection", "Thumb").</p> <p>Combination gesture have labels like (0, 0) indicating ("Up", "Pinch") or (2, 1) indicating ("Left", "Thumb").</p> <h1>File layout</h1> <p>Data are provided in Numpy and MATLAB format. Descriptions below apply for both.</p> <p>Each experimental block is provided in a separate folder.<br>Within one experimental block, the following files are provided:</p> <ul> <li>`data.npy` - Raw EMG data, with shape (items, channels, timesteps).</li> <li>`joystick_direction_labels.npy` - one-hot joystick direction labels, with shape (items, 5).</li> <li>`joystick_modifier_labels.npy` - one-hot joystick modifier labels, with shape (items, 3).</li> <li>`visual_direction_labels.npy` - one-hot visual direction labels, with shape (items, 5).</li> <li>`visual_modifier_labels.npy` - one-hot visual modifier labels, with shape (items, 3).</li> </ul> <h1>Loading data</h1> <p>For example code snippets for loading data, see the associated code repository.</p>