Time Series Data of Gaze, Head Pose, Hand Pose, and Object Positions for Object Approaches with a Given Intention

<p>This data set comprises time series data of gaze, head pose, hand pose, and object positions for object approaches with a given intention. The data was captured in the context of the following publication:</p> <ul> <li><em>Michael Fennel, Serge Garbay, Antonio Zea, Uwe D. Hanebeck</em>,&nbsp;<strong>Intention Estimation with Recurrent Neural Networks for Mixed Reality Environments</strong>,&nbsp;Proceedings of the 26th International Conference on Information Fusion (Fusion 2023) <em>(under review)</em></li> </ul> <p>A Microsoft Hololens 2 was used for recording the data at 60 fps under the modalities&nbsp;explained in detail in the above-mentioned paper.</p> <p>The file names are structured as follows:</p> <ul> <li><em>1st/2nd:</em> <ul> <li>The data with &quot;1st&quot; contains approaches to randomly placed objects&nbsp;on a grid, which are rendered in augmented reality. The user is informed about the object to approach using a visual cue. This corresponds to Section IV-A.</li> <li>The data with &quot;2nd&quot; contains approaches to real objects placed statically in a room. The user is informed about the object to approach using a voice command.</li> </ul> </li> <li><em>unfiltered:</em> Contains all approaches, including those where the user disrespects the given commands. Filtering is done as described in the paper.</li> <li><em>train/val/test:</em> The first dataset was split in a 70/20/10 ratio for training, validation, and test.</li> </ul> <p>Each data set contains the following columns. In each approach, 5 objects numbered from i=0 to i=4 are present.</p> <ul> <li>General: <ul> <li><em>time:</em>&nbsp;in seconds</li> <li><em>subject:</em> consecutive subject number</li> <li><em>handedness:</em> left (1), right (0)</li> <li><em>trial:</em> consecutive trial number per subject</li> <li><em>target_label:</em> index of the object to approach (0 to 4)</li> </ul> </li> <li>Data in world coordinates: <ul> <li><em>head_{x,y,z}:</em> head position</li> <li><em>head_quat_{w,x,y,z}:</em> head orientation quaternion</li> <li><em>W_gaze_{x,y,z}:</em> gaze direction</li> <li><em>W_r_hand_{x,y,z}:</em> right hand position</li> <li><em>W_r_hand_quat_{w,x,y,z}:</em> right hand orientation quaternion</li> <li><em>W_l_hand_{x,y,z}:</em> left hand position</li> <li><em>W_l_hand_quat_{w,x,y,z}:</em> left hand orientation quaternion</li> <li><em>W_object_i_{x,y,z}:</em> position of object i</li> <li><em>W_object_i_quat {w,x,y,z}</em>: orientation quaternion of object i</li> </ul> </li> <li>Data in egocentric coordinates (head coordinate system). This data is provided for convenience and can be derived from the other data: <ul> <li><em>gaze_{x,y,z}:</em> gaze direction</li> <li><em>r_hand_{x,y,z}:</em> right hand position</li> <li><em>r_hand_quat_{w,x,y,z}:</em> right hand orientation quaternion</li> <li><em>l_hand_{x,y,z}:</em> left hand position</li> <li><em>l_hand_quat_{w,x,y,z}:</em> left hand orientation quaternion</li> <li><em>object_i_{x,y,z}:</em> position of object i</li> <li><em>object_i_quat {w,x,y,z}</em>: orientation quaternion of object i</li> </ul> </li> </ul> <p><strong>Acknowledgment:</strong></p> <p>This work was supported by the <a href="https://robdekon.de/">ROBDEKON</a> project of the German Federal Ministry of Education and Research.</p>