REFLEX Dataset: A Multimodal Dataset of Human Reactions to Robotic Failures and Subsequent Robotic Explanations.

<p>REFLEX Dataset is a comprehensive collection of multimodal Human Behavioral reactions to Robot Failures and Explanations. <br><br>The version 1.0 is a representative sample of this dataset with the reactions from 5 users out of a total 55 users.</p> <p>This version 1.1.0 is the full dataset with the reactions from a total 55 users.<br><br>Please refer to the Readme in the zipped file for further information.</p> <p><br>This data was recorded from a user study and has been processed for anonymization.</p> <h2>About Data</h2> <p>This description gives a detailed process on how the data was collected. It should describe the conditions under which the data was recorded and also the devices used to record the data.</p> <h3>Data Organisation</h3> <p>The data is structured by strategy and participant, as shown below:</p> <pre><code>Strategy Dir/ -Participant Dir/ - analysis - questonnaire - facetorch - openface - gaze - hume - body - voice - time - video_cam1 - video_cam2 </code></pre> <p>We employed five different strategies (C1, C2, C3, D1, D2), collecting data from 11 participants for each strategy. The data for each participant is organized within a corresponding folder.</p> <p>Participants are labeled based on their assigned strategy. For example, data from the first participant under the &ldquo;Fixed Low&rdquo; (C1) strategy can be found in the C1-1 subfolder within the C1 directory.</p> <h3>Collected Data</h3> <p>Each participant folder contains various datasets related to different modalities. All visual data are collected using the camera 1 video. The collected data are outlined below:</p> <ul> <li> <p><strong>Anonymized Videos</strong>&nbsp;(<code>video_cam1.mp4</code>,&nbsp;<code>video_cam2.mp4</code>) - Visual Representation:</p> <ul> <li>Video from camera 1 (robot side of view)</li> <li>Video from camera 2 (experiment side of view)</li> </ul> </li> <li> <p><strong>Analysis</strong>&nbsp;(<code>analysis.csv</code>) - Failure Instance Description:</p> <ul> <li>Failure type</li> <li>Explanation strategy</li> <li>Explanation level</li> <li>Phase (Pre, Failure, Explanation, Resolution)</li> <li>Start/End frame and time of failure</li> <li>Task Resolved</li> </ul> </li> <li> <p><strong>Questionnaire</strong>&nbsp;(<code>questionnaire.csv</code>) - Failure Instance Description:</p> <ul> <li>Participant Data (Age, Gender, etc)</li> <li>Answers of explanation-satisfaction rate question for rounds and overall experiment</li> </ul> </li> <li> <p><strong>Facetorch</strong>&nbsp;(<code>facetorch.csv</code>) -&nbsp;<a href="https://github.com/tomas-gajarsky/facetorch" target="_blank" rel="nofollow noopener">Facetorch</a>&nbsp;- Face:</p> <ul> <li>Arousal/Valence levels</li> <li>Presence of Facial Action Units (AUs)</li> <li>Dominant Emotion (Out of six basic emotions and neutral)</li> </ul> </li> <li> <p><strong>OpenFace</strong>&nbsp;(<code>openface.csv</code>) -&nbsp;<a href="https://github.com/TadasBaltrusaitis/OpenFace" target="_blank" rel="nofollow noopener">OpenFace</a>&nbsp;- Face, Gaze, Head:</p> <ul> <li>Eye Gaze (2D and 3D Landmarks)</li> <li>Eye Direction (vector and in radians)</li> <li>Head Pose Estimation (Pose Estimation, Rotation)</li> <li>Face Landmarks (2D and 3D Landmarks)</li> <li>Facial Action Units (0.0-1.0 intensity scores, occurrences)</li> </ul> </li> <li> <p><strong>Gaze</strong>&nbsp;(<code>gaze.csv</code>) - Gaze:</p> <ul> <li>Eye Gaze Classification (e.g., Robot, Task, Miscellaneous)</li> </ul> </li> <li> <p><strong>Hume</strong>&nbsp;(<code>hume.csv</code>) -&nbsp;<a href="https://www.hume.ai/" target="_blank" rel="nofollow noopener">Hume Expression Measurement API</a>&nbsp;- Face:</p> <ul> <li>48 Emotion likelihoods</li> <li>Facial Action Units (0.0-1.0 score)</li> <li>Facial Descriptions (0.0-1.0 score)</li> </ul> </li> <li> <p><strong>Voice</strong>&nbsp;(<code>speech.csv</code>) -&nbsp;<a href="https://www.hume.ai/" target="_blank" rel="nofollow noopener">Hume Expression Measurement API</a>&nbsp;- Speech:</p> <ul> <li>Speech conversation data</li> <li>Emotional likelihoods inferred from prosody</li> </ul> </li> <li> <p><strong>Body</strong>&nbsp;(<code>body.csv</code>) -&nbsp;<a href="https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker" target="_blank" rel="nofollow noopener">MediaPipe Pose Landmark Detection</a>&nbsp;- Body:</p> <ul> <li>Pose classifications (e.g., crossed arms, arms behind back)</li> <li>2D and 3D Pose Landmarks</li> </ul> </li> <li> <p><strong>Time</strong>&nbsp;(<code>time.csv</code>) -&nbsp;<a href="https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker" target="_blank" rel="nofollow noopener">MediaPipe Pose Landmark Detection</a>:</p> <ul> <li>Associated timestamp and time for each frame of camera 1 video.</li> </ul> </li> </ul> <p>Notes</p> <ul> <li>Data was synchronized based on the `video_cam1.mp4`</li> <li>The `hume.csv` and `gaze.csv` files contain data only for frames within failure periods.</li> <li>Failure events were divided into four phases:<br>&nbsp; &nbsp; 1. Pre-failure phase: Period before the failure occurs<br>&nbsp; &nbsp; 2. Failure phase: When the actual failure action takes place<br>&nbsp; &nbsp; 3. Explanation phase: When the robot provides an explanation for the failure<br>&nbsp; &nbsp; 4. Resolution phase: When the robot guides the participant to resolve the issue</li> </ul> <h2>How to Visualize Participant Data</h2> <p>Please visit the github repository: https://github.com/andreasnaoum/reflex-viz</p>