Multimodal Sensory Learning for Object Manipulation

<p><strong>Multimodal Manipulation Learning Database</strong></p> <p>The dataset consists of data recordings for object manipulation with audio-tactile sensory feedback for object handover. It captures&nbsp;the auditory and tactile signals of a Kuka IIWA robot with an Allegro hand holding a plastic container containing different materials. The robot manipulates the container with vertical shaking and rotation motions. The data consists of force/pressure measurements on the Allegro hand using a Tekscan tactile skin sensor, auditory signals from a microphone, and the joints data of the IIWA robot and the Allegro hand joints.&nbsp;</p> <p><strong>Dataset</strong></p> <p>Each datafile is a rosbag file containing the data recording from one trial of a robot motion with one material, with rostopics on the following data:</p> <ul> <li>Kuka IIWA 7 Joint data:&nbsp;/iiwa/TorqueController/command /iiwa/eePose /iiwa/joint_states</li> <li>Allegro hand joint data: /allegro_hand_right/joint_states</li> <li>Tekscan sensor recording (tactile force/pressure sensor data on hand):&nbsp;/tekscan/frame</li> <li>Audio data (for microphone attached to hand):&nbsp;/audio/audio /audio/audio_info</li> <li>Experiment information: /trialInfo <ul> <li>which contains: <ul> <li>trial information (motion type, speed, etc.)</li> <li>start/stop of different phases of the trials</li> </ul> </li> </ul> </li> </ul> <p><strong>Motion Types</strong></p> <p>The database contains recordings for the robot executing two different motion types: vertical shaking of the object and rotation of the object.</p> <p><strong>Materials</strong></p> <p>The database contains recordings for 5 different material classes in the plastic container, as shown below: empty, vitamins, gummies, cornflakes, and rice. We used approximately the same volume of each material for each trial. We tested each material class and motion combination for a total of 10 different experimental conditions and collected 30 trials for each condition.</p> <p>The vertical motion dataset was entirely collected on 2021/08/25. The rotation dataset was split into two day. The empty, gummies and rice class data was collected on 2021/08/26. The vitamins and cornflakes classes were collected on 2021/09/13.</p> <p><strong>Database Setup</strong></p> <p>The database consists of the data&nbsp;in two formats: annotated (&#39;annotated_bags_mml.zip&#39;) and unannotated/numbered filenames (&#39;numbered_bags_mml.zip&#39;)&nbsp;datasets. The data in the two datasets are identical- the annotated filename dataset has the experimental descriptions in the filename directly (as described below).</p> <p>The annotated filenames dataset (&#39;annotated_bags_mml.zip&#39;)&nbsp;consists of a single directory with all 300 rosbag datafiles (10 experimental conditions, 30 trials each). Each rosbag (<code>.bag</code>) is saved in the directory, with filename specified (&#39;Date Recorded YYYYMMDD&#39; + &#39;_motion&#39; + &#39;_material&#39; + &#39;_trialID&#39; + &#39;.bag&#39;). Motion Types are: {&#39;vertical&#39;, &#39;rotation&#39;}.&nbsp; Materials are: {&#39;empty&#39;, &#39;cornflakes&#39;, &#39;gummies&#39;, &#39;rice&#39;, &#39;vitamins&#39;}. For each experimental condition, there are 30 datafiles with trial IDs from 0-29.</p> <p>All data recordings for the vertical motion have filenames: &#39;20210825_vertical_+ &#39;material&#39; + &#39;trialID&#39; +&#39;.bag).&nbsp;For the rotation motion, the empty, gummy and rice classes have filenames: &#39;20210826_rotation_+ &#39;material&#39; + &#39;trialID&#39; +&#39;.bag). For cornflakes and vitamins classes, the filenames are:&nbsp;&#39;20210913_rotation_+ &#39;material&#39; + &#39;trialID&#39; +&#39;.bag).</p> <p>The numbered/unannotated file dataset (&#39;numbered_bags_mml.zip&#39;) consists of the same 300 data files as in the annotated dataset&nbsp;except here the&nbsp;filenames are numbered&nbsp;&#39;{000-299}.bag&#39;. The directory contains a spreadsheet (&#39;annotations.csv&#39;)&nbsp;listing the experimental descriptions for each file name. The columns of the xls spreadsheet are {&#39;Bagfile name&#39;, &#39;Year&#39;, &#39;Month&#39;, &#39;Day&#39;, &#39;Motion/Movement (mvt_type)&#39;, &#39;Material&#39;, &#39;Trial ID&#39;}, where {Year, Month, Day} refer to the date that trial data&nbsp;was collected (either 2021/08/25, 2021/08/26, or 2021/09/13).&nbsp;</p>