Synthetic Multimodal Dataset using MuJoCo: UR5 Robot Motion

<p>Using the Mujoco environment, we simulated robot trajectory and transitions from one formation<br>&nbsp; &nbsp; &nbsp; &nbsp; to another. Mujoco is a 3D simulator, while Gym serves as an interface to the UR5 robot.<br>&nbsp; &nbsp; &nbsp; &nbsp; The robot has measurement units that allow the acquisition of the angles, positions,<br>&nbsp; &nbsp; &nbsp; &nbsp; and quaternions of the joints and the position of the end-effector. The robot is located on<br>&nbsp; &nbsp; &nbsp; &nbsp; a table with 4 cameras all from the same radius to the center of the robot just<br>&nbsp; &nbsp; &nbsp; &nbsp; rotated by 90&deg; for each of them. Using the described environment, we collected 1999 samples&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; at a rate of 10 samples per second.<br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; ################################################################<br>&nbsp; &nbsp; &nbsp; &nbsp; ################################################################<br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; camera views:</p> <p>&nbsp; &nbsp; &nbsp; &nbsp; id: 0<br>&nbsp; name: 'camera_0'<br>&nbsp; xmat: array([ 0.70710678, &nbsp;0.42537261, -0.56485232, -0.70710678, &nbsp;0.42537261,<br>&nbsp; &nbsp; &nbsp; &nbsp;-0.56485232, &nbsp;0. &nbsp; &nbsp; &nbsp; &nbsp;, &nbsp;0.79882181, &nbsp;0.60156772])<br>&nbsp; xpos: array([-2., -2., &nbsp;3.])</p> <p>&nbsp; &nbsp; &nbsp; &nbsp; id: 1<br>&nbsp; name: 'camera_1'<br>&nbsp; xmat: array([-0.70710678, &nbsp;0.42537261, -0.56485232, -0.70710678, -0.42537261,<br>&nbsp; &nbsp; &nbsp; &nbsp; 0.56485232, &nbsp;0. &nbsp; &nbsp; &nbsp; &nbsp;, &nbsp;0.79882181, &nbsp;0.60156772])<br>&nbsp; xpos: array([-2., &nbsp;2., &nbsp;3.])</p> <p>&nbsp; &nbsp; &nbsp; &nbsp; id: 2<br>&nbsp; name: 'camera_2'<br>&nbsp; xmat: array([ 0.70710678, -0.42537261, &nbsp;0.56485232, &nbsp;0.70710678, &nbsp;0.42537261,<br>&nbsp; &nbsp; &nbsp; &nbsp;-0.56485232, -0. &nbsp; &nbsp; &nbsp; &nbsp;, &nbsp;0.79882181, &nbsp;0.60156772])<br>&nbsp; xpos: array([ 2., -2., &nbsp;3.])</p> <p>&nbsp; &nbsp; &nbsp; &nbsp; id: 3<br>&nbsp; name: 'camera_3'<br>&nbsp; xmat: array([-0.70710678, -0.42537261, &nbsp;0.56485232, &nbsp;0.70710678, -0.42537261,<br>&nbsp; &nbsp; &nbsp; &nbsp; 0.56485232, &nbsp;0. &nbsp; &nbsp; &nbsp; &nbsp;, &nbsp;0.79882181, &nbsp;0.60156772])<br>&nbsp; xpos: array([2., 2., 3.])</p> <p>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; The camera data is stored as .png files with a size of 256x256&nbsp;</p> <p><br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; ################################################################<br>&nbsp; &nbsp; &nbsp; &nbsp; ################################################################<br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; The files angles.pt, angular_velocity.pt, angular_acceleration.pt contains information about<br>&nbsp; &nbsp; &nbsp; &nbsp; the motor data of the joints. The angles, velocity, acceleration is the information about the&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; Motor in each joint in following order:<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ['base_to_lik', 'base_to_rik', 'elbow_joint', 'shoulder_lift_joint', 'shoulder_pan_joint', 'wrist_1_joint', 'wrist_2_joint', 'wrist_3_joint']<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; ################################################################<br>&nbsp; &nbsp; &nbsp; &nbsp; ################################################################<br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; The files pose.pt and quaternion.pt contains information about<br>&nbsp; &nbsp; &nbsp; &nbsp; the body data of the robot. The pose of each element is in following order:<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ['base', 'base_link', 'box_2_link', 'box_link', 'drop_box', 'ee_link', 'forearm_link', 'left_inner_finger', 'left_inner_knuckle', 'right_inner_finger', 'right_inner_knuckle', 'robotiq_85_base_link', 'shoulder_link', 'upper_arm_link', 'world', 'wrist_1_link', 'wrist_2_link', 'wrist_3_link']<br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; ################################################################<br>&nbsp; &nbsp; &nbsp; &nbsp; ################################################################<br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; The file action.pt contains information about the used action in the corresponding time-step.&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; The action of each element is in following order:<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ['forearm_T', 'gripper_motor', 'shoulder_lift_T', 'shoulder_pan_T', 'wrist_1_T', 'wrist_2_T', 'wrist_3_T']<br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</p>