Find research datasets worth reusing

opencc-by-4.0Dec 2023View details →

IMU and marker-based optical motion capture from a humanoid robot

The motion capture contains walking trials from the lower body of the humanoid robot Reem-C from Pal Robotics (Barcelona, Spain). Seven IMUs were attached on the foot, lower leg, upper leg and pelvis segments. IMU data was collected at 100 Hz. Moreover, the robot motion was captured with a marker-based optical system (Qualisys AB, Göteborg, Sweden) at 150 Hz. The focus of the dataset was mainly walking. There are three trials, each with a length of about 6.5 minutes. The dataset contains the definition of the skeleton (segment lengths and coordinate locations), the actual IMU readings and the pose or kinematics from the optical system.

opencc-by-4.0Jan 2024View details →

Full Body Motion Capture of Single Individuals Following External Perturbations from Different Directions

This dataset is composed of C3D files corresponding to full body motion of participants undergoing external perturbation at shoulder height with different sensory conditions. The temporal force profiles of the perturbations are also available. The following experiment received ethical approval from an ethics committee and all participants signed an informed consent form relative to the processing of their data.  The experiments were carried on 21 healthy young adults (10 females, 11 males). All were between 20 and 38 yo with a mean age of 27.2 (std: 4.2). Mean mass was 70.2 (std: 12.1) kg and height was 1.74 (std: 0.08) m.  Participants motion was recorded using 45 reflective markers and a 23 Qualisys camera system (200Hz).  The markers were placed on participants following standardised anatomical landmarks.  The output signal of the force sensor was processed using a Butterworth low pass filter with a 5Hz cutoff frequency without phase shift.  The force sensor was synchronised with the motion capture software. Tree reflective markers were also placed along the pole in order to retrieve the exact direction of the perturbations.

opencc-by-4.0Jan 2024View details →

Kuopio gait dataset: motion capture, inertial measurement and video-based sagittal-plane keypoint data from walking trials

This dataset contains motion capture (3D marker trajectories, ground reaction forces and moments), inertial measurement unit (wearable Movella Xsens MTw Awinda sensors on the pelvis, both thighs, both shanks, and both feet), and sagittal-plane video (anatomical keypoints identified with the OpenPose human pose estimation algorithm) data. The data is from 51 willing participants and collected in the HUMEA laboratory in the University of Eastern Finland, Kuopio, Finland, between 2022 and 2023. All trials were conducted barefoot. The file structure contains an Excel file containing information of the participants, data folders under each subject (numbered 01 to 51), and a MATLAB script. The Excel file has the following data for the participants: <ul> <li>ID: ID of the participants from 1 to 51</li> <li>Age: age of the participant in years</li> <li>Gender: biological sex as M for male, F for female</li> <li>Leg: the participant's dominant leg, identified by asking which foot the participant would use to kick a football; R for right, L for left</li> <li>Height: height of the participant in centimeters</li> <li>Invalid_trials: list of invalid trials in the motion capture data (MOCAP) data, usually classified as such because the participant did not properly step on the middle force plate</li> <li>IAD: inter-asis distance in millimeters, the distance between palpated left and right anterior superior iliac spine, measured with a caliper</li> <li>Left_knee_width: width of the left knee from medial epicondyle to lateral epicondyle in millimeters, palpated and measured with a caliper</li> <li>Right_knee_width: same as above for the right knee</li> <li>Left_ankle width: width of the left ankle from medial malleolus to lateral malleolus in millimeters, palpated and measured with a caliper</li> <li>Right_ankle_width: same as above for the right ankle</li> <li>Left_thigh_length: the distance between the greater trochanter of the left femur and the lateral epicondyle of the left femur in millimeters, palpated and measured with a measuring tape</li> <li>Right_thigh_length: same as above for the right thigh</li> <li>Left_shank_length: the distance between the medial epicondyle of the femur and the medial malleolus of the tibia in millimeters, palpated and measured with a measuring tape</li> <li>Right_shank_length: same as above for the right shank</li> <li>Mass: mass in kilograms, measured on a force plate just before the walking measurements</li> <li>ICD: inter-condylar distance of the knee of the dominant leg, measured from low-field MRI</li> <li>Left_knee_width_mocap: distance between reflective MOCAP markers on the medial and lateral epicondyles of the knee in millimeters, measured from a static standing trial; -1 for missing (subject did not have those markers)</li> <li>Right_knee_width_mocap: same as above for the right knee</li> </ul> The folders under each subject (folders numbered 01 to 51) are as follows: <ul> <li>imu: "Raw" inertial measurement unit (IMU) data files that can be read with Xsens Device API (included in Xsens MT Manager 4.6, which may be unavailable these days, not sure). You won't need this if you use the data in the imu_extracted folder.</li> <li>imu_extracted: IMU data extracted from those data files using the Xsens Device API, so you don't have to. <ul> <li>The data is saved as MATLAB structs where the fields are named as a sensor ID (e.g., "B42D48"). The sensor IDs and their corresponding IMU locations are as follows: <ul> <li>pelvis IMU: B42DA3</li> <li>right femur IMU: B42DA2</li> <li>left femur IMU: B42D4D</li> <li>right tibia IMU: B42DAE</li> <li>left tibia IMU: B42D53</li> <li>right foot IMU: B42D48</li> <li>left foot IMU: B42D51 (except for subjects 01 and 02, where left foot IMU has the ID B42D4E)</li> </ul> </li> <li>Some of the data are just zeros as they couldn't be read from these sensors, but under each sensor, the fields "calibratedAcceleration", "freeAcceleration", "time", "rotationMatrix", and "quaternion" contain usable data. <ul> <li>time: Contains time stamps of the measurement at each frame recorded at 100 Hz, so if you remove the first value from all values in the time vector and divide the result by 100, you will get the time in seconds from the beginning of the walking trial.</li> <li>calibratedAcceleration and freeAcceleration: Contain triaxial acceleration data from the accelerometers of the IMU. freeAcceleration is just calibratedAcceleration without the effect of Earth's gravitational acceleration.</li> <li>rotationMatrix: Orientations of the IMU as rotation matrices.</li> <li>quaternion: Orientations of the IMU as quaternions.</li> </ul> </li> </ul> </li> <li>openpose: Trajectories of the keypoints identified from sagittal plane video frames, saved as json files. <ul> <li>The keypoints are from the BODY_25 model of OpenPose (https://cmu-perceptual-computing-lab.github.io/openpose/web/html/doc/md_doc_02_output.html).</li> <li>Each frame in the video has its own json file.</li> <li>You can use the function in the script "OpenPose_to_keypoint_table.m" in the root folder to read the keypoint trajectories and confidences of all frames in a walking trial into MATLAB tables. The function takes as argument the path to the folder containing the json files of the walking trial.</li> </ul> </li> <li>Note that some subjects (11, 14, 37, 49) do not have keypoint and IMU data.</li> </ul> The folders under each subject are divided into three ZIP archives with 17 subjects each. The script "OpenPose_to_keypoint_table.m" is a MATLAB script for extracting keypoint trajectories and confidences from JSON files into tables in MATLAB. Publication in Data in Brief: <a href="https://doi.org/10.1016/j.dib.2024.110841" target="_blank" rel="noopener">https://doi.org/10.1016/j.dib.2024.110841</a> Contact: Jere Lavikainen, jere.lavikainen@uef.fi

opencc-by-nc-4.0Nov 2021View details →

Context-Aware Activity Recognition in Logistics (CAARL) – A optical marker-based Motion Capture Dataset

CAARL is a freely accessible logistics-dataset for human activity recognition, which contains human movement and context information from two subjects. The context information includes the positions of objects such as two picking carts, a packaging table, different racks, a base and three entrances. In the ’Innovationlab Hybrid Services in Logistics’ at TU Dortmund University, two picking and one packing scenarios were recorded using an optical marker based motion capture system. Each subject and object is equipped with several markers. 140 minutes of human movements have been labelled and categorised into 8 activity classes and 19 binary coarse-semantic descriptions, also called attributes. The labelled human movements are synchronised with the context information. They have exactly the same sampling rate (same start and end). The oMoCap data is in csv format. Further formats (e.g. C3D) are available on request. CAARL is based on the set-up and scenarios of the LARa dataset, which contains only human movements. Information about LARa can be found in the dataset and the associated paper: <ul> <li>Dataset: “Logistic Activity Recognition Challenge (LARa) – A Motion Capture and Inertial Measurement Dataset”, Zenodo 2020, DOI: <a href="https://doi.org/10.5281/zenodo.3862782">10.5281/zenodo.3862782</a></li> <li>Paper: “LARa: Creating a Dataset for Human Activity Recognition in Logistics Using Semantic Attributes”, Sensors 2020, DOI: <a href="https://doi.org/10.3390/s20154083">10.3390/s20154083</a></li> </ul>   If you use the CAARL dataset for research, please cite the following paper: “Context-Aware Human Activity Recognition in Industrial Processes”, Sensors 2021, DOI: <a href="https://doi.org/10.3390/s22010134">10.3390/s22010134</a>

opencc-by-4.0Aug 2021View details →

Motion Capture Benchmark of Industrial Tasks for Ergonomic Assessment and European Historic Crafts

General Info: This benchmark provides motion capture (MoCap) files in .bvh form. The recordings were done in the span of May 2019 to January 2020 for the needs of the <a href="https://collaborate-project.eu/">CoLLaboratE</a> and <a href="http://www.mingei-project.eu/">MINGEI</a> H2020 projects funded by the European Commission. The tasks included are: <ul> <li>TV assembling</li> <li>Airplane component manufacturing</li> <li>High ergonomic hazard motions </li> <li>Silk-Weaving</li> <li>Glassblowing</li> <li>Mastic Cultivation</li> </ul> The TV assembly and airplane component manufacturing tasks were recorded in real-world conditions inside the factory during the actual production of the items. The high ergonomic hazard motions were recorded in a controlled lab environment and serve as baseline/prototype motions for ergonomic risk assessment. The silk-weaving, glassblowing, and mastic cultivation data sets were created, corresponding to movements performed by skilled craftsmen and mastic farmers. These data sets were produced in order to extract the expert's gestural knowledge and analyze their dexterity while doing their crafts. Naming Convention: All files in this benchmark follow a strict naming convention to allow for easier parsing by scripts. The names have a total of 12 or 13 characters that convey the following information: <ul> <li>The first three or fours characters label the recording session (e.g., LAB, PLN, GBBC, MCSN, etc.)</li> <li>The next three characters label the subject number (e.g., S01, S02, S03, etc.)</li> <li>The next three characters label the posture or gesture number (e.g., P01, P02, G01, G02, etc.)</li> <li>The final three characters label the repetition number (e.g., R01, R02, R03, etc.)</li> </ul> For example, LABS02P03R01 denotes a lab recording of the second subject, performing the third posture for the first time. Recording Sessions: There are six recording sessions in this benchmark, the ergonomic risk motion recorded in the lab (denoted as "LAB"), the construction of an airplane component (denoted as "PLN"), and the assembling and packaging of TVs (denoted as "TV*"), the silk weaving (denoted as "SW*"), glassblowing (denoted as "GB*"), and mastic cultivation (denoted as "MC*"). The postures are the following: LAB: <ul> <li>Standing: <ul> <li>P01: The subject stays in I-pose</li> <li>P02: The subject rotates his/her torso to the left as far the person can</li> <li>P03: The subject will laterally bend his/her torso to the left for 6 seconds</li> <li>P04: The subject bends more than 20° but less than 60°</li> <li>P05: The subject bends more than 20° but less than 60° while rotating and laterally bending the torso to the left</li> <li>P06: The subject stretches his/her arms, and bends forward more than 20° but less than 60° while rotating and laterally bending the torso to the left</li> <li>P07: The subject bends more than 60°</li> <li>P08: The subject bends more than 60° while rotating and laterally bending the torso to the left</li> <li>P09: The subject stretches his/her arms, and bends forward more than 60° while rotating and laterally bending the torso to the left</li> <li>P10: The subject upright, raises the elbows above the shoulder level with the forearms bent 90° (</li> <li>P11: The subject raises the elbows above the shoulder level with the forearms bent 90° while rotating and laterally bending the torso to the left</li> <li>P12: The subject raises the elbows above the shoulder level with the arms stretched while rotating and laterally bending the torso to the left</li> <li>P13: The subject upright, raises the hands above the head</li> <li>P14: The subject raises the hands above the head with the arms stretched while rotating and laterally bending the torso to the left</li> </ul> </li> <li>Sitting on a chair: <ul> <li>P15: The subject sits upright</li> <li>P16: The subject bends forward more than 60°</li> <li>P17: The subject bends forward more than 60° while rotating and laterally bending the torso to the left</li> <li>P18: The subject stretches the arms, and bends forward more than 60° while rotating and laterally bending the torso to the left</li> <li>P19: The subject raises the hands above the head with arms stretched</li> <li>P20: The subject raises the hands above the head with the arms stretched while rotating and laterally bending the torso to the left</li> </ul> </li> <li>Kneeling: <ul> <li>P21: The subject stays upright</li> <li>P22: The subject rotates the torso to the left as far he/she can</li> <li>P23: The subject will laterally bend the torso to the left for 6 seconds</li> <li>P24: The subject bends more than 60°</li> <li>P25: The subject bends more than 60° while rotating and laterally bending the torso to the left</li> <li>P26: The subject stretches the arms, and bends forward more than 60° while rotating and laterally bending the torso to the left</li> <li>P27: The subject upright, raises the elbows to the shoulder level with the arms stretched</li> <li>P28: The subject raises the elbows to the shoulder level with the arms stretched while rotating and laterally bending the torso to the left</li> </ul> </li> </ul> The TV assembling tasks are further divided. The subtasks are: packing the TVs on a stack for shipping (denoted as "TVP" for medium-sized TVs and "TVL" for larger TVs), placing assembling and placing electronic circuit boards on the chassis (denoted as "TVB"), and screwing the boards on the TV chassis (denoted as "TV_"). Each task is comprised of a number of postures.   TV Assembling: <ul> <li>Assembling the board and placing it on the TV chassis (TVB): <ul> <li>P01: Reaching high, above the shoulder level, to pick one component</li> <li>P02: Reaching low, below the knee level, to pick up the second component</li> <li>P03: Connecting the components and placing the board on the chassis to be screwed</li> </ul> </li> <li>Screwing an electrical circuit board on the TV chassis (TV_) : <ul> <li>P01: A screw is placed on a power tool and it is being screwed on the chassis. The process is repeated four times</li> </ul> </li> <li>Preparing TVs for Shipping (TVP & TVL): <ul> <li>P01: Placing TVs on a wooden pallet (bottom level)</li> <li>P02: Preparing to wrap the bottom level with a membrane</li> <li>P03: Wrapping the bottom level</li> <li>P04: Placing TVs on top of the bottom level (second level)</li> <li>P05: Placing TVs on top of the second level (third level)</li> <li>P06: Wrapping the second level with a plastic membrane</li> <li>P07: Wrapping the third level with a plastic membrane</li> <li>P08: Placing TVs on top of the third level (fourth level)</li> <li>P09: Wrapping the fourth level with a plastic membrane</li> </ul> </li> </ul> Riveting of an airplane floater (PLN): <ul> <li>P01: Rivet with the pneumatic hammer.</li> <li>P02: Prepare the pneumatic hammer and grab rivets. </li> <li>P03: Place the bucking bar to counteract the incoming rivet.</li> </ul> The tasks recorded for silk weaving, glassblowing, and mastic cultivation data sets were segmented by gestures (e.g., G01, G02, etc.) . The tasks recorded for these three data sets are the following: Silk weaving (SW*): <ul> <li>The creation of the punch cards (SWPC).</li> <li>Preparation of the beam (SWPB).</li> <li>Wrapping of the beam (SWWB).</li> <li>Jacquard weaving with small loom (SWSL).</li> <li>Jacquard weaving with medium size loom (SWML).</li> <li>Jacquard weaving with large loom (SWLL).</li> </ul> Glassblowing (GB*): <ul> <li>Beak cutting (GBBC).</li> <li>Blowing and shaping (GBBS).</li> <li>Cervix refining (GBCR).</li> <li>Cord laying (GBCL).</li> <li>Finish details (GBFD).</li> <li>Handle laying (GBHL).</li> <li>Transfer to punty (GBTP).</li> <li>Leg and foot laying (GBLF).</li> </ul> Mastic Cultivation (MC*): <ul> <li>Scrapping with new tool (MCSN).</li> <li>Scrapping with old tool (MCSO).</li> <li>Sweeping (MCSW).</li> <li>Dusting (MCDU).</li> <li>Embroidery A (MCEA).</li> <li>Embroidery B (MCEB).</li> <li>Embroidery with an axe (MCEX).</li> <li>Gathering (MCGA).</li> <li>Harvesting (MCHA).</li> <li>Wiping (MCWI).</li> <li>Shifting A (MCSA).</li> <li>Shifting B (MCSB).</li> <li>Cleaning with the wind (MCCW).</li> </ul> The motion capture files were processed and segmented with a 3D character animation software (MotionBuilder, Autodesk Inc., San Rafael, CA. USA) and exported to Biovision Hierarchy (BVH) files.

opencc-by-4.0Sep 2022View details →

Motion Capture Data for Hand Motion Embodiment

<h1>Dataset</h1> A dataset of human manipulation actions recorded with a motion capture system. A Qualisys motion capture system was used to record the data. We tracked individual finger movements as well as the position and orientation of the right hand. Some recordings contain additional markers at the back, shoulder, and elbow. The motion capture setup is explained <a href="https://dfki-ric.github.io/hand_embodiment/motion_capture_setup.html">here</a>. The dataset contains the original recordings of manipulation actions as well as metadata with annotations of relevant parts of the recordings (labels, start, end). Recordings are exported from the Qualisys Track Manager (QTM) as tab-separated value (TSV) files. Metadata is provided in JSON format. Related software is available at <a href="https://github.com/dfki-ric/hand_embodiment">github.com/dfki-ric/hand_embodiment</a>, which also contains code to load and use the dataset. <h1>Publication</h1> This dataset was introduced in the following paper: Alexander Fabisch, Manuela Uliano, Dennis Marschner, Melvin Laux, Johannes Brust, Marco Controzzi: "A Modular Approach to the Embodiment of Hand Motions from Human Demonstrations", Proceedings of IEEE-RAS International Conference on Humanoid Robots 2022. It is available from <a href="https://arxiv.org/abs/2203.02778">arxiv.org</a> as a preprint or from <a href="https://ieeexplore.ieee.org/document/10000165">IEEE</a>. If you use the dataset, please cite the paper as: <blockquote> @INPROCEEDINGS{Fabisch2022,   author={Fabisch, Alexander and Uliano, Manuela and Marschner, Dennis and Laux, Melvin and Brust, Johannes and Controzzi, Marco},   booktitle={2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids)},    title={A Modular Approach to the Embodiment of Hand Motions from Human Demonstrations},    year={2022},   pages={801--808},   doi={10.1109/Humanoids53995.2022.10000165} } </blockquote> <h1>Ethics Approval</h1> Experimental protocols were approved by the ethics committee of the University of Bremen. Written informed consent was obtained from all participants for participation in the study and to publish this dataset. <h1>Origin and Funding</h1> This dataset is provided by the Robotics Innovation Center, DFKI GmbH. This work was supported by the European Commission under the Horizon 2020 framework program for Research and Innovation (project acronym: APRIL, project number: 870142).

opencc-by-4.0Jan 2022View details →

DeepDance: Motion capture data of improvised dance (2019)

When using this resource, please cite Wallace, B., Nymoen, K., Martin, C.P & Tøressen, J. DeepDance: Motion capture data of improvised dance (2019) (version 2.0). Zenodo 10.5281/zenodo.5838178 Abstract This dataset comprises full-body motion capture of improvised dance as well as corresponding audio files. 30 dancers were recorded individually, improvising to six different audio files. The motion was captured in units of mm at 240Hz using a Qualisys infra-red optical system. The experiment was carried out at the University of Oslo in October 2019. For each dancer, 3 performances are recorded for each musical piece, resulting in 540 1-minute motion capture files. The dataset was collected for use as training data in deep learning for motion generation. This dataset also includes MATLAB code to visualize the motion capture files.   Music <ul> <li>Skarphedinsson, M. Wallace, B. (2019). “Song a”</li> <li>Skarphedinsson, M. Wallace, B. (2019). “Song b”</li> <li>Skarphedinsson, M. Wallace, B. (2019). “Song c”</li> <li>Skarphedinsson, M. Wallace, B. (2019). “Song d”</li> <li>Skarphedinsson, M. Wallace, B. (2019). “Song f”</li> <li>LaClair, J. Bounce. Jesse LaClair, (2018) Referenced here as “Song e”</li> </ul>   Data Description The following data types are provided: <ul> <li>Motion (marker position): Recorded with Qualisys Track Manager and saved as tab-separated .tsv files.</li> <li>Stimuli: audio .wav files containing 1 minute of the tracks described above.</li> <li>MATLAB script for animating the tsv files. (requires the MoCap Toolbox)</li> </ul> Note: Recordings which contained errors such as missing markers have been replaced by subject 001.  Acknowledgements This work was partially supported by the Research Council of Norway through its Centres of Excellence scheme, project number 262762.   Conflicts of Interest The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

opencc-by-4.0May 2022View details →

Evaluating 3D Human Motion Capture using Apple ARKit against the Vicon System: A Dataset

A journal paper which was published in Applied Sciences gives detailed information about the dataset. Reimer, L.M.; Kapsecker, M.; Fukushima, T.; Jonas, S.M. Evaluating 3D Human Motion Capture on Mobile Devices. Appl. Sci. (2022) <a href="https://www.mdpi.com/2076-3417/12/10/4806">https://www.mdpi.com/2076-3417/12/10/4806</a> Please cite the corresponding paper when using the dataset.   A dataset containing anonymized exercise data for eight exercises from ten subject. The exercise data was recorded with two iPads 11" (2021 version, Apple Inc., Cupertino, CA, USA) and a Vicon system. The two iPads were positioned frontal and in a 30° angle to the left side of the subject. The Vicon system used 14 cameras and captured the motion using the Full-body Plug-in-gait model. The dataset contains 220 files, 22 per subject. The structure of the dataset contains 10 folders, one per subject. Each folder contains two subfolders: ARKit and Vicon. Each ARKit folder holds two CSV files. Each Vicon folder holds 16 files, two per exercise: a .csv file with the motion data and a .xcp file containing meta data about the recording, including the camera setup and start/stop timestamps. Du to export problems, the ARKit files for the Side View do not always contain all joint data. The upper body joints are only available for three out of the ten subjects for the Side View.

A Facial Motion Capture System Based on Neural Network Classifier Using RGB-D Data-Figure 11. Accuracy of different method for unseen faces

Table 3 shows the mode detection accuracy of the proposed method and its combination with two other methods (uniform LBP and circular LBP) for different people. The overall accuracy of the proposed procedure is calculated as this way one video is chosen as input, and after mode detection the three aforementioned steps are applied on this video. The obtained feature vectors are given to the neural network and the corresponding labels to each frame are regarded as output. Afterwards, the overall accuracy is calculated from the confusion matrix. However, it should be noted that the expression detection criteria are the observation of a certain number of subsequent similar labels and in the case of observing a limited or sparse number of different labels the final label would not change. Figure 10 and 11 show result of different methods for seen and unseen data respectively. Table 4 shows results for seen data with proposed method and uniform LBP.

A Facial Motion Capture System Based on Neural Network Classifier Using RGB-D Data-

A Facial Motion Capture System Based on Neural Network Classifier Using RGB-D Data-Figure 9. Results of our facial motion capture system(a,b,c,d)

In test procedures, single video feature vectors consisting of different expressions are given to the neural network and the network produces the corresponding labels for each frame as output. If there is a mode in a video which is not available in the data base, the nearest available mode's label to this mode is produced. For example, in test3 and test6 videos, the surprise expression (that have been showed with number 7) is recognized as open mouth expression. At the end, considering the certain numbers of subsequent similar labels (at least 10 frames, because the minimum number of one modes' frames is related to “rising the eyebrow” mode that takes 10 frames), the expressions are detected, and a 3D show of these expressions are represented. For instance, in test8 videos that have been obtained from unseen face, the “smiling” and “open mouth” expressions are well recognized, but expressions related to rising the eyebrows are not detected properly and all the corresponding frames to this expression are regarded as normal expression. Figure 9 shows example of generated 3D models.

opencc-by-4.0May 2018View details →

A Facial Motion Capture System Based on Neural Network Classifier Using RGB-D Data-Figure 8. 3D model of some facial expressions

Face region is separated precisely from video frames by using a segmentation method based on skin color. The depth data corresponding to this separated area is taken for a 3D representation from depth data corresponding to each frame. At the end, a file is prepared for each frame consisting of face points with 6 features: X, Y, depth, red, green and blue color. These data are used for producing a 3D model and a graphical avatar for each frame (Figure 7). Figure 8 shows 3D model of some facial expressions.

A Facial Motion Capture System Based on Neural Network Classifier Using RGB-D Data-Figure 6. Proposed feed-forward neural network classifier

After the feature extraction stage, neural network is used for classifying the modes. In this study, the utilized expressions are normal, smiling, open mouth, rising the eyebrows, anger and pursing modes. In fact, they are some selective modes for face movements. It should be noted that the modes can be increased but in this case we work with these six modes. This paper used three layers feed-forward neural network (Figure 6). The proposed neural network includes 800 nodes for the input layer (400 nodes for U matrix and 400 nodes for V matrix), 100 nodes for the hidden layer and 6-nodes for output layer. From the collected data 70% are used for training, 15% for validation and the last 15% are used to evaluate the neural network.

A Facial Motion Capture System Based on Neural Network Classifier Using RGB-D Data-Figure 5. Examples of the circular LBP (Huang et al., 2011)

One limitation of the basic LBP operator is that its small 3x3 neighborhood cannot capture dominant features with large scale structures. To deal with the texture at different scales the operator was later generalized to use neighborhoods of different sizes. A local neighborhood is defined as a set of sampling points evenly spaced on a circle which is centered at the pixel to be labeled. The sampling points that do not fall within the pixels are interpolated using bilinear interpolation, thus allowing for any radius and any number of sampling points in the neighborhood. Figure 5 shows some examples of the extended LBP operator where the notation (P, R) denotes a neighborhood of P sampling points on a circle of radius of R.

A Facial Motion Capture System Based on Neural Network Classifier Using RGB- Figure 4. An example of the uniform LBP operator (Huang et al., 2011)

The original LBP operator labels the pixels of an image by means of decimal numbers called Local Binary Patterns or LBP codes, which encode the local structure around each pixel. It proceeds thus as illustrated in figure 4: Each pixel is compared with its eight neighbors in a 3x3 neighborhood by subtracting the center pixel value. The resulting strictly negative values are encoded with 0 and the others with 1. A binary number is obtained by concatenating all these binary codes in a clockwise direction starting from the top-left one and its corresponding decimal value is used for labeling. The derived binary numbers are referred to as Local Binary Patterns or LBP codes.

A Facial Motion Capture System Based on Neural Network Classifier Using RGB-D Data-Figure 7. Avatar 3D model generation

A Facial Motion Capture System Based on Neural Network Classifier Using RGB-D Data-Figure 3. Feature vectors of facial expression in database

Figure 3 shows feature vectors of facial expression of our database. Matrices ‘U’ and ‘V’ values that are obtained from this algorithm are used as feature vectors. The ‘U’ matrix represents the position and the ‘V’ matrix represents the change of direction. In the following, the proposed method is combined with some other feature extraction methods (LBP uniform approach and LBP circular approach) and the obtained results will be mentioned.

A Facial Motion Capture System Based on Neural Network Classifier Using RGB-D Data-Figure 2. Facial expression recognition in proposed method

In this stage, a video is prepared using the color data captured from Kinect camera. The face region in each frame is obtained from the video using Viola-Jones algorithm (Figure 2). Because of different distance from the Kinect camera, the obtained images from the face must be re-sized, in order to have the same size. At the end, the colored images are converted to gray-scaled images.