GENEA Challenge 2023 Dataset Files

<p>This Zenodo repository contains the main dataset for the GENEA Challenge 2023, which is based on the Talking With Hands 16.2M dataset.</p> <p><strong>Notation:</strong></p> <p>Please take note of the following nomenclature when reading this document:</p> <ul> <li> <p><em>main agent</em>&nbsp;refers to the speaker in the dyadic interaction for which the systems generated motions.&nbsp;</p> </li> </ul> <ul> <li> <p><em>interlocutor</em>&nbsp;refers to the speaker in front of the main agent.&nbsp;</p> </li> </ul> <p><strong>Contents:</strong></p> <p>The &ldquo;genea2023_trn&quot; and &quot;genea2023_val&quot; zip files contain audio files (in WAV format), time-aligned transcriptions (in TSV format), and motion files (in BVH format) for the training and validation datasets, respectively.</p> <p>The &quot;genea2023_test&quot; zip file contains audio files (in WAV format) and transcriptions (in TSV format) for the test set, but no motion. The corresponding test motion is available at:</p> <p><a href="https://zenodo.org/record/8146028">https://zenodo.org/record/814602</a><a href="https://zenodo.org/record/8146027">7</a></p> <p>Each zip file also contains a &quot;metadata.csv&quot; file that contains information for all files regarding the speaker ID and whether or not the motion files contain finger motion.</p> <p>Note that the speech audio in the data sometimes has been replaced by silence for the purpose of anonymisation.</p> <p>In the test set, files with indices from 0 to 40 correspond to &quot;matched&quot; interactions (the core test set), where main agent and interlocutor data come from the same conversation, whilst file indices from 41 to 69 correspond to &quot;mismatched&quot; interactions (the extended test set), where main agent and interlocutor data come from different conversations.</p> <p><strong>Folder&nbsp;structure:</strong></p> <ul> <li>main-agent/ (main agent): Encapsulates BVH, TSV, WAV data subfolders for the main agent.</li> <li>interloctr/ (interlocutor): Encapsulates BVH, TSV, WAV data subfolders for the interlocutor.</li> <li>bvh/ (motion): Time-aligned 3D full-body motion-capture data in BVH format from a speaking and gesticulating actor. Each file is a single person, but each data sample contains files for both the main agent and the interlocutor.</li> <li>wav/ (audio): Recorded audio data in WAV format from a speaking and gesticulating actor with a close-talking microphone. Parts of the audio recordings have been muted to omit personally identifiable information.</li> <li>tsv/ (text): Word-level time-aligned text transcriptions of the above audio recordings in TSV format (tab-separated values). For privacy reasons, the transcriptions do not include references to personally identifiable information, similar to the audio files.</li> </ul> <p><strong>Data processing scripts:</strong></p> <p>We provide a number of optional scripts for encoding and processing the challenge data:</p> <p>Audio: Scripts for extracting basic audio features, such as spectrograms, prosodic features, and mel-frequency cepstral coefficients (MFCCs) can be found&nbsp;<a href="https://github.com/genea-workshop/Speech_driven_gesture_generation_with_autoencoder/blob/GENEA_2022/data_processing/tools.py#L105">at this link</a>.</p> <p>Text: A script to encode text transcriptions to word vectors using FastText is available here:&nbsp;<a href="https://gist.github.com/youngwoo-yoon/c9e5d5b04994478011911f88eb15caa4">tsv2wordvectors.py</a></p> <p>Motion: If you wish to encode the joint angles from the BVH files to and from an exponential map representation, you can use scripts by&nbsp;<a href="https://github.com/simonalexanderson">Simon Alexanderson</a>&nbsp;based on the&nbsp;<a href="https://omid.al/projects/pymo/">PyMo library</a>, which are available here:</p> <ul> <li> <p><a href="https://github.com/genea-workshop/Speech_driven_gesture_generation_with_autoencoder/blob/GENEA_2022/data_processing/bvh2features.py">bvh2features.py</a></p> </li> <li> <p><a href="https://github.com/genea-workshop/Speech_driven_gesture_generation_with_autoencoder/blob/GENEA_2022/data_processing/features2bvh.py">features2bvh.py</a></p> </li> </ul> <p><strong>Attribution:</strong></p> <p>If you use this material, please cite our latest paper on the GENEA Challenge 2023. At the time of writing (2023-07-25) this is our ACM ICMI 2023 paper:</p> <p>Taras Kucherenko, Rajmund Nagy, Youngwoo Yoon, Jieyeon Woo, Teodor Nikolov, Mihail Tsakov, and Gustav Eje Henter. 2023. The GENEA Challenge 2023: A large-scale evaluation of gesture generation models in monadic and dyadic settings. In Proceedings of the ACM International Conference on Multimodal Interaction (ICMI &rsquo;23). ACM.</p> <p>Also, please cite the paper about the original dataset from Meta Research:</p> <p>Gilwoo Lee, Zhiwei Deng, Shugao Ma, Takaaki Shiratori, Siddhartha S. Srinivasa, and Yaser Sheikh. 2019. Talking With Hands 16.2M: A large-scale dataset of synchronized body-finger motion and audio for conversational motion analysis and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV &rsquo;19). IEEE, 763&ndash;772.</p> <p>&nbsp;</p> <p>The motion and audio files are based on the Talking With Hands 16.2M dataset at&nbsp;<a href="https://github.com/facebookresearch/TalkingWithHands32M/">https://github.com/facebookresearch/TalkingWithHands32M/</a>. The material is available under a CC BY NC 4.0 Attribution-NonCommercial 4.0 International license, with the text provided in LICENSE.txt.</p> <p>&nbsp;</p> <p>To find more GENEA Challenge 2023 material on the web, please see:</p> <ul> <li><a href="https://genea-workshop.github.io/2023/challenge/">https://genea-workshop.github.io/2023/challenge/</a></li> </ul> <p>&nbsp;</p> <p>If you have any questions or comments, please contact:</p> <ul> <li>The GENEA Challenge organisers &lt;genea-challenge@googlegroups.com&gt;</li> </ul>