Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
8
datasets available to search
ShareScore release 0.9.0
Dataset results
8 results for “audio-visual speech”
The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)
<p><strong>Description</strong></p> <p>The Ryerson Audio-Visual Database of Emotional Speech and Song (<a href="https://affectivedatascience.com/datasets.html#ravdess">RAVDESS</a>) contains 7356 files (total size: 24.8 GB). The dataset contains 24 professional actors (12 female, 12 male), vocalizing two lexically-matched statements in a neutral North American accent. Speech includes calm, happy, sad, angry, fearful, surprise, and disgust expressions, and song contains calm, happy, sad, angry, and fearful emotions. Each expression is produced at two levels of emotional intensity (normal, strong), with an additional neutral expression. All conditions are available in three modality formats: Audio-only (16bit, 48kHz .wav), Audio-Video (720p H.264, AAC 48kHz, .mp4), and Video-only (no sound). Note, there are no song files for Actor_18.</p> <p>The RAVDESS was developed by Dr <a href="https://affectivedatascience.com/people/livingstone_sr" target="_blank" rel="noopener">Steven R. Livingstone</a>, who now leads the <a href="https://affectivedatascience.com">Affective Data Science Lab</a>, and Dr <a href="https://www.torontomu.ca/psychology/about-us/our-people/faculty/frank-russo/" target="_blank" rel="noopener">Frank A. Russo</a> who leads the <a href="https://psychlabs.torontomu.ca/smartlab/">SMART Lab</a>.</p> <p><strong>Citing the RAVDESS</strong></p> <p>The RAVDESS is released under a Creative Commons Attribution license, so please cite the RAVDESS if it is used in your work in any form. Published academic papers should use the academic paper citation for our PLoS1 paper. Personal works, such as machine learning projects/blog posts, should provide a URL to this Zenodo page, though a reference to our PLoS1 paper would also be appreciated.</p> <p><em>Academic paper citation</em></p> <blockquote> <p>Livingstone SR, Russo FA (2018) The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE 13(5): e0196391. <a href="https://doi.org/10.1371/journal.pone.0196391">https://doi.org/10.1371/journal.pone.0196391</a>.</p> </blockquote> <p><em>Personal use citation</em></p> <blockquote> <p>Include a link to this Zenodo page - <a href="../record/1188976">https://zenodo.org/record/1188976</a></p> </blockquote> <p><strong>Commercial Licenses</strong></p> <p>Commercial licenses for the RAVDESS can be purchased. For more information, please visit our <a href="https://psychlabs.torontomu.ca/smartlab/ravdess-commercial-licensing/">license page of fees</a>, or contact us at <a href="mailto:ravdess@gmail.com?subject=RAVDESS%20Commercial%20License">ravdess@gmail.com</a>.</p> <p><strong>Contact Information</strong></p> <p>If you would like further information about the RAVDESS, to purchase a commercial license, or if you experience any issues downloading files, please contact us at <a href="mailto:ravdess@gmail.com?subject=RAVDESS%20feedback%20from%20Zenodo">ravdess@gmail.com</a>.</p> <p><strong>Example Videos</strong></p> <p>Watch a sample of the RAVDESS <a href="https://www.youtube.com/watch?v=Y7OQoNEu3dY">speech</a> and <a href="https://www.youtube.com/watch?v=XQkmH4oYZkg">song</a> videos.</p> <p><strong>Emotion Classification Users</strong></p> <p>If you're interested in using machine learning to classify emotional expressions with the RAVDESS, please see our new RAVDESS Facial Landmark Tracking data set [<a href="../record/3255102">Zenodo project page</a>].</p> <p><strong>Construction and Validation</strong></p> <p>Full details on the construction and perceptual validation of the RAVDESS are described in our PLoS ONE paper - <a href="https://doi.org/10.1371/journal.pone.0196391">https://doi.org/10.1371/journal.pone.0196391</a>.</p> <p>The RAVDESS contains 7356 files. Each file was rated 10 times on emotional validity, intensity, and genuineness. Ratings were provided by 247 individuals who were characteristic of untrained adult research participants from North America. A further set of 72 participants provided test-retest data. High levels of emotional validity, interrater reliability, and test-retest intrarater reliability were reported. Validation data is open-access, and can be downloaded along with our paper from <a href="http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0196391">PLoS ONE</a>.</p> <p><strong>Contents</strong></p> <p><em>Audio-only files</em></p> <p>Audio-only files of all actors (01-24) are available as two separate zip files (~200 MB each):</p> <ul> <li>Speech file (Audio_Speech_Actors_01-24.zip, 215 MB) contains 1440 files: 60 trials per actor x 24 actors = 1440. </li> <li>Song file (Audio_Song_Actors_01-24.zip, 198 MB) contains 1012 files: 44 trials per actor x 23 actors = 1012.</li> </ul> <p><em>Audio-Visual and Video-only files</em></p> <p>Video files are provided as separate zip downloads for each actor (01-24, ~500 MB each), and are split into separate speech and song downloads:</p> <ul> <li>Speech files (Video_Speech_Actor_01.zip to Video_Speech_Actor_24.zip) collectively contains 2880 files: 60 trials per actor x 2 modalities (AV, VO) x 24 actors = 2880.</li> <li>Song files (Video_Song_Actor_01.zip to Video_Song_Actor_24.zip) collectively contains 2024 files: 44 trials per actor x 2 modalities (AV, VO) x 23 actors = 2024.</li> </ul> <p><em>File Summary</em></p> <p>In total, the RAVDESS collection includes 7356 files (2880+2024+1440+1012 files).</p> <p><strong>File naming convention</strong></p> <p>Each of the 7356 RAVDESS files has a unique filename. The filename consists of a 7-part numerical identifier (e.g., 02-01-06-01-02-01-12.mp4). These identifiers define the stimulus characteristics: <br><br><em>Filename identifiers </em></p> <ul> <li>Modality (01 = full-AV, 02 = video-only, 03 = audio-only).</li> <li>Vocal channel (01 = speech, 02 = song).</li> <li>Emotion (01 = neutral, 02 = calm, 03 = happy, 04 = sad, 05 = angry, 06 = fearful, 07 = disgust, 08 = surprised).</li> <li>Emotional intensity (01 = normal, 02 = strong). NOTE: There is no strong intensity for the 'neutral' emotion.</li> <li>Statement (01 = "Kids are talking by the door", 02 = "Dogs are sitting by the door").</li> <li>Repetition (01 = 1st repetition, 02 = 2nd repetition).</li> <li>Actor (01 to 24. Odd numbered actors are male, even numbered actors are female).</li> </ul> <p><br><em>Filename example: 02-01-06-01-02-01-12.mp4 </em></p> <ol> <li>Video-only (02)</li> <li>Speech (01)</li> <li>Fearful (06)</li> <li>Normal intensity (01)</li> <li>Statement "dogs" (02)</li> <li>1st Repetition (01)</li> <li>12th Actor (12)</li> <li>Female, as the actor ID number is even.</li> </ol> <p><strong>License information</strong></p> <p>The RAVDESS is released under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">CC BY-NC-SA 4.0</a> </p> <p>Commercial licenses for the RAVDESS can also be purchased. For more information, please visit our <a href="https://smartlaboratory.org/ravdess/commercial-licensing/">license fee page</a>, or contact us at <a href="mailto:ravdess@gmail.com?subject=RAVDESS%20Commercial%20License">ravdess@gmail.com.</a></p> <p><strong>Related Data sets</strong></p> <ul> <li>RAVDESS Facial Landmark Tracking data set [<a href="../record/3255102">Zenodo project page</a>].</li> </ul>
The Grid Audio-Visual Lombard Speech Corpus
<p>Lombard Grid is a bi-view audiovisual Lombard speech corpus which can be used to support joint computational/behavioral studies in speech perception. The corpus includes 54 talkers, with 100 utterances per talker (50 Lombard and 50 plain utterances). This dataset follows the same sentence format as the audiovisual <a href="https://asa.scitation.org/doi/10.1121/1.5042758">Grid corpus</a>, and can thus be considered as an extension of that corpus. The sentence sets used in the Lombard Grid corpus are unique, however, and have not been utilized by the Grid corpus.</p> <p>It offers two synchronised views of the talkers (front and side) to facilitate analysis of speech from different angles. A bespoke head-mounted camera system was used to collect both front and profile views of the talkers.</p> <p><strong>Statistics</strong>: 54 talkers: 30 female talkers and 24 male talkers; 5,400 (audio, front video and side video) utterances (16,200 files in total): 50% Lombard utterances, 50% plain reference utterances.</p> <p>The dataset is described in detail in the paper,</p> <p>Najwa Alghamdi, Steve Maddock, Ricard Marxer, Jon Barker and Guy J. Brown,, "A corpus of audio-visual Lombard speech with frontal and profile views", The Journal of the Acoustical Society of America 143, El523 (2018) </p> <p>The paper is available online at <a href="http://eprints.whiterose.ac.uk/131924/">White Rose Online Research</a>.</p> <p>------------------------------------------------------------------------------------</p> <p><strong>Notes on Filenaming</strong></p> <p><strong>Filename format</strong></p> <p>SPKR_COND_UTTERANCE.wav|.mov - e.g., s8_p_sbbi9p.wav</p> <p>*SPKR = s1 to s55</p> <p>*COND = l or p, where l=> Lombard, p=> plain (i.e. non-Lombard)</p> <p>*UTTERANCE = 6-character Grid utterance code, e.g. 'pgag6a' which means 'place green at g 6 again'</p> <p><strong>Metadata format</strong></p> <p>*SPKR = s1 to s55</p> <p>*SESSION = 1 or 2</p> <p>*INDEX = 1 to 10 for ordering of the recording blocks</p> <p>*SUBINDEX = 1 to 10 for ordering of utterance in a 10-utterance block.</p> <p>*COND = l or r, where l=> Lombard, p=> plain (i.e. non-Lombard)</p> <p>*UTTERANCE = 6-character Grid utterance code, e.g. 'pgag6a' which means 'place green at g 6 again'</p> <p>If a sentence is spoken incorrectly then the filename will be</p> <p>_WRONG.wav e.g. s8_2_38_8_r_lrwizp_WRONG_lrbizp.wav</p> <p>*TRANS = the Grid utterance code for what was actually said.</p>
Materials used in "Measuring audio-visual speech intelligibility under dynamic listening conditions using virtual reality"
<p>The materials in this record are the audio and video files, together with various configuration files, used by the "SEAT" software in the study described in</p> <p>Moore, Green, Brookes & Naylor (2022) "Measuring audio-visual speech intelligibility under dynamic listening conditions using virtual reality"</p> <p>They are shared in this form so that the experiment may be reproduced. For any other use please contact the authors to obtain the original database(s) from which these materials are derived.</p> <p>The materials were created to be compatible with v0.3 of SEAT, which is available from <a href="https://github.com/ImperialCollegeLondon/sap-elospheres-audiovisual-test/releases/tag/v0.3">GitHub</a>. Note that the materials must be placed at <code>C:\seat_experiments\cafe_AV.</code></p>
The Grid Audio-Visual Speech Corpus
<p>The Grid Corpus is a large multitalker audiovisual sentence corpus designed to support joint computational-behavioral studies in speech perception. In brief, the corpus consists of high-quality audio and video (facial) recordings of 1000 sentences spoken by each of 34 talkers (18 male, 16 female), for a total of 34000 sentences. Sentences are of the form "put red at G9 now".</p> <p>audio_25k.zip contains the wav format utterances at a 25 kHz sampling rate in a separate directory per talker<br> alignments.zip provides word-level time alignments, again separated by talker<br> s1.zip, s2.zip etc contain .jpg videos for each talker [note that due to an oversight, no video for talker t21 is available]</p> <p>The Grid Corpus is described in detail in the paper jasagrid.pdf included in the dataset.</p>
Cross-modal matching of audio-visual German and French fluent speech in infancy
<p>The present study examined when and how the ability to cross-modally match audio-visual fluent speech develops in 4.5-, 6- and 12-month-old German-learning infants. In Experiment 1, 4.5- and 6-month-old infants’ audio-visual matching ability of native (German) and non-native (French) fluent speech was assessed by presenting auditory and visual speech information sequentially, that is, in the absence of temporal synchrony cues. The results showed that 4.5-month-old infants were capable of matching native as well as non-native audio and visual speech stimuli, whereas 6-month-olds perceived the audio-visual correspondence of native language stimuli only. This suggests that intersensory matching narrows for fluent speech between 4.5 and 6 months of age. In Experiment 2, auditory and visual speech information was presented simultaneously, therefore, providing temporal synchrony cues. Here, 6-month-olds were found to match native as well as non-native speech indicating facilitation of temporal synchrony cues on the intersensory perception of non-native fluent speech. Intriguingly, despite the fact that audio and visual stimuli cohered temporally, 12-month-olds matched the non-native language only. Results were discussed with regard to multisensory perceptual narrowing during the first year of life.</p>
Data from: Contributions of local speech encoding and functional connectivity to audio-visual speech perception
Seeing a speaker's face enhances speech intelligibility in adverse environments. We investigated the underlying network mechanisms by quantifying local speech representations and directed connectivity in MEG data obtained while human participants listened to speech of varying acoustic SNR and visual context. During high acoustic SNR speech encoding by temporally entrained brain activity was strong in temporal and inferior frontal cortex, while during low SNR strong entrainment emerged in premotor and superior frontal cortex. These changes in local encoding were accompanied by changes in directed connectivity along the ventral stream and the auditory-premotor axis. Importantly, the behavioral benefit arising from seeing the speaker's face was not predicted by changes in local encoding but rather by enhanced functional connectivity between temporal and inferior frontal cortex. Our results demonstrate a role of auditory-frontal interactions in visual speech representations and suggest that functional connectivity along the ventral pathway facilitates speech comprehension in multisensory environments.
Data from: Contributions of local speech encoding and functional connectivity to audio-visual speech perception
Open the record for dataset details and reuse information.
Cross-modal matching of audio-visual German and French fluent speech in infancy
<p>The present study examined when and how the ability to cross-modally match audio-visual fluent speech develops in 4.5-, 6- and 12-month-old German-learning infants. In Experiment 1, 4.5- and 6-month-old infants’ audio-visual matching ability of native (German) and non-native (French) fluent speech was assessed by presenting auditory and visual speech information sequentially, that is, in the absence of temporal synchrony cues. The results showed that 4.5-month-old infants were capable of matching native as well as non-native audio and visual speech stimuli, whereas 6-month-olds perceived the audio-visual correspondence of native language stimuli only. This suggests that intersensory matching narrows for fluent speech between 4.5 and 6 months of age. In Experiment 2, auditory and visual speech information was presented simultaneously, therefore, providing temporal synchrony cues. Here, 6-month-olds were found to match native as well as non-native speech indicating facilitation of temporal synchrony cues on the intersensory perception of non-native fluent speech. Intriguingly, despite the fact that audio and visual stimuli cohered temporally, 12-month-olds matched the non-native language only. Results were discussed with regard to multisensory perceptual narrowing during the first year of life.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.