SPEECH-COCO

<p><strong>SpeechCoco</strong></p> <p><em>Introduction</em></p> <p>Our corpus is an extension of the MS COCO image recognition and captioning dataset. MS COCO comprises images paired with a set of five captions. Yet, it does not include any speech. Therefore, we used&nbsp;<a href="https://www.voxygen.fr/">Voxygen&#39;s text-to-speech system</a>&nbsp;to synthesise&nbsp;the available captions.</p> <p>The addition of speech as a new modality enables MSCOCO to be used for researches in the field of language acquisition, unsupervised term discovery, keyword spotting, or semantic embedding using speech and vision.</p> <p>Our corpus is licensed under a&nbsp;<a href="https://creativecommons.org/licenses/by/4.0/legalcode">Creative Commons Attribution 4.0 License</a>.</p> <p><em>Data Set</em></p> <ul> <li> <p>This corpus contains&nbsp;<strong>616,767</strong>&nbsp;spoken captions from MSCOCO&#39;s val2014 and train2014 subsets (respectively 414,113 for train2014 and 202,654 for val2014).</p> </li> <li> <p>We used 8 different voices. 4 of them have a British accent (Paul, Bronwen, Judith, and Elizabeth) and the 4 others have an American accent (Phil, Bruce, Amanda, Jenny).</p> </li> <li> <p>In order to make the captions sound more natural, we used SOX&nbsp;<em>tempo</em>&nbsp;command, enabling us to change the speed without changing the pitch. 1/3 of the captions are 10% slower than the original pace, 1/3 are 10% faster. The last third of the captions was kept untouched.</p> </li> <li> <p>We also modified approximately 30% of the original captions and added&nbsp;<strong>disfluencies</strong>&nbsp;such as &quot;um&quot;, &quot;uh&quot;, &quot;er&quot; so that the captions would sound more natural.</p> </li> <li> <p>Each WAV file is paired with a JSON file containing various information: timecode of each word in the caption, name of the speaker, name of the WAV file, etc. The JSON files have the following data structure:</p> </li> </ul> <pre><code class="language-json">{ "duration": float, "speaker": string, "synthesisedCaption": string, "timecode": list, "speed": float, "wavFilename": string, "captionID": int, "imgID": int, "disfluency": list }</code></pre> <ul> <li> <p>On average, each caption comprises 10.79 tokens, disfluencies included. The WAV files are on average 3.52 seconds long.</p> </li> </ul> <p><em>Repository</em></p> <p>The repository is organized as follows:</p> <ul> <li> <p>CORPUS-MSCOCO (~75GB once decompressed)</p> <blockquote> <ul> <li> <p><strong>train2014/</strong>&nbsp;:&nbsp;folder contains 413,915 captions</p> <ul> <li> <p>json/</p> </li> <li> <p>wav/</p> </li> <li> <p>translations/</p> <ul> <li> <p>train_en_ja.txt</p> </li> <li> <p>train_translate.sqlite3</p> </li> </ul> </li> <li> <p>train_2014.sqlite3</p> </li> </ul> </li> <li> <p><strong>val2014/</strong>&nbsp;:&nbsp;folder contains 202,520 captions</p> <ul> <li> <p>json/</p> </li> <li> <p>wav/</p> </li> <li> <p>translations/</p> <ul> <li> <p>train_en_ja.txt</p> </li> <li> <p>train_translate.sqlite3</p> </li> </ul> </li> <li> <p>val_2014.sqlite3</p> </li> </ul> </li> <li> <p><strong>speechcoco_API/</strong></p> <ul> <li> <p>speechcoco/</p> <ul> <li> <p>__init__.py</p> </li> <li> <p>speechcoco.py</p> </li> </ul> </li> <li> <p>setup.py</p> </li> </ul> </li> </ul> </blockquote> </li> </ul> <p><em>Filenames</em></p> <p><strong>.wav</strong>&nbsp;files contain the spoken version of a caption</p> <p><strong>.json</strong>&nbsp;files contain all the metadata of a given WAV file</p> <p><strong>.sqlite3</strong>&nbsp;files are SQLite databases containing all the information contained in the JSON files</p> <p>We adopted the following naming convention for both the WAV and JSON files:</p> <p><em>imageID_captionID_Speaker_DisfluencyPosition_Speed[.wav/.json]</em></p> <p><em>Script</em></p> <p>We created a script called&nbsp;<strong>speechcoco.py</strong>&nbsp;in order to handle the metadata and allow the user to easily find captions according to specific filters. The script uses the *.db files.</p> <p>Features:</p> <ul> <li> <p><strong>Aggregate all the information in the JSON files into a single SQLite database</strong></p> </li> <li> <p><strong>Find captions according to specific filters (name, gender and nationality of the speaker, disfluency position, speed, duration, and words in the caption).</strong>&nbsp;<em>The script automatically builds the SQLite query. The user can also provide his own SQLite query.</em></p> </li> </ul> <p><em>The following Python code returns all the captions spoken by a male with an American accent for which the speed was slowed&nbsp;down by 10% and that contain &quot;keys&quot; at any position</em></p> <pre><code class="language-python"># create SpeechCoco object db = SpeechCoco(train_2014.sqlite3, train_translate.sqlite3, verbose=True) # filter captions (returns Caption Objects) captions = db.filterCaptions(gender="Male", nationality="US", speed=0.9, text='%keys%') for caption in captions: print('\n{}\t{}\t{}\t{}\t{}\t{}\t\t{}'.format(caption.imageID, caption.captionID, caption.speaker.name, caption.speaker.nationality, caption.speed, caption.filename, caption.text))</code></pre> <pre><code>... 298817 26763 Phil 0.9 298817_26763_Phil_None_0-9.wav A group of turkeys with bushes in the background. 108505 147972 Phil 0.9 108505_147972_Phil_Middle_0-9.wav Person using a, um, slider cell phone with blue backlit keys. 258289 154380 Bruce 0.9 258289_154380_Bruce_None_0-9.wav Some donkeys and sheep are in their green pens . 545312 201303 Phil 0.9 545312_201303_Phil_None_0-9.wav A man walking next to a couple of donkeys. ...</code></pre> <ul> <li> <p><strong>Find all the captions belonging to a specific image</strong></p> </li> </ul> <pre><code class="language-python">captions = db.getImgCaptions(298817) for caption in captions: print('\n{}'.format(caption.text))</code></pre> <pre><code>Birds wondering through grassy ground next to bushes. A flock of turkeys are making their way up a hill. Um, ah. Two wild turkeys in a field walking around. Four wild turkeys and some bushes trees and weeds. A group of turkeys with bushes in the background.</code></pre> <ul> <li> <p><strong>Parse the timecodes and have them structured</strong></p> </li> </ul> <p><strong>input</strong>:</p> <pre><code>... [1926.3068, "SYL", ""], [1926.3068, "SEPR", " "], [1926.3068, "WORD", "white"], [1926.3068, "PHO", "w"], [2050.7955, "PHO", "ai"], [2144.6591, "PHO", "t"], [2179.3182, "SYL", ""], [2179.3182, "SEPR", " "] ...</code></pre> <p><strong>output</strong>:</p> <pre><code class="language-python">print(caption.timecode.parse())</code></pre> <pre><code>... { 'begin': 1926.3068, 'end': 2179.3182, 'syllable': [{'begin': 1926.3068, 'end': 2179.3182, 'phoneme': [{'begin': 1926.3068, 'end': 2050.7955, 'value': 'w'}, {'begin': 2050.7955, 'end': 2144.6591, 'value': 'ai'}, {'begin': 2144.6591, 'end': 2179.3182, 'value': 't'}], 'value': 'wait'}], 'value': 'white' }, ...</code></pre> <ul> <li> <p><strong>Convert the timecodes to Praat TextGrid files</strong></p> </li> </ul> <pre><code class="language-python">caption.timecode.toTextgrid(outputDir, level=3)</code></pre> <ul> <li> <p><strong>Get the words, syllables and phonemes between</strong>&nbsp;<em>n</em>&nbsp;<strong>seconds/milliseconds</strong></p> </li> </ul> <p><em>The following Python code returns all the words between 0.2 and 0.6 seconds for which at least 50% of the word&#39;s total length is within the specified interval</em></p> <pre><code class="language-python">pprint(caption.getWords(0.20, 0.60, seconds=True, level=1, olapthr=50))</code></pre> <pre><code>... 404537 827239 Bruce US 0.9 404537_827239_Bruce_None_0-9.wav Eyeglasses, a cellphone, some keys and other pocket items are all laid out on the cloth. . [ { 'begin': 0.0, 'end': 0.7202778, 'overlapPercentage': 55.53412863758955, 'word': 'eyeglasses' } ] ...</code></pre> <ul> <li> <p><strong>Get the translations of the selected captions</strong></p> </li> </ul> <p><em>As for now, only japanese translations are available. We also used</em>&nbsp;<a href="http://www.phontron.com/kytea/">Kytea</a>&nbsp;<em>to tokenize and tag the captions translated with Google Translate</em></p> <pre><code class="language-python">captions = db.getImgCaptions(298817) for caption in captions: print('\n{}'.format(caption.text)) # Get translations and POS print('\tja_google: {}'.format(db.getTranslation(caption.captionID, "ja_google"))) print('\t\tja_google_tokens: {}'.format(db.getTokens(caption.captionID, "ja_google"))) print('\t\tja_google_pos: {}'.format(db.getPOS(caption.captionID, "ja_google"))) print('\tja_excite: {}'.format(db.getTranslation(caption.captionID, "ja_excite")))</code></pre> <pre><code> Birds wondering through grassy ground next to bushes. ja_google: 鳥は茂みの下に茂った地面を抱えています。 ja_google_tokens: 鳥 は 茂み の 下 に 茂 っ た 地面 を 抱え て い ま す 。 ja_google_pos: 鳥/名詞/とり は/助詞/は 茂み/名詞/しげみ の/助詞/の 下/名詞/した に/助詞/に 茂/動詞/しげ っ/語尾/っ た/助動詞/た 地面/名詞/じめん を/助詞/を 抱え/動詞/かかえ て/助詞/て い/動詞/い ま/助動詞/ま す/語尾/す 。/補助記号/。 ja_excite: 低木と隣接した草深いグラウンドを通って疑う鳥。 A flock of turkeys are making their way up a hill. ja_google: 七面鳥の群れが丘を上っています。 ja_google_tokens: 七 面 鳥 の 群れ が 丘 を 上 っ て い ま す 。 ja_google_pos: 七/名詞/なな 面/名詞/めん 鳥/名詞/とり の/助詞/の 群れ/名詞/むれ が/助詞/が 丘/名詞/おか を/助詞/を 上/動詞/のぼ っ/語尾/っ て/助詞/て い/動詞/い ま/助動詞/ま す/語尾/す 。/補助記号/。 ja_excite: 七面鳥の群れは丘の上で進んでいる。 Um, ah. Two wild turkeys in a field walking around. ja_google: 野生のシチメンチョウ、野生の七面鳥 ja_google_tokens: 野生 の シチメンチョウ 、 野生 の 七 面 鳥 ja_google_pos: 野生/名詞/やせい の/助詞/の シチメンチョウ/名詞/しちめんちょう 、/補助記号/、 野生/名詞/やせい の/助詞/の 七/名詞/なな 面/名詞/めん 鳥/名詞/ちょう ja_excite: まわりで移動しているフィールドの2羽の野生の七面鳥 Four wild turkeys and some bushes trees and weeds. ja_google: 4本の野生のシチメンチョウといくつかの茂みの木と雑草 ja_google_tokens: 4 本 の 野生 の シチメンチョウ と いく つ か の 茂み の 木 と 雑草 ja_google_pos: 4/名詞/4 本/接尾辞/ほん の/助詞/の 野生/名詞/やせい の/助詞/の シチメンチョウ/名詞/しちめんちょう と/助詞/と いく/名詞/いく つ/接尾辞/つ か/助詞/か の/助詞/の 茂み/名詞/しげみ の/助詞/の 木/名詞/き と/助詞/と 雑草/名詞/ざっそう ja_excite: 4羽の野生の七面鳥およびいくつかの低木木と雑草 A group of turkeys with bushes in the background. ja_google: 背景に茂みを持つ七面鳥の群 ja_google_tokens: 背景 に 茂み を 持 つ 七 面 鳥 の 群 ja_google_pos: 背景/名詞/はいけい に/助詞/に 茂み/名詞/しげみ を/助詞/を 持/動詞/も つ/語尾/つ 七/名詞/なな 面/名詞/めん 鳥/名詞/ちょう の/助詞/の 群/名詞/むれ ja_excite: 背景の低木を持つ七面鳥のグループ</code></pre> <p>&nbsp;</p>