RUSSE'2018: Human-Annotated Sense-Disambiguated Word Contexts for Russian

<p>This dataset contains human-annotated sense identifiers for 2562 contexts of 20 words used in the <a href="https://russe.nlpub.org/2018/wsi/">RUSSE&#39;2018</a> shared task on Word Sense Induction and Disambiguation for the Russian language; part of the&nbsp;<em>bts-rnc</em>&nbsp;evaluation dataset. These sense identifiers are disambiguated as according to the sense inventory of the <a href="http://gramota.ru/slovari/info/bts/">Large Explanatory Dictionary of Russian</a>.</p> <p>The annotation is done on December 1, 2017, on the&nbsp;<a href="https://tolokanyandex.com/">Yandex.Toloka</a>&nbsp;crowdsourcing platform. In particular, 80 pre-annotated contexts are used for&nbsp;training the human annotators, 2562 contexts are annotated by humans such that each&nbsp;context was annotated by 9 different annotators. The annotation reliability&nbsp;is indicated by a high value of Krippendorff&#39;s&nbsp;&alpha; = 0.83. After the annotation, every context was additionally inspected (&ldquo;curated&rdquo;) by the organizers of the shared task.</p> <p>The following words are represented:&nbsp;<em>акция</em> (action / stock), <em>байка</em> (yarn / tale), <em>гвоздика</em> (carnation / nail), <em>гипербола</em> (hyperbole), <em>град</em> (avalanche), <em>гусеница</em> (grub), <em>домино</em> (domino), <em>кабачок</em> (marrow / pub), <em>капот</em> (hood), <em>карьер</em> (mine / career), <em>кок</em> (cook), <em>крона</em> (top / crown), <em>круп</em> (croup), <em>мандарин</em> (mandarine), <em>рок</em> (fate / rock), <em>слог</em> (syllable), <em>стопка</em> (glass, stack), <em>таз</em> (bowl), <em>такса</em> (rate / badger-dog), <em>шах</em> (shah / check).</p> <p>The following files are included in this dataset:</p> <ul> <li>Toloka assignments (training:&nbsp;<em>tasks-train.tsv</em>, annotation: <em>tasks-test.tsv</em>)</li> <li>Toloka output (non-aggregated: <em>assignments_01-12-2017.tsv.xz</em>, aggregated:&nbsp;<em>aggregated_results_pool_1036853__2017_12_01.tsv</em>)</li> <li>annotator agreement report (<em>agreement.txt</em>)</li> <li>curated report (<em>report-curated.tsv.xz</em> and a supplementary file&nbsp;<em>tasks-eval.tsv.xz</em>)</li> <li>the final aggregated dataset (<em>bts-rnc-crowd.tsv</em>)</li> </ul> <p>The <em>bts-rnc-crowd.tsv</em>&nbsp;file has the following format: <em>id</em>, <em>lemma</em>, <em>sense_id</em>, <em>left</em> hand side context, <em>word</em> form, <em>right</em> hand side context, list of&nbsp;<em>senses</em>. The encoding is UTF-8 and the line breaks are LF (UNIX).</p>