A hybrid approach to the small unannotated corpus-based language comparison and its application to the Old East Slavic charters - Supplementary material 5 (Corpus-based language distance measurement results)

<h1>General description</h1> <p>These are the results of the experiments with the use of <a href="https://doi.org/10.5281/zenodo.11395683" target="_blank" rel="noopener">corpus-based language distance measurement package</a> on the material of <a href="https://doi.org/10.5281/zenodo.14057668" target="_blank" rel="noopener">Old East Slavic</a>, <a href="https://doi.org/10.5281/zenodo.14148179" target="_blank" rel="noopener">modern East Slavic</a>, and <a href="https://doi.org/10.5281/zenodo.14148561">modern standard Slavic</a> lects. There are 40 possible experiments for each data set, divided by the usage of:</p> <ul> <li>topic antimodelling heuristic,</li> <li>Soerensen-Dice coefficient-based normalisation,</li> <li>the presence of hybridisation of frequency-based metric for coinciding units and combined frequency-based metric and string similarity measure for non-coinciding units,</li> <li>hybridisation type,</li> <li>exact type of string similarity measure used for combination,</li> <li>alphabet entropy-based normalisation for vector-based string similarity measures.</li> </ul> <p>In addition, modern standard Slavic dataset undergoes experiments 4 times that differ by the share of its size, used for measurements (0.1, 0.3, 0.6 and 1).</p> <p>For further information on each of the experiment parameters, refer to the documentation of the package.</p> <h1>Data set structure</h1> <h2>Executive summary</h2> <p>Data set consists of 240 folders that represent information on the experiments and 1&nbsp;<code>.csv</code>-file that aggregates the resulting values into a single table.</p> <p>Folders with indices 1-20 and 121-140 contain experiments with 0.1 share of the modern standard Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <p>Folders with indices 21-40 and 141-160 contain experiments with 0.3 share of the modern standard Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <p>Folders with indices&nbsp; 41-60 and 161-180 contain experiments with 0.6 share of the modern standard Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <p>Folders with indices&nbsp; 61-80 and 181-200 contain experiments with the full share of the modern standard Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <p>Folders with indices&nbsp; 81-100 and 201-220 contain experiments with the full share of the modern East Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <p>Folders with indices&nbsp; 101-120 and 221-240 contain experiments with the full share of the Old East Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <h2>.csv-file</h2> <p>Named <code>aggregated_results.csv</code>, lies in the root of the dataset. Separator is <strong>comma</strong> (<strong>,</strong>). Contains&nbsp;<strong>13 columns</strong> and&nbsp;<strong>241 row</strong>. The first row is&nbsp;<strong>header</strong>, the other 240 rows contain description for each conducted experiment and its resulting values, according to the columns. The columns are the following (in <strong>rtl</strong> order):</p> <ul> <li><strong>X.</strong> (<em>int</em>) - experiment ID; column is used as index.</li> <li><strong>Material&nbsp;</strong>(<em>string</em>) - the data set used for language distance measurement. The possible values are: <ul> <li>Slavic standard - Croatian, Slovenian, Slovak standard lects.</li> <li>Modern East Slavic - Northern Russian lect Megra, Central Russian lect Belogornoje, and Northern Belarusian lect Zialionka.</li> <li>Old East Slavic - Novgorod, Polack and Smolensk parts of the Old East Slavic continuum.</li> </ul> </li> <li><strong>Gensim&nbsp;</strong>(<em>int</em>) - the binary numeric indicator (0 or 1) of using the heuristic of&nbsp;<em>topic antimodelling</em>, namely, cleaning the words that were defined as a topic words by <em>gensim</em> Latent Dirichlet Association implementation (Rehurek &amp; Sojka, 2010). The intention of using this heuristic is to remove the tokens that are characteristic for the genre of the texts presented in the corpus for the sake of increasing the presence of the tokens that are characteristic of the lects themselves.</li> <li><strong>Split</strong> (<em>float</em>) - the used share of the data set (from 0 to 1); required to check the influence of the data set size on the metric efficiency.</li> <li><strong>Hybridisation</strong> (<em>string</em>) - the indicator of implementation of the hybridisation between the frequency-based metric between the 3-shingles (character 3-grams) that coincide for the compared lect pair, and the combination of frequency-based metric and string similarity measure between the 3-shingles that do not coincide for the compared lect pair. The possible values are: <ul> <li>TRUE: the experiment utilises hybridisation</li> <li>FALSE: the experiment does not utilise hybridisation.</li> </ul> </li> <li><strong>Hybridisation_type</strong> (<em>string</em>) - the indicator of how the frequency metric between coinciding 3-shingles and the combined metric between non-coinciding 3-shingles undergo the hybridisation process. The values are:<br> <ul> <li>JOINED - the approach is to multiply the means of the two.</li> <li>ARRAY -&nbsp;the approach is to join all the values into a single list, and then to score the mean.</li> <li>NOT_USED - experiment does not employ hybridisation. (<strong>Hybridisation&nbsp;</strong>is&nbsp;FALSE).</li> </ul> </li> <li><strong>Soerensen_normalisation</strong> (<em>string</em>) - the indicator of whether the frequency-based metric value undergoes normalisation with the division by Soerensen-Dice coefficient (a measure of number of coincidences between two lists) (Soerensen, 1948), in order to compensate the skewing between the coinciding and non-coinciding 3-shingles of the lects. The values are: <ul> <li>NOT_USED - <strong>Hybridisation_type&nbsp;</strong>is ARRAY, so there are no values to use Soerensen-Dice coefficient on.</li> <li>TRUE - the frequency-based metric undergoes division by the Soerensen-Dice coefficient.</li> <li>FALSE -&nbsp;the algorithm does not apply the normalisation by the Soerensen-Dice coefficient.</li> </ul> </li> <li><strong>Alphabet_normalisation</strong>&nbsp;(<em>string</em>) - indicator of whether the algorithm applies normalisation with the alphabet entropy&nbsp; (Shannon, 1948), the measure of differences in the skewings of symbols distribution in the texts, between the given lects. The values are:<br> <ul> <li>NOT_USED&nbsp; - a heuristic may not be implemented; present either in the cases, when <strong>Hybridisation&nbsp;</strong>is&nbsp;FALSE, or when the next parameter, <strong>Auxiliary_metrics</strong> is not&nbsp;VDND or&nbsp;VWJDND.</li> <li>TRUE<strong>&nbsp;</strong>- the experiment employs the heuristic.</li> <li>FALSE - the experiment does not employ the heuristic.</li> </ul> </li> <li><strong>Auxiliary_metrics&nbsp;</strong>(<em>string</em>) - the string similarity measure, used for the combination with the frequency-based metric for non-coinciding 3-shingles between analysed lects. There are five possible values: <ul> <li>LDND (Levenshtein distance normalised between analysed 3-shingles) (Holman et al., 2008).</li> <li>WJWDND (weighted Jaro-Winkler distance normalised between analysed 3-shingles) (Gueddah et al., 2015).</li> <li>VDND&nbsp;(Euclidean distance between the sums of symbol vector values between 3-shingles).</li> <li>VWJDND&nbsp;(VDND multiplied by scoring Jaro (Jaro, 1989) distance between analysed 3-shingles).</li> </ul> </li> <li><strong>Outgroup.identification</strong> (<em>string</em>) - the indicator of whether the outgroup detected in the given experiment coincides with the lect that preliminary manual classification supposes to be the outgroup. There are two possible values:<br> <ul> <li>CORRECT - the detected outgroup coincides with the supposed one.</li> <li>INCORRECT - the detected outgroup does not coincide with the supposed one.</li> </ul> </li> <li><strong>Outer.distance.split&nbsp;</strong>(<em>float</em>) - the length of the outgroup branch.</li> <li><strong>Inner.distance.split</strong> (<em>float</em>) - the distance between the split between the outgroup and the ingroup, and the split between the two ingroup lects.</li> <li><strong>Split.difference</strong> (<em>float</em>) - the division of <strong>Outer.distance.split</strong> by <strong>Inner.distance.split</strong>.</li> </ul> <h2>Folders</h2> <p>Each folder contains 6 files, each named according to the used experiment setup:</p> <ul> <li>3&nbsp;<code>.csv</code>-files that contain unit-by-unit comparison between each pair of the analysed lects. Each&nbsp;<code>.csv</code>-file is&nbsp;<strong>semi-colon</strong>-separated, and has&nbsp;<strong>4&nbsp;</strong>columns, <strong>header row</strong>, and rows that describes each unit-to-unit comparison. The columns contain the following information (in <strong>rtl</strong> order):<br> <ul> <li><code>[Name of the first compared lect]</code> : unit (character 3-shingle, or just 3-shingle) of the [name of the first compared lect] that undergoes comparison with units of the [name of the second compared lect]; datatype: string.</li> <li><code>[Name of the second compared lect] </code>: unit (character 3-shingle, or just 3-shingle) of the [name of the second compared lect] that undergoes comparison with units of the [name of the first compared lect]; if units coincide, contains value <code>id.</code>; datatype: string.</li> <li><code>[Experiment setup]</code>: name of the metric, a combination of the [experiment setup](concatenated through <code><em>-</em></code> parameters) and its exact part, which compares the two units; datatype: string. The possible values are: <ul> <li><code>[experiment setup] - DistRank</code> - the frequency-based metric that compares identical units</li> <li><code>[experiment setup] - hybrid</code> - the string similarity measure for non-identical units, combined with the frequency-based metric&nbsp;</li> </ul> </li> <li><code>Distance</code>: value of the metric; datatype: float</li> </ul> </li> <li><code>.info</code>-file that contains data on branch lengths along with coincidence/non-coincidence of the detected outgroup with the manually defined one. The file is a&nbsp;<strong>tabular-separated plain text</strong> that always contains three values: coincidence (CORRECT)/non-coincidence (INCORRECT) of the yielded classification with the supposed one; outer distance split (the length of the outgroup branch; datatype: float) and inner distance split (the length of the ingroup branch before split of its lects; datatype: float).</li> <li><code>.newick</code> -file that contains the result of an experiment, the phylogenetic tree built by UPGMA classifier. One can read it with <a href="https://cran.r-project.org/web/packages/TreeTools/vignettes/load-trees.html">ape::read.tree</a> (R), or <a href="https://biopython.org/wiki/Phylo">Phylo.read</a> (Python).</li> <li><code>.png</code> -file that contains the phylogenetic tree visualisation.</li> </ul> <p>&nbsp;</p> <h1>How-to</h1> <p>For the analysis of the results, download and unpack the archive, and further&nbsp;refer to the <a href="https://doi.org/10.5281/zenodo.14169792" target="_blank" rel="noopener">companion R notebook</a>.&nbsp;</p>