Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
27
datasets available to search
ShareScore release 0.9.0
Dataset results
27 results for “language comparison”
EDICTOR 3: Interactive Tool for Computer-Assisted Language Comparison
This software offers the most recent and mostly stable version of the EDICTOR tool, also available for direct usage from <a href="https://edictor.org">edictor.org/</a>.
CLDF dataset derived from Lai and List's "Comparison of Rgyalrongic Languages" from 2023
<p>Cite the source of the dataset as:</p> <blockquote> <p>Lai, Yunfan and List, Johann-Mattis (2023): Lexical Data for the Historical Comparison of Rgyalrongic Languages [Dataset, Version 1.0]. Leipzig: Max Planck Institute for Evolutionary Anthropology.</p> </blockquote>
A hybrid approach to the small unannotated corpus-based language comparison and its application to the Old East Slavic charters - Supplementary material 1 (Old East Slavic)
<h1>Old East Slavic charters (XII - XIV century)</h1> <h2>General description</h2> <p>A set of nine historical Old East Slavic legal texts from Smolensk, Polack and Novgorod from the end of the XII century to the first half of the XIV century. The source for Smolensk charters is Avanesov (1963), which contains original texts as well as their initial deciphering (not machine-readable). The source for Polack charters is Horoshkevich (2015), containing trascribed machine-readable texts that required only an additional check and preparation. The source for Novgorod charters is Napierskij (1857), carrying original texts and their initial non-machine-readable deciphering. All the texts underwent an additional preprocessing of reconstructed and contracted parts deletion in order to better represent the actual texts under consideration and exclude as many research biases as it is possible. The last step was a manual tokenisation and the joining of each text into a single string.</p> <p>The data statement is available among the downloadable files.</p> <h2>How-to</h2> <p>This section contains the tutorials that allow to use this data with the intended pipelines.</p> <h3>Corpus-based distance measurement package</h3> <p>The source code for package is available <a href="https://doi.org/10.5281/zenodo.13958502" target="_blank" rel="noopener">here</a>, the manual is available in the <a href="https://github.com/The-One-Who-Speaks-and-Depicts/corpus_distance/blob/dev/README.md" target="_blank" rel="noopener">README</a> section of the repository.</p> <p>To use this dataset for the measurement of distance between Smolensk, Polack and Novgorod lects, and their subsequent clusterisation, following steps should be completed:</p> <ol> <li>Download the <a href="https://github.com/The-One-Who-Speaks-and-Depicts/corpus_distance/blob/dev/example/Corpus_distance_tutorial.ipynb" target="_blank" rel="noopener">Jupyter notebook</a> that streamlines the package use.</li> <li>Download the dataset.</li> <li>Put the dataset into a selected folder on your computer (make sure there are no other files within this folder).</li> <li>Insert the path to the directory into <code>CONTENT_DIR</code> variable in the Jupyter notebook.</li> <li>Run the notebook, adjusting the parameters, if necessary.</li> </ol> <h2> </h2>
A hybrid approach to the small unannotated corpus-based language comparison and its application to the Old East Slavic charters - Supplementary material 3 (Modern standard Slavic lects)
<h1>Modern standard Slavic lects (Croatian, Slovak, Slovenian)</h1> <h2>General description</h2> <p>The dataset consists of texts, written in three modern stanard Slavic lects: Croatian, Slovak, and Slovenian. The texts are parallel in order to compensate for the possible genre influences. The text is John’s Gospel in each of the given languages.</p> <h3>Sources</h3> <p>Croatian original text is from the <a href="https://www.wordproject.org/bibles/cr/index.htm">Ivan Šarić’s translation</a> of New Testament. Slovenian text is from the <a href="https://www.bible.com/bible/2319/JHN.1.SSP">standard Slovenian translation</a> of the New Testament. Slovak text is from the modern <a href="https://svatepismo.sk/evanjelium-podla-jana-1">Catholic translation</a> of the New Testament.</p> <p>The data statement is available among the downloadable files.</p> <h2>How-to</h2> <p>This section contains the tutorials that allow to use this data with the intended pipelines.</p> <h3>Corpus-based distance measurement package</h3> <p>The source code for package is available <a href="https://doi.org/10.5281/zenodo.13958502" target="_blank" rel="noopener">here</a>, the manual is available in the <a href="https://github.com/The-One-Who-Speaks-and-Depicts/corpus_distance/blob/dev/README.md" target="_blank" rel="noopener">README</a> section of the repository.</p> <p>To use this dataset for the measurement of distance between Slovak, Croatian and Slovenian lects, and their subsequent clusterisation, following steps should be completed:</p> <ol> <li>Download the <a href="https://github.com/The-One-Who-Speaks-and-Depicts/corpus_distance/blob/dev/example/Corpus_distance_tutorial.ipynb" target="_blank" rel="noopener">Jupyter notebook</a> that streamlines the package use.</li> <li>Download the dataset.</li> <li>Put the dataset into a selected folder on your computer (make sure there are no other files within this folder).</li> <li>Insert the path to the directory into <code>CONTENT_DIR</code> variable in the Jupyter notebook.</li> <li>Run the notebook, adjusting the parameters, if necessary.</li> </ol> <h2> </h2>
Figure 11. Comparison of MultiNet and MST representations-Representing Mental Spaces and Dynamics of Natural Language Semantics
<p>In the sentence (7) there are two clauses describing two hypothetical situations. These two<br> situations are connected by the relation COND in MultiNet representation (Figure 11).</p>
Figure 10. Comparison of MultiNet and MST representations.-Representing Mental Spaces and Dynamics of Natural Language Semantics
<p>Helbig [18, p. 507] defines the expression (s MCONT c) as “a specification of the<br> informational or mental content c of a mental or informational process s… By default, the second<br> argument c is assumed to be a hypothetical object or situation.”<br> MCONT relation properties are roughly equivalent to those of subject-verb complexes<br> (SVSBs) of MST. MCONT relation can generally be treated as capturing the idea behind what<br> philosophers call propositional attitudes in representational theories of mind [12] or opaque<br> contexts [10] in semantics. For example, consider the sentence in (3) rewritten below as (6) and<br> semantic representation of which is given in Figure 10.</p>
Figure 8. Comparison of MultiNet and MST representations-Representing Mental Spaces and Dynamics of Natural Language Semantics
<p>Figure 8 shows the representation of the<br> first reading of the sentence (5) in MWR. The MultiNet is shown at top of the figure, and the corresponding mental space representation is shown at the bottom side. Blue circles mark<br> hypothetical or nonreal objects and situations and red circles mark real objects and situations.<br> Facticity value for John is real, for unicorn it is nonreal, and for the process of riding, marked with<br> blue broken circle, it is non-real as well.</p>
Figure 9. Comparison of MultiNet and MST representations-Representing Mental Spaces and Dynamics of Natural Language Semantics
<p>Second interpretation of the sentence (5) can be represented with changing John’s Facticity<br> attribute-value from [FACT = real] to [FACT = nonreal] making the whole situation and its<br> elements non-real (Figure 9).</p>
CLDF dataset derived from Wang's "Language Contact and Language Comparison" from 2004
<p>Cite the source of the dataset as:</p> <blockquote> <p>Wang, Feng (2004): Language contact and language comparison. The case of Bai. PhD thesis. Hong Kong: City University of Hong Kong.</p> </blockquote>
The comparison of structured abstracts generated by the ChatGPT language model with the author's original abstracts derived from research papers
<p>The study utilized publications in the field of information science, both in Polish and English, which appeared in the journal <em>Zagadnienia Informacji Naukowej – Studia Informacyjne</em> during the 2022-2023 period. A total of 10 research papers were selected – 5 in Polish (PL1-PL5) and 5 in English (EN1-EN5).</p>
Comparison of Blogshot Formats With Standard Cochrane Plain Language Formats
ClinicalTrials.gov study NCT03542201. IPD Sharing: NO. Countries: 1. Publications: 2.
Comparison of ABLLS-R and Portage Guide in the Development of Receptive Language Skills Autism Spectrum Disorder.
ClinicalTrials.gov study NCT06292520. IPD Sharing: NO. Countries: 1. Publications: 4.
Comparison of Different Frames of Numerical Information in Cochrane Plain Language Summaries
ClinicalTrials.gov study NCT03554252. IPD Sharing: NO. Countries: 1. Publications: 2.
A hybrid approach to the small unannotated corpus-based language comparison and its application to the Old East Slavic charters - Supplementary material 5 (Corpus-based language distance measurement results)
<h1>General description</h1> <p>These are the results of the experiments with the use of <a href="https://doi.org/10.5281/zenodo.11395683" target="_blank" rel="noopener">corpus-based language distance measurement package</a> on the material of <a href="https://doi.org/10.5281/zenodo.14057668" target="_blank" rel="noopener">Old East Slavic</a>, <a href="https://doi.org/10.5281/zenodo.14148179" target="_blank" rel="noopener">modern East Slavic</a>, and <a href="https://doi.org/10.5281/zenodo.14148561">modern standard Slavic</a> lects. There are 40 possible experiments for each data set, divided by the usage of:</p> <ul> <li>topic antimodelling heuristic,</li> <li>Soerensen-Dice coefficient-based normalisation,</li> <li>the presence of hybridisation of frequency-based metric for coinciding units and combined frequency-based metric and string similarity measure for non-coinciding units,</li> <li>hybridisation type,</li> <li>exact type of string similarity measure used for combination,</li> <li>alphabet entropy-based normalisation for vector-based string similarity measures.</li> </ul> <p>In addition, modern standard Slavic dataset undergoes experiments 4 times that differ by the share of its size, used for measurements (0.1, 0.3, 0.6 and 1).</p> <p>For further information on each of the experiment parameters, refer to the documentation of the package.</p> <h1>Data set structure</h1> <h2>Executive summary</h2> <p>Data set consists of 240 folders that represent information on the experiments and 1 <code>.csv</code>-file that aggregates the resulting values into a single table.</p> <p>Folders with indices 1-20 and 121-140 contain experiments with 0.1 share of the modern standard Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <p>Folders with indices 21-40 and 141-160 contain experiments with 0.3 share of the modern standard Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <p>Folders with indices 41-60 and 161-180 contain experiments with 0.6 share of the modern standard Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <p>Folders with indices 61-80 and 181-200 contain experiments with the full share of the modern standard Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <p>Folders with indices 81-100 and 201-220 contain experiments with the full share of the modern East Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <p>Folders with indices 101-120 and 221-240 contain experiments with the full share of the Old East Slavic dataset; the first sequence applies topic antimodelling heuristic, the second sequence does not employ it.</p> <h2>.csv-file</h2> <p>Named <code>aggregated_results.csv</code>, lies in the root of the dataset. Separator is <strong>comma</strong> (<strong>,</strong>). Contains <strong>13 columns</strong> and <strong>241 row</strong>. The first row is <strong>header</strong>, the other 240 rows contain description for each conducted experiment and its resulting values, according to the columns. The columns are the following (in <strong>rtl</strong> order):</p> <ul> <li><strong>X.</strong> (<em>int</em>) - experiment ID; column is used as index.</li> <li><strong>Material </strong>(<em>string</em>) - the data set used for language distance measurement. The possible values are: <ul> <li>Slavic standard - Croatian, Slovenian, Slovak standard lects.</li> <li>Modern East Slavic - Northern Russian lect Megra, Central Russian lect Belogornoje, and Northern Belarusian lect Zialionka.</li> <li>Old East Slavic - Novgorod, Polack and Smolensk parts of the Old East Slavic continuum.</li> </ul> </li> <li><strong>Gensim </strong>(<em>int</em>) - the binary numeric indicator (0 or 1) of using the heuristic of <em>topic antimodelling</em>, namely, cleaning the words that were defined as a topic words by <em>gensim</em> Latent Dirichlet Association implementation (Rehurek & Sojka, 2010). The intention of using this heuristic is to remove the tokens that are characteristic for the genre of the texts presented in the corpus for the sake of increasing the presence of the tokens that are characteristic of the lects themselves.</li> <li><strong>Split</strong> (<em>float</em>) - the used share of the data set (from 0 to 1); required to check the influence of the data set size on the metric efficiency.</li> <li><strong>Hybridisation</strong> (<em>string</em>) - the indicator of implementation of the hybridisation between the frequency-based metric between the 3-shingles (character 3-grams) that coincide for the compared lect pair, and the combination of frequency-based metric and string similarity measure between the 3-shingles that do not coincide for the compared lect pair. The possible values are: <ul> <li>TRUE: the experiment utilises hybridisation</li> <li>FALSE: the experiment does not utilise hybridisation.</li> </ul> </li> <li><strong>Hybridisation_type</strong> (<em>string</em>) - the indicator of how the frequency metric between coinciding 3-shingles and the combined metric between non-coinciding 3-shingles undergo the hybridisation process. The values are:<br> <ul> <li>JOINED - the approach is to multiply the means of the two.</li> <li>ARRAY - the approach is to join all the values into a single list, and then to score the mean.</li> <li>NOT_USED - experiment does not employ hybridisation. (<strong>Hybridisation </strong>is FALSE).</li> </ul> </li> <li><strong>Soerensen_normalisation</strong> (<em>string</em>) - the indicator of whether the frequency-based metric value undergoes normalisation with the division by Soerensen-Dice coefficient (a measure of number of coincidences between two lists) (Soerensen, 1948), in order to compensate the skewing between the coinciding and non-coinciding 3-shingles of the lects. The values are: <ul> <li>NOT_USED - <strong>Hybridisation_type </strong>is ARRAY, so there are no values to use Soerensen-Dice coefficient on.</li> <li>TRUE - the frequency-based metric undergoes division by the Soerensen-Dice coefficient.</li> <li>FALSE - the algorithm does not apply the normalisation by the Soerensen-Dice coefficient.</li> </ul> </li> <li><strong>Alphabet_normalisation</strong> (<em>string</em>) - indicator of whether the algorithm applies normalisation with the alphabet entropy (Shannon, 1948), the measure of differences in the skewings of symbols distribution in the texts, between the given lects. The values are:<br> <ul> <li>NOT_USED - a heuristic may not be implemented; present either in the cases, when <strong>Hybridisation </strong>is FALSE, or when the next parameter, <strong>Auxiliary_metrics</strong> is not VDND or VWJDND.</li> <li>TRUE<strong> </strong>- the experiment employs the heuristic.</li> <li>FALSE - the experiment does not employ the heuristic.</li> </ul> </li> <li><strong>Auxiliary_metrics </strong>(<em>string</em>) - the string similarity measure, used for the combination with the frequency-based metric for non-coinciding 3-shingles between analysed lects. There are five possible values: <ul> <li>LDND (Levenshtein distance normalised between analysed 3-shingles) (Holman et al., 2008).</li> <li>WJWDND (weighted Jaro-Winkler distance normalised between analysed 3-shingles) (Gueddah et al., 2015).</li> <li>VDND (Euclidean distance between the sums of symbol vector values between 3-shingles).</li> <li>VWJDND (VDND multiplied by scoring Jaro (Jaro, 1989) distance between analysed 3-shingles).</li> </ul> </li> <li><strong>Outgroup.identification</strong> (<em>string</em>) - the indicator of whether the outgroup detected in the given experiment coincides with the lect that preliminary manual classification supposes to be the outgroup. There are two possible values:<br> <ul> <li>CORRECT - the detected outgroup coincides with the supposed one.</li> <li>INCORRECT - the detected outgroup does not coincide with the supposed one.</li> </ul> </li> <li><strong>Outer.distance.split </strong>(<em>float</em>) - the length of the outgroup branch.</li> <li><strong>Inner.distance.split</strong> (<em>float</em>) - the distance between the split between the outgroup and the ingroup, and the split between the two ingroup lects.</li> <li><strong>Split.difference</strong> (<em>float</em>) - the division of <strong>Outer.distance.split</strong> by <strong>Inner.distance.split</strong>.</li> </ul> <h2>Folders</h2> <p>Each folder contains 6 files, each named according to the used experiment setup:</p> <ul> <li>3 <code>.csv</code>-files that contain unit-by-unit comparison between each pair of the analysed lects. Each <code>.csv</code>-file is <strong>semi-colon</strong>-separated, and has <strong>4 </strong>columns, <strong>header row</strong>, and rows that describes each unit-to-unit comparison. The columns contain the following information (in <strong>rtl</strong> order):<br> <ul> <li><code>[Name of the first compared lect]</code> : unit (character 3-shingle, or just 3-shingle) of the [name of the first compared lect] that undergoes comparison with units of the [name of the second compared lect]; datatype: string.</li> <li><code>[Name of the second compared lect] </code>: unit (character 3-shingle, or just 3-shingle) of the [name of the second compared lect] that undergoes comparison with units of the [name of the first compared lect]; if units coincide, contains value <code>id.</code>; datatype: string.</li> <li><code>[Experiment setup]</code>: name of the metric, a combination of the [experiment setup](concatenated through <code><em>-</em></code> parameters) and its exact part, which compares the two units; datatype: string. The possible values are: <ul> <li><code>[experiment setup] - DistRank</code> - the frequency-based metric that compares identical units</li> <li><code>[experiment setup] - hybrid</code> - the string similarity measure for non-identical units, combined with the frequency-based metric </li> </ul> </li> <li><code>Distance</code>: value of the metric; datatype: float</li> </ul> </li> <li><code>.info</code>-file that contains data on branch lengths along with coincidence/non-coincidence of the detected outgroup with the manually defined one. The file is a <strong>tabular-separated plain text</strong> that always contains three values: coincidence (CORRECT)/non-coincidence (INCORRECT) of the yielded classification with the supposed one; outer distance split (the length of the outgroup branch; datatype: float) and inner distance split (the length of the ingroup branch before split of its lects; datatype: float).</li> <li><code>.newick</code> -file that contains the result of an experiment, the phylogenetic tree built by UPGMA classifier. One can read it with <a href="https://cran.r-project.org/web/packages/TreeTools/vignettes/load-trees.html">ape::read.tree</a> (R), or <a href="https://biopython.org/wiki/Phylo">Phylo.read</a> (Python).</li> <li><code>.png</code> -file that contains the phylogenetic tree visualisation.</li> </ul> <p> </p> <h1>How-to</h1> <p>For the analysis of the results, download and unpack the archive, and further refer to the <a href="https://doi.org/10.5281/zenodo.14169792" target="_blank" rel="noopener">companion R notebook</a>. </p>
COMPARISON OF NUMBER CATEGORIES IN UZBEK, RUSSIAN, AND ENGLISH LANGUAGES
Open the record for dataset details and reuse information.
THE TYPOLOGY OF LINGUISTIC AND CULTURAL TYPES OF VERBALIZERS OBJECTIFYING THE CONCEPT OF "COMPARISON" IN MULTI-SYSTEM LANGUAGES.
Open the record for dataset details and reuse information.
A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineering
<p>The file contains the training dataset used in our study. And, it includes the results of all NLUs for both tasks.</p>
Presurgical Language Mapping With fMRI: Comparison of BOLD and fASL Techniques
ClinicalTrials.gov study NCT01793714. IPD Sharing: Not stated. Countries: 1. Publications: 0.
Follow-up of Cochlear Implanted Children at 3 Years : Comparison of Electrophysiological and Speech-language Results
ClinicalTrials.gov study NCT04192968. IPD Sharing: NO. Countries: 1. Publications: 0.
Comparison of a Speech-Language Assessment Tool for Pragmatic and Discourse Skills in Schizophrenia Spectrum Disorders
ClinicalTrials.gov study NCT05742893. IPD Sharing: NO. Countries: 1. Publications: 0.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.