Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
6
datasets available to search
ShareScore release 0.9.0
Dataset results
6 results for “Distributional semantics”
Biolinks, datasets and algorithms supporting semantic-based distribution and similarity for scientific publications
<p><strong>Background: </strong>Finding articles related to a publication of interest remains a challenge in the Life Sciences domain as the number of scientific publications grows day by day. Publication repositories such as PubMed and Elsevier provides a list of similar articles. There, similarity is commonly calculated based on title, abstract and some keywords assigned to articles. Here we present the datasets and algorithms used in Biolinks. Biolinks uses ontological concepts extracted from publication and makes it possible to calculate a distribution score according to semantic groups as well as a semantic similarity based on either all identified annotations or narrowed to one or more particular semantic groups. Biolinks supports both title and abstract only as well as full-text.</p> <p><strong>Materials: </strong>In a previous work [1], 4,240 articles from the TREC-05 collection [2] were selected. The title-and-abstract for those 4,240 articles were annotated with Unified Medical Language System (UMLS) concepts, such annotations are refer to as our TA-dataset and correspond to the JSON files under the pubmed folder in the JSON-LD.zip file. From those 4,240 articles, full-text was available for only 62. The title-and-abstract annotations for those 62 articles, TAFT-dataset, are located under the pubmed-pmc folder in the JSON-LD.zip file, which also contains the full-text annotations under the folder pmc, FT-dataset. The list corresponding to articles with title-and-abstract is found in the genomics.qrels.large.pubmed.onlyRelevants.titleAndAbstract.tsv file, while those with full-text are recorded in the genomics.qrels.large.pmc.onlyRelevants.fullContent.tsv file.</p> <p>Here we include the annotations on title and abstract as well as those for full-text for all our datasets (profiles.zip). We also provide the global similarity matrices (similarity.zip).</p> <p><strong>Methods:</strong> The TA-dataset was used to calculate the Information Gain (IG) according to the UMLS semantic groups, see IG_umls_groups.PMID.xlsx. A new grouping is proposed for Biolinks, see biolinks_groups.tsv. The IG was calculated for Biolinks groups as well, IG_biolinks_groups.PMID.xlsx, showing a improvement around 5%.</p> <p>In order to assess the similarity metric regarding the cohesion of TREC-05 groups, we used Silhouette Coefficient analyses. An additional dataset Stem-TAFT-dataset was used and compared to TAFT and FT datasets.</p> <p>Biolinks groups were used to calculate a semantic group distribution score for each article in all our datasets. A semantic similarity metric based on PubMed related articles [3] is also provided; the Biolinks groups can be used to narrow the similarity to one or more selected groups. All the corresponding algorithms are open-access and available on GitHub under the license Apache-2.0, a frozen version, biotea-io-parser-master.zip, is provided here. In order to facilitate the analysis of our datasets based on the annotations as well as the distribution and similarity scores, some web-based visualization components were created. All of them open-access and available in GitHub under the license Apache-2.0; frozen versions are provided here, see files biotea-vis-annotation-master.zip, biotea-vis-similarity-master.zip, biotea-vis-tooltip-master.zip and biotea-vis-topicDistribution-master.zip. These components are brought together by biotea-vis-biolinks-master.zip. A demo is provided at http://ljgarcia.github.io/biotea-biolinks/; this demo was built on top of GitHub pages, a frozen version of the gh-pages branch is provided here, see biotea-biolinks-gh-pages.zip.</p> <p><strong>Conclusions: </strong>Biolinks assigns a weight to each semantic group based on the annotations extracted from either title-and-abstract or full-text articles. It also measures similarity for a pair of documents using the semantic information. The distribution and similarity metrics can be narrowed to a subset of the semantic groups, enabling researchers to focus on what is more relevant to them.</p> <p> </p> <p>[1] Garcia Castro, L.J., R. Berlanga, and A. Garcia, <em>In the pursuit of a semantic similarity metric based on UMLS annotations for articles in PubMed Central Open Access.</em> Journal of Biomedical Informatics, 2015. <strong>57</strong>: p. 204-218</p> <p>[2] Text Retrieval Conference 2005 - Genomics Track. <em>TREC-05 Genomics Track ad hoc relevance judgement</em>. 2005 [cited 2016 23rd August]; Available from: http://trec.nist.gov/data/genomics/05/genomics.qrels.large.txt</p> <p>[3] Lin, J. and W.J. Wilbur, <em>PubMed related articles: a probabilistic topic-based model for content similarity.</em> BMC Bioinformatics, 2007. <strong>8</strong>(1): p. 423</p>
Improving Hypernymy Extraction with Distributional Semantic Classes
<p>In this paper, we show for the first time how distributionally-induced semantic classes can be helpful for extraction of hypernyms. We present a method for (1) inducing sense-aware semantic classes using distributional semantics and (2) using these induced semantic classes for filtering noisy hypernymy relations. Denoising of hypernyms is performed by labeling each semantic class with its hypernyms. On one hand, this allows us to filter out wrong extractions using the global structure of the distributionally similar senses. On the other hand, we infer missing hypernyms via label propagation to cluster terms. We conduct a large-scale crowdsourcing study showing that processing of automatically extracted hypernyms using our approach improves the quality of the hypernymy extraction both in terms of precision and recall. Furthermore, we show the utility of our method in the domain taxonomy induction task, achieving the state-of-the-art results on a benchmarking dataset.</p> <p>This particular page contains datasets related to the paper. Namely the input induced word senses, a database of hypernyms, and the output clusters of senses labeled with hypernyms -- the distributional semantic classes. The semantic classes are of two granularities, as described in the paper (coarse and fine grained). </p>
Langmark: annotations for scenes with semantic inconsistencies connecting distributional semantic models to vision science – data and code
<p>Data (including object annotations) and code from the following manuscript:</p> <p><em>Langmark: annotations for scenes with semantic inconsistencies connecting distributional semantic models to vision science</em>.</p>
Investigating Distributional Robustness: Semantic Perturbations Using Generative Models (ImageNet Examples)
<p>This dataset contains examples of semantically-perturbed images, for NeurIPS 2020 submission #4915.</p> <p>There are four top-level folders, each containing results for semantic perturbations restricted to adjust the activation values at only certain layers of the BigGAN generative network: the first six layers, the middle six layers, the last six layers, and all layers.</p> <p>Within each top-level folder, there are a further four folders, each corresponding to a classifier neural network whose evaluation is being evaluated. These are EfficientNet-B4 with NoisyStudent training [1], the standard ResNet50 [2], a pixel-perturbation-robust ResNet50 trained by Engstrom et al. [3] and another trained by Wong et al. [4], using their "Fast is better than free" technique.</p> <p>Within each of these, there are many folders, named 'version_$N'. Each one of these contains three images: the unperturbed generated image, named unpert_generated_x_grid_0.png; the semantically-perturbed generated image, named generated_x_grid_0.png; and an image named semantic_pert_diffs_grid_0.png showing the pixel-space effect of the semantic perturbation, that is, the diff between the perturbed and unperturbed images. Note that if the perturbed and unperturbed images are identical, and the classifier misclassifies the unperturbed images, and so we skip this example.</p> <p>Along with the 'version_$N' folders containing the images, there exists a file for each classifier named results.json. Each top-level item in this JSON file corresponds to one 'version_$N' example. There are 5 attributes: 'label', indicating the target label of the unperturbed image; 'magnitude', which gives the magnitude of the semantic perturbation found; 'skipped_cla', which is 1 if the example is skipped because the classifier did not correctly classify the unperturbed image; 'skipped_judge', which is 1 if the human judged that the unperturbed image did not match its label, so this example is skipped; and 'pert_judgement', which is 1 if the semantically-perturbed image is judged by the human to be of the same class as the unperturbed image. These judgements on these images were used to construct the main graphs in the paper.</p> <p> </p> <p>[1] Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. Self-training with Noisy Student improves ImageNet classification. CoRR, abs/1911.04252, 2019. URL http://arxiv.org/abs/1911.04252.</p> <p>[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.90. URL https://doi.org/10.1109/CVPR.2016.90.</p> <p>[3] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, and Dimitris Tsipras. Robustness (Python library), 2019. URL ttps://github.com/MadryLab/robustness.</p> <p>[4] Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL<br> https://openreview.net/forum?id=BJx040EFvH.</p>
EIAH data model: semantic interoperability between distributed digital repositories
<p>The authors described their information architecture project aimed at improving access to the Encyclopaedia of Iranian architectural history (EIAH) by signalling relationships between concepts and between concepts and documents. The outcome will be presented in a semantic portal or might be used for complex search queries by end users.</p>
ARENA_Hierarchical Organization of Distributed Semantic Knowledge in the Human Language System_Language Study Pt. 1: stimulus set
<p>P4_WP1_01 Language Study Pt. 1: stimulus set</p> <p> </p> <p>Folder structure: </p> <p>Raw_input<br>The raw_input contains the text for every chapter of the book to be used in the experiment (Moonwalk mit Einstein: Wie aus einem vergeßlichen Mann ein Gedächtnis-Champion wurde, by Joshua Foer, translated by Ulla Rahn-Huber. Published by Riemann Verlag (28 Mar. 2011)). <br>Additionally, in the folder there is the pretrained vector model for German words (model comes from https://fasttext.cc/docs/en/crawl-vectors.html), ratings for concreteness and word frequency (all references are included in the scripts). Moreover, there is a translated version of the Things labels for intersecting the single words with the THINGS dataset (Hebart MN, Dickter AH, Kidder A, Kwok WY, Corriveau A, et al. (2019) THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PLOS ONE 14(10): e0223792. https://doi.org/10.1371/journal.pone.0223792)</p> <p>Scripts<br>In this folder, there are 6 scripts to sample single words from the raw text. The scripts are numbered according to the intended order of use. <br>- 1_from_text_to_df.py: from raw text only nouns and verbs are extracted with their relative word frequency, concreteness, lemma form, and number of characters. </p> <p>- 2_cluster_words.py: cluster analysis of word vectors to sample the semantic space as broadly as possible. Loosely based on Pereira, F., Lou, B., Pritchett, B. et al. Toward a universal decoder of linguistic meaning from brain activation. Nat Commun 9, 963 (2018). https://doi.org/10.1038/s41467-018-03068-4. </p> <p>- 3_compute_orthographic_density.py: add information for OND20 for both word forms and lemma forms. </p> <p>- 4_syntactic_valency.py: this applies only to verbs. It counts the number of arguments necessary for a verb to saturate its syntactic valency (e.g., subject + object). </p> <p>- 5_sampling_nouns.py: it samples nouns by preserving the distributions of all variables. The number of characters for every word is kept under 10. Additionally, the extreme quantiles of concreteness are matched by all other variables to ensure that more concrete and more abstract words in the set are still match along the other ratings. </p> <p>- 6_sampling_verbs.py: same as above but for verbs. </p> <p>Stimuli <br>In this folder, the pool of words before sampling is included. Note that some words have been manually excluded for several reasons: e.g., parsed wrongly in their lemma form; offensive words; words coming from other languages. </p> <p> </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.