Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
6
datasets available to search
ShareScore release 0.9.0
Dataset results
6 results for “stopwords”
StopWords dataset: Integration of a set of stopwords in English and Portuguese - rev. 1
<p><br>StopWords dataset: Integration of a set of stopwords in English and Portuguese - rev. 1</p> <p>================================</p> <p>StopWords dataset - rev. 1 (two MS-Excel files)</p> <p>-------------<br><strong>StopWords Integrated</strong><br>Basic integration of a set of stopwords (English and Portuguese) for use in Text Mining tasks.</p> <p>File name 1: StopWords_Integrated_Favaretto.xlsx</p> <p>Tab 1 of MS-Excel: pt_accent (215 words)<br>Column Label: stopwords_pt</p> <p>Tab 2 of MS-Excel: pt_noaccent (208 words)<br>Column Label: stopwords_pt_na</p> <p>Tab 3 of MS-Excel: en (213 words)<br>Column Label: stopwords_en</p> <p><br>-------------<br><strong>StopWords Extended</strong><br>Extension of a set of stopwords (English and Portuguese) for use in Text Mining tasks.</p> <p>File name 2: StopWords_Extended_Favaretto.xlsx</p> <p>Tab 1 of MS-Excel: pt_extend (614 words)<br>Column Label: stopwords_pt_extend</p> <p>Tab 2 of MS-Excel: en_extended (483 words)<br>Column Label: stopwords_en_extend</p> <p><br>================================</p> <p><br><strong>Warning</strong>: Some words in this set of stopwords may even be misspelled intentionally, as they may occur in practice in texts that are not written correctly.</p> <p><strong>Aviso</strong>: Algumas palavras deste conjunto de stopwords podem até mesmo ter grafia errada de forma intencional, pois podem ocorrer na prática em textos não escritos corretamente.</p> <p><br>================================</p> <p>Source: elaborated by Prof. Dr. José Eduardo Ricciardi Favaretto based on a mix of several different sources</p> <p>https://orcid.org/0000-0002-0143-0809<br>https://lattes.cnpq.br/3790103269421610<br>https://linkedin.com/in/favaretto</p> <p><br>================================</p>
Lists of Uzbek Stopwords
<p>The dataset presents 3 lists of stopwords in the Uzbek language. The lists were constructed using three automatic methods applied to the same corpus. </p> <p>The corpus was constructed by obtaining a source of 25 school textbooks, it was named "School corpus". The total number of words in the "School corpus" is 731156. And them 47165 words are unique words. The corpus can be re-constructed using the list of urls of all files comprised in the corpus. The list is part of the dataset (list_of_urls_of_school_corpus.txt).</p> <p>Description of the methods and the lists:</p> <p>A set of grammar rules and the TDIDF algorithm was used to automatically collect a list of single-word stopwords. 2357 stopwords were collected. The name of the file: stopwords_unigrams.txt.</p> <p>A bigram method was used to extract a list of 4548 bigrams (pairs) of stopwords. The name of the file: stopwords_bigrams.txt.</p> <p>A set of two-word collocations as stopwords was also extracted. The list has 24490 pairs of stopwords. The name of the file: stopwords_bigrams_with_collocations.txt.</p>
Lists of Karakalpak Stopwords
<p>The dataset presents 3 lists of stopwords in the Karakalpak language. The lists were constructed using three automatic methods applied to the same corpus. </p> <p>The corpus was constructed by obtaining a source of 23 school textbooks, it was named "Karakalpak School Corpus". The corpus can be re-constructed using the list of urls of all files comprised in the corpus. The list is part of the dataset (list_of_urls_for_karakalpak_school_corpus.txt).</p> <p>Description of the methods and the lists:</p> <p>A set of grammar rules and the TDIDF algorithm were used to automatically collect a list of single-word stopwords. 4014 stopwords were collected. The name of the file: Karakalpak_stopwords_unigrams.txt.</p> <p>A bigram method was used to extract a list of 3740 bigrams (pairs) of stopwords. The name of the file: Karakalpak_stopwords_bigram.txt.</p> <p>A set of two-word collocations as stopwords was also extracted. The list has 20745 pairs of stopwords. The name of the file: Karakalpak_stopwords_bigrams_with_collocations.txt.</p>
Lists of stopwords, polarity shifters and AnAwords of Bosnian language
<p>The dataset comprises three lists, a list of stopwords, a list of polarity shifters and a list of AnAwords (in two files) of the Bosnian language.</p> <p><strong>Stopwords</strong> refer to a set of words contained in a stop list that are deliberately filtered out or "stopped" during the processing of natural language data, specifically text. These words are typically common and frequently occurring words in a language that are considered to have little or no significance in determining the meaning or context of a text.</p> <p><strong>AnAwords (intensifiers and diminishers)</strong> refer to a set of words primarily functioning as intensifiers and diminishers, often manifesting as adverbs of manner and adjectives. The compilation of AnAwords is based on categorization, which includes six sublists: maximizers, boosters, approximators, relative intensifiers, diminishers, and minimizers. The list is split into two parts (intensifiers and diminishers) in two separate files.</p> <p><strong>Polarity shifters </strong>are words that can affect the <strong>polarity</strong> of a phrase, inverting or weakening it. When these words are content <strong>words</strong>, such as verbs, nouns, and adjectives, we refer to them as <strong>polarity</strong> <strong>shifters</strong>.</p>
SE Stopwords
<p><strong>Overview</strong></p> <p>This repository contains stopword lists specifically tailored for natural language processing (NLP) tasks applied to software development documents. It aims to enhance the efficiency and accuracy of NLP applications on various types of software documentation, including bug reports, commit messages, and API documentation.</p> <p><strong>Background and Motivation</strong></p> <p>Stop words, deemed non-predictive, are often eliminated in NLP tasks. However, the definition of uninformative vocabulary remains vague, leading most algorithms to use general knowledge-based stop lists. The effectiveness of stop word elimination, particularly in domain-specific settings, is debated among academics.</p> <p>In a recent <a href="https://arxiv.org/abs/2303.10439">paper</a>, we investigated the usefulness of stop word removal in a software engineering context. To achieve this, we replicated and experimented with three software engineering research tools from related work. A corpus of software engineering domain-related text was constructed from 10,000 Stack Overflow questions, and 200 domain-specific stop words were identified using traditional information-theoretic methods.</p> <p>The results demonstrated that using domain-specific stop words significantly improved the performance of research tools compared to a general stop list. Moreover, 17 out of 19 evaluation measures showed better performance.</p> <p><strong>Comparison to Baseline across 19 Metrics</strong></p> <p>The table below summarizes the performance improvements when using different stopword lists compared to the baseline across 19 metrics.</p> <table> <thead> <tr> <th>Stop word list</th> <th>Better</th> <th>Worse</th> <th>Same</th> </tr> </thead> <tbody> <tr> <td>SE Domain (TF-IDF) (<a href="https://github.com/ctreude/SE-stopwords/blob/main/stopwords_lists/tfidf_approach.txt">link</a>)</td> <td>17</td> <td>1</td> <td>1</td> </tr> <tr> <td>SE Domain (Poisson) (<a href="https://github.com/ctreude/SE-stopwords/blob/main/stopwords_lists/possion_approach.txt">link</a>)</td> <td>12</td> <td>5</td> <td>2</td> </tr> <tr> <td>Technology Domain (<a href="https://github.com/ctreude/SE-stopwords/blob/main/stopwords_lists/technology_domain_stopwords.txt">link</a>)</td> <td>9</td> <td>9</td> <td>1</td> </tr> <tr> <td>Large (<a href="https://github.com/ctreude/SE-stopwords/blob/main/stopwords_lists/large_stopwords.txt">link</a>)</td> <td>11</td> <td>8</td> <td>0</td> </tr> <tr> <td>Medium (<a href="https://github.com/ctreude/SE-stopwords/blob/main/stopwords_lists/medium.txt">link</a>)</td> <td>11</td> <td>7</td> <td>1</td> </tr> <tr> <td>Small (<a href="https://github.com/ctreude/SE-stopwords/blob/main/stopwords_lists/small.txt">link</a>)</td> <td>13</td> <td>5</td> <td>1</td> </tr> <tr> <td>Very Small (<a href="https://github.com/ctreude/SE-stopwords/blob/main/stopwords_lists/very_small.txt">link</a>)</td> <td>10</td> <td>7</td> <td>2</td> </tr> <tr> <td>No Stop Words</td> <td>4</td> <td>12</td> <td>3</td> </tr> </tbody> </table> <p><strong>Usage Instructions</strong></p> <p>These stopword lists can be used to filter out uninformative words from software development documents, thereby improving the understanding and analysis of textual data in the software development domain.</p> <p>To use these lists in your NLP tasks, simply import them into your project and apply them as filters during the pre-processing stage.</p> <p><strong>Folder Structure</strong></p> <pre><code>SE-stopwords |-- data_for_replications (contains all the required data for replicating software engineering tools) | |-- Maalej_Dataset (original data for app review tool) | `-- queries (queries used for RACKTool) |-- stackoverflow_questions (more than 10k top reviewed questions on stackoverflow) |-- stopwords_lists (all the stoplists) |-- replications `-- stackoverflow (code for creating the domain-specific corpus) </code></pre> <p><strong>Detailed Results for the Three Replicated Tools</strong></p> <p>The results may vary by a small fraction depending on the trial, but they should be approximately the same as the tables below.</p> <p><em><strong>Tool 1 (App Review)</strong></em></p> <table> <thead> <tr> <th> </th> <th><strong>PD  (bug report)</strong></th> <th><strong>RT  (rating)</strong></th> <th><strong>FR  (feature request)</strong></th> <th><strong>UE  (user experience)</strong></th> </tr> </thead> <tbody> <tr> <td> </td> <td>Pre  Rec  F1</td> <td>Pre  Rec  F1</td> <td>Pre  Rec  F1</td> <td>Pre  Rec  F1</td> </tr> <tr> <td><strong>SE domain (Poisson)</strong></td> <td>10.0% 37.5% 15.8%</td> <td>72.1% 78.0% 74.9%</td> <td>7.1% 29.8% 11.5%</td> <td>11.6% 32.0% 17.0%</td> </tr> <tr> <td><strong>SE domain (TF-IDF)</strong></td> <td>10.7% 40.2% 16.9%</td> <td>72.2% 78.2% 75.1%</td> <td>7.9% 33.3% 12.8%</td> <td>11.7% 32.5% 17.2%</td> </tr> </tbody> </table> <p><em><strong>Tool 2 (RACK)</strong></em></p> <table> <thead> <tr> <th> </th> <th><strong>Top-10</strong></th> <th><strong>MRR@10</strong></th> <th><strong>MAP@10</strong></th> <th><strong>MR@K</strong></th> </tr> </thead> <tbody> <tr> <td><strong>SE domain (Poisson)</strong></td> <td>83.85%</td> <td>52.29%</td> <td>43.27%</td> <td>54.47%</td> </tr> <tr> <td><strong>SE domain (TF-IDF)</strong></td> <td>84.17%</td> <td>53.20%</td> <td>45.82%</td> <td>56.8%</td> </tr> </tbody> </table> <p><em><strong>Tool 3 (Requirements Change Impact Analysis)</strong></em></p> <table> <thead> <tr> <th> </th> <th><strong>SE domain (Poisson)</strong></th> <th><strong>SE domain (TF-IDF)</strong></th> </tr> </thead> <tbody> <tr> <td><strong>Query 2</strong></td> <td>0.588</td> <td>0.588</td> </tr> <tr> <td><strong>Query 4</strong></td> <td>0.981</td> <td>0.981</td> </tr> <tr> <td><strong>Query 5</strong></td> <td>0.602</td> <td>0.602</td> </tr> </tbody> </table> <p><strong>Citation</strong></p> <p>If you make use of this work, please cite:</p> <pre><code>@inproceedings{fan2023stop, title={Stop Words for Processing Software Engineering Documents: Do they Matter?}, author={Yaohou Fan and Chetan Arora and Christoph Treude}, booktitle={2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE)}, year={2023}, organization={IEEE} } </code></pre>
stopword
<p>stop word in Chinese</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.