Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
167
datasets available to search
ShareScore release 0.9.0
Dataset results
167 results for “large language models”
Exploring the Impact of the Output Format on the Evaluation of Large Language Models for Code Translation
<p>Please find the most up to date version of the dataset at<br>https://github.com/RISElabQueens/forge24-code-translation</p> <p> </p> <p>Thank you</p>
Artifact of "How Effective are Large Language Models in Generating Software Specifications?"
<p>We release the artifact of paper "How Effective are Large Language Models in Generating Software Specifications?" For more details, please refer to the readme.md in the folder.</p>
Malicious Large Language Models Detection using Metadata Information
<p># Introduction<br># This is the replication package for the paper "Malicious Large Language Models Detection using Metadata Information".</p> <p><br>## Task Definition<br>Given the information of LLM, the task is to identify whether it is a malicious LLM that may attack software systems. We treat the task as binary classification (0/1), where 1 stands for malicious LLMs and 0 for malicious-free LLMs.</p> <p><br>## The static of dataset<br>### Data Format<br>Before preprocessing dataset, each line in the uncompressed file represents multiple metadata of one large language model (LLM). One row is illustrated below.<br> - **idx:** the index of example<br> - **repo_id:** the id of LLM (e.g., microsoft/codebert-base)<br> - **tags:** the tags of LLM<br> - **pipeline_tag:** the pipeline_tag of LLM<br> - **downloads:** the number of downloads<br> - **created_time:** the created time of LLM<br> - **modelCard:** the text content of LLM<br> - **num_discussion:** the number of discussions<br> - **discussion:** the discussions of LLM<br> - **para_size:** the size of LLM<br> - **tensor_type:** the type of LLM<br> - **num_commit:** the number of commits<br> - **commit:** the commit of LLM</p> <p>After preprocessing dataset, you can obtain three .csv files, i.e. train.csv, valid.csv, test.csv<br> - **idx:** the index of example.<br> - **repo_id:** the id of LLM (e.g., microsoft/codebert-base).<br> - **tags:** the tags of LLM.<br> - **pipeline_tags:** the pipeline_tag of LLM.<br> - **created_time:** the created time of LLM.<br> - **model_size:** the size of LLM.<br> - **Tensor_type:** the type of LLM.<br> - **is_model_card:** Whether to include model card. If the model has model card, the value is 1, otherwise, the value is 0.<br> - **malicious_model_card:** Whether to include keywords describing the malicious model in model card. If the model card has malicious keywords, the value is 1, otherwise, the value is 0<br> - **repository_link:** Whether to include repository link: GitHub link, Arxiv link, homepage link, bugs link and issues link in model card. If the model card has link, the value is 1, otherwise, the value is 0.<br> - **dataset_info:** Whether to include the adopted dataset information in model card. If the model card has dataset information, the value is 1, otherwise, the value is 0.<br> - **metrics_info:** Whether to include the evaluated metrics information in model card. If the model card has evaluation metrics information, the value is 1, otherwise, the value is 0.<br> ?- **script_info:** Whether to include script information. If the model has script information, the value is 1, otherwise, the value is 0.<br> - **config_content:** The content of the configuration script file. This value is string type.<br> - **stakeholder_name:** The name of authors, contributors, and maintainers. This value is string type.<br> - **number_discussion:** The number of discussion. <br> - **num_pr:** The number of pull request.<br> - **malicious_discussion:** Whether the discussion contains malicious behavior keywords. If the discussion has malicious behavior keywords, the value is 1, otherwise, the value is 0.<br> - **number_commit:** The number of commit.<br> - **malicious_commit:** Whether the title and message of commits contain malicious behavior keywords. If the commit has malicious behavior keywords, the value is 1, otherwise, the value is 0.<br> - **z_download:** The z-score of number of download.<br> - **z_like:** The z-score of number of likes.</p> <p><br>### Data Statistics<br>Data statistics of the dataset are shown in the below table:</p> <p>| #File Names | #Examples |<br>| ------------------- | :------------------------: |<br>| dataset_feature.csv | 578,502 (560,257/18,245) | <br>| train_imbalance.csv | 462,801 (448205/14596) |<br>| valid_imbalance.csv | 57,849 (56025/1824) |<br>| test_imbalance.csv | 57,852 (56027/1825) |<br>| train_balance.csv | 29192 (14596/14596) |<br>| valid_balance.csv | 3648 (1824/1824) |<br>| test_balance.csv | 3650 (1825/1825) |<br>| train_imbalance_50.csv | 289252 (280129/9123) |<br>| train_imbalance_60.csv | 347101 (336154/10947) |<br>| train_imbalance_70.csv | 404952 (392180/12772) |</p> <p>29646 models contain github link<br>all_dataset.csv 596383<br>all_dataset_information.csv 589140 (safe_dataset.csv(570549), unsafe_dataset.csv(18591))<br>safe_dataset_information.csv (559582), unsafe_dataset_information.csv (18212)</p> <p>Description Feature: 'malicious_model_card', 'repository_link', 'dataset_info', 'metrics_info', 'config_conteng'<br>Stakeholder Feature: 'stakeholder_name'<br>Event Feature: 'num_pr', 'number_commit', 'malicious_commit',<br>Context Feature: 'z_download', 'z_like'</p> <p><br>## Pipeline-MPTMHunter<br>We also provide a pipeline that fine-tunes [MPTMHunter](https://doi.org/10.5281/zenodo.12578531) on this task.</p> <p>### Experimental environment configuration<br>```bash<br>huggingface_hub 0.23.1<br>libxgboost 2.0.3<br>lightgbm 4.3.0<br>networkx 3.2.1<br>nltk 3.8.1<br>numpy 1.26.3<br>openssl 3.0.13<br>pandas 2.2.1<br>pillow 10.2.0<br>scikit-learn 1.4.2<br>scipy 1.13.0<br>torch 2.3.0+cu118<br>torchaudio 2.3.0+cu118<br>torchvision 0.18.0+cu118<br>tqdm 4.66.2<br>transformers 4.37.2<br>xgboost 2.0.3<br>```</p> <p>### Dataset Collection Script<br>```bash<br>python ./script/DataExtraction.ipynb<br>python ./script/dataset_spider.py<br>python ./script/config_crawl.py<br>```</p> <p>### Dataset Preprocess Script<br>```bash<br>python feature_generation.py --input_file='../dataset/dataset_information.csv' --output_file='../dataset/dataset_feature.csv'</p> <p>python feature_generation.py --input_file='../dataset/real_world_dataset_information_0701.csv' --output_file='../dataset/real_world_dataset_feature_0701.csv'<br>```</p> <p>### Model Training Script<br>```bash<br>python run_codet5_lstm.py --output_dir='../saved_models/codet5_lstm_imbalance_final' --model_type=codet5 --tokenizer_name='../models/codet5' --model_name_or_path='../models/codet5' --do_train --train_data_file='../dataset/train_imbalance_70.csv' --eval_data_file='../dataset/valid_imbalance_70.csv' --test_data_file='../dataset/test_imbalance.csv' --epoch=3 --block_size=510 --train_batch_size=64 --eval_batch_size=64 --learning_rate=2e-5 --max_grad_norm=1.0 --evaluate_during_training --seed=123456<br>```</p> <p>### Model Inference Script<br>```bash<br>python run_codet5_lstm.py --output_dir='../saved_models/codet5_lstm_imbalance' --model_type=codet5 --tokenizer_name='../models/codet5' --model_name_or_path='../models/codet5' --do_eval --do_test --train_data_file='../dataset/train_imbalance_70.csv' --eval_data_file='../dataset/valid_imbalance_70.csv' --test_data_file='../dataset/test_imbalance.csv' --epoch=3 --block_size=510 --train_batch_size=64 --eval_batch_size=64 --learning_rate=2e-5 --max_grad_norm=1.0 --evaluate_during_training --seed=123456<br>```</p> <p>### Evaluation Script<br>```bash<br>python ../evaluation/evaluation.py -a ../dataset/test_balance.csv -p ../saved_models/codebert_imbalance_all/predictions.txt<br>python ../evaluation/evaluation.py -a ../dataset/test_imbalance.csv -p ../dataset/predictions.txt<br>```</p> <p>## Result<br>The results on the test set are shown as below (We use the OpenTextClassification as the baseline):</p> <p>| Methods | ACC | Precision | Recall | F1-Score |<br>| Random Forest | 97.18% | 83.80% | 13.04% | 22.57% |<br>| LR | 96.80% | 43.46% | 4.55% | 8.23% |<br>| LightGBM | 97.62% | 95.18% | 25.97% | 40.81% |<br>| TextRNN | 96.86% | 73.33% | 0.60% | 1.20% |<br>| TextCNN | 98.89% | 95.61% | 68.00% | 79.47% |<br>| TextRCNN | 98.87% | 95.36% | 67.62% | 79.13% |<br>| TextRNN_Att | 98.94% | 95.29% | 69.86% | 80.62% |<br>| MPTMHunter | ** 99.99% ** | ** 99.95% ** | ** 99.78% ** | ** 99.86% ** |</p>
TrustMus benchmark: The Role of Large Language Models in Musicology: Are We Ready to Trust the Machines?
<p>TrustMus is an initial, rigorously validated benchmark designed to assess the accuracy and reliability of large language models (LLMs) in the domain of musicology. This dataset includes a collection of 400 human-validated multiple-choice questions, categorized into four thematic areas: People (Ppl), Instruments and Technology (I&T), Genres, Forms, and Theory (Thr), and Culture and History (C&H).</p> <p>The questions are derived from <em>The Grove Dictionary Online</em> using a semi-automated methodology. The process involves generating initial questions with a fine-tuned retrieval-augmented generation (RAG) model, filtering them through a series of automated checks, and finally validating them through expert human annotation. TrustMus is introduced in an initial paper, providing a critical resource for researchers and developers aiming to evaluate and improve LLM performance in this specialized field of musicology.</p> <p>This benchmark is discussed in the paper : </p> <p><strong>BibTeX Citation:</strong></p> <div> <div><code>@inproceedings{ramoneda2024trustmus,</code><br><code> title={The Role of Large Language Models in Musicology: Are We Ready to Trust the Machines?},</code><br><code> author={Ramoneda, Pedro and Parada-Cabaleiro, Emilia and Weck, Benno and Serra, Xavier},</code><br><code> booktitle={Proceedings of the 3rd Workshop on NLP for Music and Audio (NLP4MusA)},</code><br><code> year={2024},</code><br><code> month={November},</code><br><code> address={San Francisco, USA},</code><br><code> organization={Co-located with ISMIR'2024}</code><br><code>}</code><br><br> <div> </div> </div> </div>
Replication Package for "On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?"
<h1>Replication Package for "On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?"</h1> <h2>Authors</h2> <p>Matteo Esposito, University of Oulu, Finland Francesco Palagiano, Multitel di Lerede Alessandro & C. s.a.s. Valentina Lenarduzzi, University of Oulu, Finland Davide Taibi, University of Oulu, Finland</p> <h2>Content Overview</h2> <p>This replication package contains the following materials:</p> <ul> <li>Data: RAW Questionnaire dataset.</li> <li>Contact Information</li> </ul> <p>For any issues, questions, or further assistance, please do not hesitate to contact the paper's authors. We are here to help!</p>
Dataset of the paper "Software Supply Chain Meets Large Language Models: Can Dependency-related Problems Be Solved?"
<p>This is the dataset of the paper "Software Supply Chain Meets Large Language Models: Can Dependency-related Problems Be Solved?"</p>
Data from: experiment using Large Language Models for unit testing generation
Open the record for dataset details and reuse information.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.