Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
111
datasets available to search
ShareScore release 0.9.0
Dataset results
111 results for “LLMs”
Programming Problems Submitted for Evaluation of LLMs GPT-3.5 and Gemini Pro 1.0
<p>Problems extracted from platforms LeetCode and BeeCrowd for evaluation of LLMs GPT3.5 and Gemini Pro 1.0.</p> <p>The data from the plataforms has the following columns and values:</p> <p> </p> <table> <tbody> <tr> <td><strong>LeetCode Data</strong></td> <td> </td> <td><strong>BeeCrowd Data</strong></td> <td> </td> </tr> <tr> <td><strong>Column</strong></td> <td><strong>Doc</strong></td> <td><strong>Column</strong></td> <td><strong>Doc</strong></td> </tr> <tr> <td>problem_level</td> <td>easy | medium | hard</td> <td>problem_level</td> <td><1...10></td> </tr> <tr> <td>problem_link</td> <td><LeetCode link for the problem></td> <td>problem_link</td> <td><BeeCrowd link for the problem></td> </tr> <tr> <td>prompt</td> <td><text submitted to LLM></td> <td>prompt</td> <td><text submitted to LLM></td> </tr> <tr> <td>response_code</td> <td><code provided by the LLM></td> <td>response_code</td> <td><code provided by the LLM></td> </tr> <tr> <td>response_evaluation</td> <td>True | False</td> <td>response_evaluation</td> <td>True | False</td> </tr> <tr> <td>execution_time_ms</td> <td><time></td> <td>execution_time_ms</td> <td><time></td> </tr> <tr> <td>memory_usage_mb</td> <td><memory></td> <td>error_generated</td> <td>Wrong Answer | Time Limit Exceeded | Memory Limit Exceeded....</td> </tr> <tr> <td>error_generated</td> <td>Wrong Answer | Time Limit Exceeded | Memory Limit Exceeded....</td> <td>attempts_number</td> <td>1 | 2 | 3</td> </tr> <tr> <td>attempts_number</td> <td>1 | 2 | 3</td> <td>author</td> <td><author's name></td> </tr> <tr> <td>contains_image</td> <td>True | False</td> <td>source</td> <td><origin institution></td> </tr> <tr> <td>related_topic_1</td> <td><topic></td> <td>origin_country</td> <td><origin country></td> </tr> <tr> <td>related_topic_2</td> <td><topic></td> <td>contains_image</td> <td>True | False</td> </tr> <tr> <td>related_topic_3</td> <td><topic></td> <td>related_topic_1</td> <td><topic></td> </tr> <tr> <td>related_topic_4</td> <td><topic></td> <td> </td> <td> </td> </tr> <tr> <td>related_topic_5</td> <td><topic></td> <td> </td> <td> </td> </tr> </tbody> </table> <p> </p>
Analyzing and Mitigating (with LLMs) the Security Misconfigurations of Helm Charts from Artifact Hub
<p>In the corresponding scientific paper, we proposed a pipeline to mine Helm charts from Artifact Hub, a popular centralized repository, and analyze them using state-of-the-art open-source tools like Checkov and KICS. First, such a pipeline runs several chart analyzers and identifies the common and unique misconfigurations reported by each tool. Secondly, it uses LLMs to suggest mitigation for each misconfiguration. Finally, the chart refactoring previously generated is analyzed again by the same tools to see whether it satisfies the tool's policies.</p> <p>In this dataset, you can find all the Helm chart templates downloaded from Artifact Hub (available in June 2024), all the outputs of the tools analyzing such templates, the CSV result files with all LLM queries and answers, and the snippets selected for the manual analysis.</p>
Scholarly Wikidata: Population and Exploration of Conference Data in Wikidata using LLMs
<p>This dataset provides the input data and intermediate results of the paper titled "Scholarly Wikidata: Population and Exploration of Conference Data in Wikidata using Large Language Models and Semantic Web Techniques". It contains the following resources.</p> <ul> <li>conference proceedings front matter links - these links can be used to download the pdf files of the conference proceeding front matters that include information about the number of submitted and accepted papers that can be used to calculate acceptance rates, names of all conference organization committee members, list of programme committee and senior programme member names for each track with other interesting facts such as the main topics of the submitted papers and emerging topics according to the editors, etc.</li> <li>web crawl of conference websites - this contains a set of crawled content from each conference website in both HTML and text formats. Each file contains web pages from a specific conference along with the page URL, page title, and page content. Information such as important dates (deadlines) and other announcements can be extracted from the content of the web sites. </li> <li>papers and paper-authors list for each conference in a given conference series - this contains the paper list along with their corresponding authors for each conference series extracted from DBLP. </li> <li>OpenRefine projects - this contains examples of open refile projects that were used to perform entity linking and reconciliation as well as the schemas that was used to map the tabular data columns to Wikidata properties, and qualifiers and cell values to Wikidata entities.</li> <li>evaluation benchmark - this contains the outputs of LLM generations for the tasks (a) extracting the number of submitted and accepted papers per each track at a given conference, (b) extraction of organizers with their roles for each conference, (c) extraction of programme committee members with track and their role (member, SPC member), and (d) extraction of important dates or deadlines for each activity (submission, notification, etc.) in each track. </li> </ul> <p>The corresponding source code is available at the <a href="https://github.com/scholarly-wikidata/scholarly-wikidata/">scholary-data repo</a>.</p>
Leveraging LLMs for Efficient Topic Reviews
<p>Our experiment started with the formulation of a specific research topic. From this topic, we designed a precise query to extract relevant papers from the Scopus database. The query was carefully constructed using a combination of keywords related to the literature review and automation.</p> <p>Scopus Query Statement</p> <p>Q1: ( literature AND review OR slr OR overview OR survey OR insights OR screening) AND ( semi-automated OR semi-automation OR automation ) OR ( machine AND<br>learning OR deep AND learning )</p>
LLMs Languages Least Moderated: Testing Cross-National Moderation in the context of the EU and the US Elections on Chatbots
<p>AI Forensics had <a href="https://aiforensics.org/work/bing-chat-elections">previously exposed</a> that Microsoft Copilot's answers to simple election-related questions contained factual errors 30% of the time. In collaboration with Nieuwsuur, we uncovered how chatbots can recommend and support the dissemination of disinformation as a campaign strategy. Following those investigations as well as a request for information from the European Commission, Microsoft and Google introduced “moderation layers" to their chatbots so that they refuse to answer election-related prompts.</p> <p><strong>This dataset was produced as part<span> of project "LLMs: Languages Least Moderated" at the 2024 Digital Methods Summer School and Data Sprint, which AI Forensics facilitated</span> to allow participants to evaluate and compare the effectiveness of these safeguards in different scenarios.</strong> In particular, we investigated the consistency with which electoral moderation was triggered, depending the language of the prompt and the electoral context.</p>
Data and code on the Moral Machine experiment on large language models (LLMs)
<p>As large language models (LLMs) have become more deeply integrated into various sectors, understanding how they make moral judgments has become crucial, particularly in the realm of autonomous driving. This study utilized the Moral Machine framework to investigate the ethical decision-making tendencies of prominent LLMs, including GPT-3.5, GPT-4, PaLM 2, and Llama 2, to compare their responses to human preferences. While LLMs' and humans' preferences such as prioritizing humans over pets and favoring saving more lives are broadly aligned, PaLM 2 and Llama 2, especially, evidence distinct deviations. Additionally, despite the qualitative similarities between the LLM and human preferences, there are significant quantitative disparities, suggesting that LLMs might lean toward more uncompromising decisions, compared to the milder inclinations of humans. These insights elucidate the ethical frameworks of LLMs and their potential implications for autonomous driving.</p>
Data and code on the Moral Machine experiment on large language models (LLMs)
Open the record for dataset details and reuse information.
Data for study - LLMS curricula and syllabuses
<p>The aim of this study is to analyze the ability of the LLM to generate curricula for education programs and syllabuses for specific courses at HEIs level. Four solutions providing LLM functionality were tested: GPT-3.5, GPT-4, Bard and Gemini.</p> <p>The dataset contains the following elements:</p> <ul> <li>comparison of curricula generated by models with existing curricula for two fields of study: Management and Economics (at economic universities in Poland) - Polish and English language versions</li> <li>comparison of syllabuses generated by models with existing syllabuses at economic universities in Poland (4 courses for 2 fields of study) - Polish and English language versions</li> </ul>
LLMs for Automatic Financial Statement Analysis
<p>Financial Accounting provides standards for the recording and valuation of economic elements, facilitating the representation of financial activity in standardized models called Financial Statements, such as the Balance Sheet or the Income Statement. Financial Statements are analyzed using different techniques, including the use of ratios or indicators that reveal vital information for understanding, controlling, and planning the financial management of an organization. While it is feasible to achieve a certain degree of automation in Financial Statement Analysis, especially in the computation of financial ratios, human intervention remains essential to extract conclusions that serve as the basis for decision-making. This exploratory study aims to automate the analysis process through the employment of Large Language Models. Leveraging Financial Statements and indicators calculated automatically as the base knowledge, this study applies these language models to autonomously generate an understandable Financial Statement Analysis.</p>
Model Generation from Requirements with LLMs: an Exploratory Study - Replication Package
<p>This is a replication package for the paper "<span>Model Generation from Requirements </span><span>with LLMs: an Exploratory Study</span>", by Sallam Abualhaija, Chetan Arora, and Alessio Ferrari.</p> <p><strong>Abstract: </strong>Complementing natural language (NL) requirements with graphical models can improve stakeholders’ communication and provide directions for system design. However, creating models from requirements involves manual effort. The advent of generative large language models (LLMs), ChatGPT being a notable example, offers promising avenues for automated assistance in model generation. This paper investigates the reliability of ChatGPT in generating sequence diagrams from NL requirements. Specifically, we conduct a qualitative study examining the sequence diagrams generated by ChatGPT for 28 requirements documents of various types and from different domains. Our study aims to uncover potential issues that emerge in the models generated by ChatGPT, thereby hindering its applicability in practice. Observations have systematically been captured through evaluation logs, and categorized through thematic analysis. Our results indicate that, although the models generally conform to the standard and exhibit a reasonable level of understandability, their correctness with respect to the specified requirements often presents challenges. This issue is particularly pronounced in the presence of requirements smells, such as ambiguity and inconsistency. The insights derived from this study can influence the practical utilization of LLMs in the RE process, and open the door to novel RE-specific prompting strategies targeting effective model generation.</p> <p>The replication package consists of the following folders:</p> <p><strong>logs:</strong> includes the evaluation logs produced by each evaluator</p> <p><strong>original-documents: </strong>includes the original requirements documents used for the evaluation</p> <p><strong>RQ1 - quantitative analysis:</strong> includes the analysis made on the scores given to each model and model variant. It includes five files:</p> <p>- results.csv: numerical results of the evaluation for each criterion<br>- analysis-results.Rmd: R file used to perform the quantitative analysis (requires R Studio to be executed)<br>- analysis-results.html: html file produced by analysis-results.Rmd<br>- cross-check.csv: file with the cross-checking of the two assessors applied to a subset of the models<br>- symmary_results.xlsx: final output of the quantitative results in terms of Wilcoxon signed rank tests</p> <p><strong>RQ2 - thematic analysis: </strong>includes the codebook produced by the thematic analysis of the issues in generating models with ChatGPT</p>
Dataset for paper: The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews
Open the record for dataset details and reuse information.
Accountability in Code Review: The Role of Intrinsic Drivers and the Impact of LLMs
<p>The replication package for the paper "<span>Accountability in Code Review: The Role of Intrinsic Drivers and the Impact of LLMs</span>".</p> <p>Accountability is an innate part of social systems. It maintains stability and ensures positive pressure on individuals' decision-making. As actors in a social system, software developers are accountable to their team and organization for their decisions. However, the drivers of accountability and how it changes behavior in software development are less understood. In this study, we look at how the social process of code review affects software engineers' sense of accountability for code quality. Since code review is increasingly involving LLM assistance, we also evaluate the impact on accountability when introducing LLM-assisted code reviews. We carried out a two-phased sequential qualitative study ($\textbf{interviews} \rightarrow \textbf{focus groups}$). In Phase I (16 interviews), we sought to investigate the intrinsic drivers of software engineers influencing their sense of accountability for code quality, relying on self-reported claims. In Phase II, we tested these traits in a more natural setting by simulating traditional peer-led reviews with focus groups and then LLM-assisted review sessions. We found that there are four key intrinsic drivers of accountability for code quality: personal standards, professional integrity, pride in code quality, and maintaining one's reputation. In a traditional peer-led review, we observed a transition from \emph{individual} to \emph{collective accountability} when code reviews are initiated. We also found that the introduction of LLM-assisted reviews disrupts this accountability process, challenging the reciprocity of accountability taking place in peer-led evaluations, i.e., one cannot be accountable to an LLM. Our findings imply that the introduction of AI into SE must preserve social integrity and collective accountability mechanisms.</p>
Dateset for "The Political Preferences of LLMs"
<p>results.rar: contains the results of administering 11 political orientation tests to 24 conversational LLMs + 5 base models + 3 politically aligned LLMs.</p> <p>.log files contain the logs during automated test administration </p> <p>.png files contain a screen capture of LLM test results</p> <p>.json files contain the LLM test results in JSON format</p> <p>.jsonl files contain all the LLM responses to test questions</p> <p>stance detection.rar: annotation samples for stance detection accuracy estimation</p> <p>finetuning_politically_aligned_models.rar: data for fine tuning politically aligned models</p> <p>tabulated_results.csv contains the mean and standard deviation of test results across test retakes (n=10) for all models. </p>
Supplementary Material for "Investigating Software Development Teams Members' Perceptions of Data Privacy in the Use of Large Language Models (LLMs)"
<h3>ABSTRACT<strong>: </strong></h3> <p><strong>Context</strong>: Large Language Models (LLMs) have revolutionized natural language generation and understanding. However, they raise significant data privacy concerns, especially when sensitive data is processed and stored by third parties. <br><strong>Goal</strong>: This paper investigates the perception of software development teams members regarding data privacy when using LLMs in their professional activities. Additionally, we examine the challenges faced and the practices adopted by these practitioners. <br><strong>Method</strong>: We conducted a survey with 78 ICT practitioners from five regions of the country. <br><strong>Results</strong>: Software development teams members have basic knowledge about data privacy and LGPD, but most have never received formal training on LLMs and possess only basic knowledge about them. Their main concerns include the leakage of sensitive data and the misuse of personal data. To mitigate risks, they avoid using sensitive data and implement anonymization techniques. The primary challenges practitioners face are ensuring transparency in the use of LLMs and minimizing data collection. Software development teams members consider current legislation inadequate for protecting data privacy in the context of LLM use. <br><strong>Conclusions</strong>: The results reveal a need to improve knowledge and practices related to data privacy in the context of LLM use. According to software development teams members, organizations need to invest in training, develop new tools, and adopt more robust policies to protect user data privacy. They advocate for a multifaceted approach that combines education, technology, and regulation to ensure the safe and responsible use of LLMs.</p>
Evaluation Results of English / Japanese LLMs Using Swallow-Evaluation ver.202407
<h1>Evaluation Results of English / Japanese LLMs Using Swallow-Evaluation ver.202407</h1> <p>This dataset is the source material of our observational analysis paper, "Significance and Effectiveness of Training LLM with Japanese Texts [Saito+, 2024]." It includes the evaluation results of 35 LLMs on 19 Japanese and English tasks, using <a href="https://github.com/swallow-llm/swallow-evaluation">Swallow-evaluation</a> ver.202407.</p> <p>As part of the Swallow project at <a href="https://www.titech.ac.jp/english">Tokyo Institute of Technology</a>, this dataset was developed to enable rigorous and comprehensive comparison of Japanese and English LLMs developed in Japan and worldwide.</p> <h2>Details</h2> <h3>Tasks</h3> <p>Evaluation experiments are conducted on LLMs using 10 datasets for Japanese language understanding and generation tasks, and 9 datasets for English language understanding and generation tasks. All evaluation scores are normalized within a range from 0 (lowest) to 1 (highest). Refer to the reference [Saito+, 2024] for the complete list of evaluation tasks and datasets, evaluation metrics, and task configurations.</p> <h3>Environment</h3> <p>The evaluaitons were primarily conducted on A100 nodes (AIST), using Python as the programming language.</p> <h3>Limitation</h3> <p>While efforts were made to evaluate under fair conditions, considering the unique specifications of each LLM (such as tokenization and system prompts), minor differences in evaluation specifics (like prompt formatting and dependence on eval. environment) may cause task evaluation scores to change independently of the LLM’s performance.</p> <h2>Reference</h2> <p>```<br>@techreport{<br> weko_238505_1,<br> author = "齋藤,幸史郎 and 水木,栄 and 大井,聖也 and 中村,泰士 and 塩谷,泰平 and 前田,航希 and Youmi,Ma and 服部,翔 and 藤井,一喜 and 岡本,拓己 and 石田,茂樹 and 高村,大也 and 横田,理央 and 岡崎,直観",<br> title = "LLMに日本語テキストを学習させる意義",<br> booktitle = "研究報告自然言語処理(NL)",<br> pages = "1--15",<br> year = "2024",<br> institution = "東京工業大学, 東京工業大学/産業技術総合研究所, 東京工業大学, 東京工業大学, 東京工業大学, 東京工業大学, 東京工業大学, 東京工業大学, 東京工業大学, 東京工業大学, 東京工業大学, 産業技術総合研究所, 東京工業大学, 東京工業大学",<br> number = "12",<br> month = "aug" <br>}<br>```</p> <h2>License</h2> <p>This dataset is licensed under a <a href="http://creativecommons.org/licenses/by-sa/4.0/" rel="nofollow">Creative Commons Attribution-ShareAlike 4.0 International License</a>.</p> <h2>Creators</h2> <p>Swallow LLM (<a href="https://github.com/swallow-llm">GitHub</a>, <a href="https://swallow-llm.github.io/index.en.html">Official Web Page</a>)</p>
Results and log of LLM-KG-Bench runs described in article "Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?", Frey et al. 2023
<p>Results and log of LLM-KG-Bench runs described in article ""Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?", Frey et al. 2023, to appear in proceedings for workshop DL4KG@ISWC 2023.</p> <p>For data on task FactExtractStatic please contact authors.</p>
Topology of phenomenological experience: leveraging LLMs and feature similarity
<p>08/28/2023 Presented at the Monash Neuroscience of Consciousness (MoNoC) lab</p> <p>Title: Topology of phenomenological experience: leveraging LLMs and feature similarity</p> <p>https://www.youtube.com/watch?v=Xj5Ww3tCfY4</p> <p>See interactive graphs here: <a href="https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqblRSdVlPeEtOdnhWMVJsaUJCOUJ3cDdMNloyQXxBQ3Jtc0tsSUc4cHJUZ1E2Y19ndmU5Y3VRSFg1Ny1RV3JnQUhUa1Q0QWcwOVFBcmV2R0FFeVdwZ1ZjQ3VlN0tXVVV6OXYyd3JCZDBPRlBOc1VTUk9ib1pNM043S1l3UFgzOHFyalRkX01kNjk1U0d3ZUZpTkVmOA&q=https%3A%2F%2Fyoungzielee.github.io%2F&v=Xj5Ww3tCfY4">https://youngzielee.github.io/</a></p>
Manually Annotated Event Log of Users Prompts in LLMs for Conceptual Modeling
<p>This repository contains the supplementary material for our paper at ER 2024 Conference. </p> <p>The data contains the results of an empirical study with 76 undergraduate information systems students. The students submitted the course assignments in 39 groups (of one or two students). The assignment used for the study required use case modeling with UML use case diagrams and domain modeling with UML class diagrams. The groups were first expected to interact with an LLM and then, if needed, to manually improve their models. Groups were randomly assigned to interact with either GPT 4.0 or Code Llama 34B Instruct in one of three application domains. </p> <p>The participants were instructed to engage with the LLM until they were satisfied with the results or opted to skip further refinement. The interaction log contains the following fields: User ID, Input (the user prompt), Response (the modeling artifacts), and the Prompt Number (within user ID).</p>
Supplemental data: Benchmarking of Exomiser and seven LLMs (GPT o1 preview, GPT o1 mini, GPT-4o, Gemini Flash 2.0, Meditron-70B, Meditron3-70B, and Medfound-175B) for differential diagnosis using phenopackets
Open the record for dataset details and reuse information.
Artifact for "Program Repair and LLMs for Intelligent Tutoring of Programming"
<p>This is the supplementary material for our work on "Program Repair and LLMs for Intelligent Tutoring of Programming". The corresponding paper has been submitted to the Technical Symposium on Computer Science Education (SIGCSE TS) 2025.</p> <p><span>Our material includes (1) the artifacts for our </span><strong><span>user evaluat</span><span>ions</span></strong><span> and </span><span>(beyond what we describe in our paper) (2) </span><span>additional information on the </span><strong><span>student </span><span>projects</span></strong><span> </span><span>used in our software engineering course. In this course, third-year CS students focus on developing the intelligent tutoring system.</span></p> <p><strong>1. Artifacts for the <em>User Evaluations</em></strong></p> <p>The <code>user_evaluation/students</code> folder includes the main artifacts for the conducted experiments with the first-year CS students:</p> <ul> <li><code>students_questionnaire_part1.pdf</code> -- It shows the PDF version of the web form of our survey for students before they solve the programming assignment.</li> <li><code>students_questionnaire_part2_groupA.pdf</code> -- It shows the PDF version of the web form of our survey for group A students after they solve programming assignments with ITS.</li> <li><code>students_questionnaire_part2_groupB.pdf</code> -- It shows the PDF version of the web form of our survey for group B students after they solve the programming assignments without ITS.</li> <li><code>students_responses.xlsx</code> -- It shows all students' responses.</li> </ul> <p>The <code>user_evaluation/tutors</code> folder includes the main artifacts for the conducted surveys with the CS tutors:</p> <ul> <li><code>tutors_questionnaire_part1.pdf</code> -- It shows the PDF version of the web form of our survey for tutors before they grade students' programming assignments.</li> <li><code>tutors_questionnaire_part3.pdf</code> -- It shows the PDF version of the web form of our in-person interview for tutors after they grade students' programming assignments.</li> <li><code>tutors_responses.xlsx</code> -- It shows all tutors' responses.</li> </ul> <p><br><strong>2. Course Project Descriptions</strong></p> <p>The <code>projects</code> folder includes the main artifacts for the conducted experiments with the first-year CS students:</p> <ul> <li><code>2022_project.pdf</code> -- It includes the project descriptions, which have been provided to the third-year students in the first instance of our Software Engineering course.</li> <li><code>2023_project.pdf</code> -- It includes the project descriptions, which have been provided to the third-year students in the second instance of our Software Engineering course.</li> <li><code>project_schedule.pdf</code> -- It includes the overview of the assignments in our course illustrating the project's overall schedule.</li> </ul>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.