Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
76
datasets available to search
ShareScore release 0.7.1
Dataset results
76 results for “LLM”
Invasion Biology WikiProject Scientific Papers: Text Data Mining and LLM-based Information Extraction of Species, Locations, Habitats, and Ecosystems
<p>This dataset contains the abstract and full-text for publication DOIs from the Invasion Biology WikiProject (DOI: <a href="https://www.doi.org/10.5281/zenodo.12518036">10.5281/zenodo.12518036</a>). The data was retrieved using the <a href="https://ask.orkg.org/">ask.orkg.org</a> <a href="https://api.ask.orkg.org/docs#tag/Semantic-Neural-Search/operation/explore_documents_index_explore_get">API</a>. For the <a href="https://github.com/jd-coderepos/invasion-biology-IE/blob/main/scripts/ask-doi-list-fulltext-search.py">script</a> used to obtain the data, refer to the accompanying GitHub repository: <a href="https://github.com/jd-coderepos/invasion-biology-IE/" target="_blank" rel="noopener">https://github.com/jd-coderepos/invasion-biology-IE/</a>.</p> <p>The resulting CSV file includes the following fields: <code>"ASK ID"</code>, <code>"DOI"</code>, <code>"Title"</code>, <code>"Abstract"</code>, and <code>"Full-text"</code>.</p> <p>Of the 49,438 queried DOIs, the ASK database provided:</p> <ul> <li><strong>Total DOIs processed:</strong> 12,636</li> <li><strong>DOIs with neither abstract nor full-text:</strong> 36 (abstract token count was less than 10)</li> <li><strong>DOIs with abstracts but no full-text:</strong> 12,636</li> <li><strong>DOIs with both abstract and full-text:</strong> 2,834</li> </ul> <p>The second part of the dataset contains structured information extracted from the publications using the GPT-4o Large Language Model. This structured data is included in the zipped folder <code>structured-publications.zip</code>.</p> <p>The accompanying GitHub repository provides access to the code and scripts used at various stages of the information extraction (IE) process.</p> <p><strong>Theme of the Study:</strong><br>"Mining for Species, Locations, Habitats, and Ecosystems from Scientific Papers in Invasion Biology: A Large-Scale Exploratory Study with Large Language Models."</p>
Curlie Enhanced with LLM Annotations: Two Datasets for Advancing Homepage2Vec's Multilingual Website Classification
<h3>Advancing Homepage2Vec with LLM-Generated Datasets for Multilingual Website Classification</h3> <p>This dataset contains two subsets of labeled website data, specifically created to enhance the performance of Homepage2Vec, a multi-label model for website classification. The datasets were generated using Large Language Models (LLMs) to provide more accurate and diverse topic annotations for websites, addressing a limitation of existing Homepage2Vec training data.</p> <p><strong>Key Features:</strong></p> <ul> <li><strong>LLM-generated annotations:</strong> Both datasets feature website topic labels generated using LLMs, a novel approach to creating high-quality training data for website classification models.</li> <li><strong>Improved multi-label classification:</strong> Fine-tuning Homepage2Vec with these datasets has been shown to improve its macro F1 score from 38% to 43% evaluated on a human-labeled dataset, demonstrating their effectiveness in capturing a broader range of website topics.</li> <li><strong>Multilingual applicability:</strong> The datasets facilitate classification of websites in multiple languages, reflecting the inherent multilingual nature of Homepage2Vec.</li> </ul> <p><strong>Dataset Composition:</strong></p> <ul> <li><strong>curlie-gpt3.5-10k:</strong> 10,000 websites labeled using GPT-3.5, context 2 and 1-shot</li> <li><strong>curlie-gpt4-10k:</strong> 10,000 websites labeled using GPT-4, context 2 and zero-shot</li> </ul> <p><strong>Intended Use:</strong></p> <ul> <li>Fine-tuning and advancing Homepage2Vec or similar website classification models</li> <li>Research on LLM-generated datasets for text classification tasks</li> <li>Exploration of multilingual website classification</li> </ul> <p><strong>Additional Information:</strong></p> <ul> <li><strong>Project and report repository:</strong> https://github.com/CS-433/ml-project-2-mlp</li> </ul> <p><strong>Acknowledgments:</strong></p> <p>This dataset was created as part of a project at EPFL's Data Science Lab (DLab) in collaboration with <a href="https://people.epfl.ch/robert.west">Prof. Robert West</a> and <a href="https://tizianopiccardi.github.io/" rel="nofollow">Tiziano Piccardi.</a></p>
LLM generated Python Compiler Test Dataset
<p>This dataset is generated by integrating Large Language Models (LLMs) with AFL++ fuzzing to enhance compiler testing for CPython. It includes original Python test scripts created by LLMs such as Mistral 7B, Codellama 7B, and Gemma 7B, targeted at various compiler functionalities. These scripts were subjected to fuzzing, resulting in a rich collection of test cases that tests potential vulnerabilities. An optional minimization process with AFL-cmin refined the dataset, ensuring it focuses on test cases that significantly contribute to code coverage and bug discovery. This dataset serves as a valuable resource for improving compiler design and testing efficiency, supporting further research and development in AI-driven software testing methods.</p> <p>please see references for citations of software used in this development</p>
LLM Research Repository
<p><strong>Overview</strong></p> <p>Welcome to the Large Language Models (LLM) Repository, a curated collection aimed at researchers, practitioners, and enthusiasts in the field of Natural Language Processing (NLP). This repository offers resources related to Large Language Models, including research papers, theses, tools, datasets, courses, open-source models, and benchmarks. </p> <p><strong>1. Research Papers</strong></p> <p>A compilation of seminal and cutting-edge research papers that shape the field of Large Language Models. This section includes:</p> <ul> <li>Foundational Papers: Groundbreaking papers that laid the framework for LLM research.</li> <li>Recent Advances: Latest research examining novel architectures, training techniques, and applications.</li> <li>Survey and Review Articles: Comprehensive surveys and reviews that aggregate findings and offer insightful analysis on various aspects of LLMs.</li> </ul> <p><strong>2. Theses</strong></p> <p>A collection of master's and doctoral theses that focus on various facets of LLMs, providing in-depth explorations of core concepts, novel methodologies, and empirical studies from all over the world.</p> <p><strong>3. Tools</strong></p> <p>A list of tools and libraries essential for working with Large Language Models. This section encompasses:</p> <ul> <li>Development Frameworks: Popular libraries and frameworks.</li> <li>Utilities: Tools for data preprocessing, model deployment, and inference acceleration.</li> </ul> <p><strong>4. Datasets</strong></p> <p>A collection of datasets tailored for training and evaluating Large Language Models. This section includes:</p> <ul> <li>Text Corpora: Large-scale text datasets from diverse domains such as news articles, books, and social media.</li> <li>Annotated Datasets: Datasets with human annotations for tasks such as named entity recognition, sentiment analysis, and machine translation.</li> </ul> <p><strong>5. Courses</strong></p> <p>A list of university courses related to Large Language Models.</p> <p><strong>6. Open Source Models</strong></p> <p>Access to state-of-the-art open-source Large Language Models, allowing you to leverage pre-trained models for various applications. This section includes:</p> <ul> <li>Model Repositories: Links to GitHub repositories and model zoos hosting popular LLMs such as GPT, BERT, T5, and their variants.</li> <li>Pre-trained Models: Ready-to-use models available through platforms like the Hugging Face Model Hub, including detailed usage instructions and licensing information.</li> <li>Customized Implementations: Specialized versions and fine-tuned models tailored for specific tasks or domains.</li> </ul> <p><strong>7. Benchmarks</strong></p> <p>A suite of benchmarks designed to evaluate the performance and robustness of Large Language Models. This section features:</p> <ul> <li>Standard Benchmarks: Widely-accepted benchmarks like GLUE, SuperGLUE, and the LAMBADA dataset.</li> <li>Challenge Sets: Curated datasets that test specific capabilities of models, such as commonsense reasoning, multilingual understanding, or adversarial robustness.</li> </ul>
AstroChat - A Dataset of synthetically generated conversations for LLM supervised fine-tuning in the domain of Space Mission Engineering and Astronautics
<h1>AstroChat Dataset Description</h1> <h2>Purpose and Scope</h2> <p>The AstroChat dataset is a collection of 901 dialogues, synthetically generated, tailored to the specific domain of Astronautics / Space Mission Engineering. This dataset will be frequently updated following feedback from the community. If you would like to contribute, please reach out in the community discussion.</p> <h2>Intended Use</h2> <p>The dataset is intended to be used for supervised fine-tuning of chat LLMs (Large Language Models). Due to its currently limited size, you should use a pre-trained instruct model and ideally augment the AstroChat dataset with other datasets in the area of (Science Technology, Engineering and Math).</p> <h2>DATASET DESCRIPTION</h2> <h3>Access</h3> <ul> <li>Manual download from Hugging face hub: <a href="https://huggingface.co/datasets/patrickfleith/Astro-Ultrachat" rel="nofollow">https://huggingface.co/datasets/patrickfleith/AstroChat</a></li> <li>Or with python:</li> </ul> <pre><code>from datasets import load_dataset dataset = load_dataset("patrickfleith/AstroChat") </code></pre> <h3>Structure</h3> <p>901 generated conversations between a simulated user and AI-assistant (more on the generation method below). Each instance is made of the following field (column):</p> <ul> <li><strong>id</strong>: a unique identifier to refer to this specific conversation. Useeful for traceability purposes, especially for further processing task or merge with other datasets.</li> <li><strong>topic</strong>: a topic within the domain of Astronautics / Space Mission Engineering. This field is useful to filter the dataset by topic, or to create a topic-based split.</li> <li><strong>subtopic</strong>: a subtopic of the topic. For instance in the topic of <code>Propulsion</code>, there are subtopics like <code>Injector Design</code>, <code>Combustion Instability</code>, <code>Electric Propulsion</code>, <code>Chemical Propulsion</code>, etc.</li> <li><strong>persona</strong>: description of the persona used to simulate a user</li> <li><strong>opening_question</strong>: the first question asked by the user to start a conversation with the AI-assistant</li> <li><strong>messages</strong>: the whole conversation messages between the user and the AI assistant in already nicely formatted for rapid use with the transformers library. A list of messages where each message is a dictionary with the following fields: <ul> <li><strong>role</strong>: the role of the speaker, either <code>user</code> or <code>assistant</code></li> <li><strong>content</strong>: the message content. For the assistant, it is the answer to the user's question. For the user, it is the question asked to the assistant.</li> </ul> </li> </ul> <p><strong>Important</strong> See the full list of topics and subtopics covered below.</p> <h3>Metadata</h3> <p>Dataset is version controlled and commits history is available here: <a href="https://huggingface.co/datasets/patrickfleith/Astro-Ultrachat/commits/main" rel="nofollow">https://huggingface.co/datasets/patrickfleith/AstroChat/commits/main</a></p> <h3>Generation Method</h3> <p>We used a method inspired from Ultrachat dataset. Especially, we implemented our own version of Human-Model interaction from <strong>Sector I: Questions about the World</strong> of their paper:</p> <p><em>Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., ... & Zhou, B. (2023). Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233.</em></p> <h4>Step-by-step description</h4> <ul> <li>Defined a set of user persona</li> <li>Defined a set of topics/ disciplines within the domain of Astronautics / Space Mission Engineering</li> <li>For each topics, we defined a set of subtopics to narrow down the conversation to more specific and niche conversations (see below the full list)</li> <li>For each subtopic we generate a set of opening questions that the user could ask to start a conversation (see below the full list)</li> <li>We then distil the knowledge of an strong Chat Model (in our case ChatGPT through then api with <code>gpt-4-turbo</code> model) to generate the answers to the opening questions</li> <li>We simulate follow-up questions from the user to the assistant, and the assistant's answers to these questions which builds up the messages.</li> </ul> <h3>Future work and contributions appreciated</h3> <ul> <li>Distil knowledge from more models (Anthropic, Mixtral, GPT-4o, etc...)</li> <li>Implement more creativity in the opening questions and follow-up questions</li> <li>Filter-out questions and conversations which are too similar</li> <li>Ask topic and subtopic expert to validate the generated conversations to have a sense on how reliable is the overall dataset</li> </ul> <h3>Languages</h3> <p>All instances in the dataset are in english</p> <h3>Size</h3> <p>901 synthetically-generated dialogue</p> <h2>USAGE AND GUIDELINES</h2> <h3>License</h3> <p>AstroChat © 2024 by Patrick Fleith is licensed under Creative Commons Attribution 4.0 International</p> <h4>Restrictions</h4> <p>No restriction. Please provide the correct attribution following the license terms.</p> <h4>Citation</h4> <p><em>Patrick Fleith, AstroChat – A Dataset of synthetically generated conversations for LLM supervised fine-tuning in the domain of Space Mission Engineering and Astronautics, (2024).</em></p> <h4>Update Frequency</h4> <p>Will be updated based on feedbacks. I am also looking for contributors. Help me create more datasets for Space Engineering LLMs :)</p> <h4>Have a feedback or spot an error?</h4> <p>Use the community discussion tab directly on the huggingface AstroChat dataset page.</p> <h4>Contact Information</h4> <p>Reach me here on the community tab or on LinkedIn (Patrick Fleith) with a Note.</p> <h3>Number of conversation per topic category</h3> <pre><code>Space Propulsion Systems 135 Human Spaceflight 50 Entry Descent and Landing (EDL) 45 Mechanisms 45 Planetary Rovers 45 Attitude Determination and Control 45 Telecommunication 41 Space Business 40 Structures 40 Materials 40 Launchers, Launches, Launch Operations 36 Power System 35 Payload S/S and Optics 35 Reliability, Availability, Maintainability, and Safety (RAMS) 35 Space Missions Operations 31 Space Environment 30 Command and Data System 30 Orbital Mechanics 30 Space Law 26 Ground Systems 25 Thermal Control 25 Space Processes 20 Planetary Science and Exploration 17 </code></pre> <h3>Topics and subtopics covered</h3> <p>topic: [ Space Law ]</p> <p>subtopics:</p> <ul> <li>Space Law Basics</li> <li>1998 ISS agreement</li> <li>Outer Sppace Treaty</li> <li>Geostationary Orbit Regulations</li> <li>Space Traffic Management</li> <li>French Space Law</li> </ul> <p>topic: [ Space Business ]</p> <p>subtopics:</p> <ul> <li>New Space</li> <li>Satellite Insurance</li> <li>Financing Space Project (in EU)</li> <li>Commercial Satellite Launch Services</li> <li>Space Tourism</li> <li>Business Models for Space Stations</li> <li>Public-private Partnerships</li> <li>Economic Impact of Space Technologies</li> </ul> <p>topic: [ Space Missions Operations ]</p> <p>subtopics:</p> <ul> <li>Flight control team</li> <li>Flight Dynamics</li> <li>Procedure Preparation and Validation</li> <li>Mission Planning</li> <li>Extravehicular Activities (EVAs)</li> <li>Collision Avoidance Manoeuvres</li> <li>Mission Termination and De-Orbit Strategies</li> </ul> <p>topic: [ Human Spaceflight ]</p> <p>subtopics:</p> <ul> <li>Astronaut Selection</li> <li>Astronaut Training</li> <li>research experiments onboard of the ISS</li> <li>Human Mission to Mars Design</li> <li>Environmental Control and Life Support Systems</li> <li>Moon Surface Habitats</li> <li>Microgravity effects</li> <li>Space Suit Design and Operation</li> <li>Space Medicine</li> <li>Space Food</li> </ul> <p>topic: [ Space Environment ]</p> <p>subtopics:</p> <ul> <li>Micrometeorites</li> <li>Space Radiation</li> <li>Solar Cycle</li> <li>Spacecraft Hardening</li> <li>Space Environment Effects on Satellites</li> <li>Magneto-sphere and Radiation Belt</li> </ul> <p>topic: [ Space Propulsion Systems ]</p> <p>subtopics:</p> <ul> <li>Liquid Rocket Engines</li> <li>Solid Rocket Motors</li> <li>Hybrid Rocket Engines</li> <li>Staging and Ignition Systems</li> <li>Propellant Feed Systems</li> <li>Nozzle Designs</li> <li>Thermodynamics</li> <li>Turbopumps and/or Combustion Chambers</li> <li>Specific Impulse and Thrust-to-Weight Ratios</li> <li>Chemical Monopropellant Technologies</li> <li>Chemical Bipropellant Systems</li> <li>Nuclear Thermal Propulsion</li> <li>Fuel Handling and Storage</li> <li>Nuclear Propulsion Thermal Neutron Absorbers</li> <li>Nuclear Propulsion Heat Exchangers</li> <li>Green Propellants</li> <li>Bipropellant Injector Design</li> <li>Electric Ion Thrusters</li> <li>Hall Effect Thrusters</li> <li>Electrothermal Thrusters</li> <li>Grid and Cathode Technologies</li> <li>Aerospike Engines</li> <li>Variable Specific Impulse Magnetoplasma Rocket (VASIMR)</li> <li>Bipropellant Mixing Ratios and Combustion</li> <li>Cryogenic Propellant Handling</li> <li>Oxydizer and Fuel Combinations</li> <li>Long-term Impacts of Propellant Residues in the Atmosphere</li> <li>Propellant Tank Pressurization</li> </ul> <p>topic: [ Space Processes ]</p> <p>subtopics:</p> <ul> <li>Trade Studies</li> <li>Margins, Coningencies, Reserves</li> <li>Systems Engineering</li> <li>Quality Assurance</li> </ul> <p>topic: [ Ground Systems ]</p> <p>subtopics:</p> <ul> <li>Ground Stations</li> <li>Ground Support Equipments</li> <li>Control Centers</li> <li>Tracking Systems</li> <li>AntennasGround Systems Engineering</li> </ul> <p>topic: [ Planetary Rovers ]</p> <p>subtopics:</p> <ul> <li>Mars Rovers</li> <li>Lunar Rovers</li> <li>Rover Instrumentation</li> <li>Rover Power Systems</li> <li>Rover Thermal Control</li> <li>Rover Autonomy</li> <li>Wheels Design</li> <li>Legged Rovers</li> <li>Hazard Avoidance</li> </ul> <p>topic: [ Planetary Science and Exploration ]</p> <p>subtopics:</p> <ul> <li>Astrobiology</li> <li>Exoplanets</li> <li>AsteroidsJupiter</li> <li>Saturn</li> <li>Search for Extraterrestrial Life</li> </ul> <p>topic: [ Structures ]</p> <p>subtopics:</p> <ul> <li>Structural Design and Analysis</li> <li>Load Path Determination</li> <li>Vibration and Acoustic Testing</li> <li>Thermal Protection Systems</li> <li>Composite Structures</li> <li>Joining Techniques (e.g., Welding, Bolting, Bonding)</li> <li>Manufacturing Tolerances and Quality Control</li> <li>Deployable Structures (e.g., Antennas, Solar Arrays)</li> </ul> <p>topic: [ Mechanisms ]</p> <p>subtopics:</p> <ul> <li>Actuators and Dampers</li> <li>Gimbals and Bearings</li> <li>Latch and Release Devices</li> <li>Hinges and Deployment Systems</li> <li>Robotic Arms and Tools</li> <li>Valves and Fluid Control Systems</li> <li>Thermal Expansion Joints</li> <li>Drive Systems and Motors</li> <li>Reliability and Lifetime Analysis</li> </ul> <p>topic: [ Materials ]</p> <p>subtopics:</p> <ul> <li>Composite Materials</li> <li>Metals and Alloys</li> <li>Polymers and Plastics</li> <li>Nano-materials</li> <li>Radiation Shielding Materials</li> <li>Thermal Insulation Materials</li> <li>Corrosion and Oxidation Resistance</li> <li>Material Testing and Characterization</li> </ul> <p>topic: [ Entry Descent and Landing (EDL) ]</p> <p>subtopics:</p> <ul> <li>Aerodynamics and Aeroheating</li> <li>Powered Descent</li> <li>Landing Gear and Systems</li> <li>Heat Shield Design and Materials</li> <li>Hazard Avoidance</li> <li>Surface Interaction (Airbags, Crushable Structures)</li> <li>Entry, Descent, and Landing Sequencing</li> <li>EDL on Mars</li> <li>Parachute Systems Design</li> </ul> <p>topic: [ Reliability, Availability, Maintainability, and Safety (RAMS) ]</p> <p>subtopics:</p> <ul> <li>System Reliability Modeling</li> <li>Failure Modes, Effects, and Criticality Analysis (FMECA)</li> <li>Risk Assessment and Management</li> <li>Safety-Critical Systems Design</li> <li>Availability Modeling and Prediction</li> <li>Lifecycle Cost and Duration Analysis</li> <li>Hazardous Material Handling</li> </ul> <p>topic: [ Orbital Mechanics ]</p> <p>subtopics:</p> <ul> <li>Interplanetary Trajectories</li> <li>Gravity Assist Maneuvers</li> <li>Orbit Determination and Propagation</li> <li>Space Situational Awareness and Debris Tracking</li> <li>Mission Design and Analysis Tools</li> <li>Orbit Decay and Re-entry Predictions</li> </ul> <p>topic: [ Launchers, Launches, Launch Operations ]</p> <p>subtopics:</p> <ul> <li>Launcher Types (e.g., expendable, reusable)</li> <li>Launch Vehicles</li> <li>Launch Sites and Infrastructure</li> <li>Countdown Procedures and Sequencing</li> <li>Launch Window Determination and Trajectory Analysis</li> <li>Ground and Launch Crew Training</li> <li>Payload Integration and Fairing Design</li> <li>Environmental and Weather Constraints</li> </ul> <p>topic: [ Attitude Determination and Control ]</p> <p>subtopics:</p> <ul> <li>Sensors for Attitude Determination (e.g., Gyroscopes, Star Trackers)</li> <li>Actuators for Attitude Control (e.g., Reaction Wheels, Thrusters)</li> <li>Control Algorithms (e.g., PID, Kalman Filter)</li> <li>Momentum Exchange Devices</li> <li>Attitude Dynamics Modeling</li> <li>On-Orbit Attitude Reconfiguration</li> <li>Fault Detection and Response Strategies</li> <li>Sun and Earth Sensors</li> <li>Magnetic Torquers and Gravity Gradient Stabilization</li> </ul> <p>topic: [ Payload S/S and Optics ]</p> <p>subtopics:</p> <ul> <li>Payload Design and Integration</li> <li>Spectral Imaging and Multi-spectral Sensors</li> <li>Infrared and Ultraviolet Optics</li> <li>Calibration and Validation of Optical Systems</li> <li>Image Processing and Data Analysis</li> <li>Thermal Control for Sensitive Optics</li> <li>Data Downlink and Communication Interfaces</li> </ul> <p>topic: [ Power System ]</p> <p>subtopics:</p> <ul> <li>Solar Panels and Arrays</li> <li>Battery Types and Management Systems (e.g., Li-ion, NiMH)</li> <li>Energy Storage Technologies</li> <li>Fault Protection and Isolation</li> <li>Harness and Cabling</li> <li>Alternative Power Sources (e.g., RTGs, Fuel Cells)</li> <li>Power Budgeting and Load Analysis</li> </ul> <p>topic: [ Thermal Control ]</p> <p>subtopics:</p> <ul> <li>Active Thermal Control Systems (e.g., Heat Pumps, Louvers)</li> <li>Environmental Testing and Validation</li> <li>Heating and Cooling Hardware</li> <li>Thermal Protection for Entry, Descent, and Landing</li> <li>Cryogenic Thermal Management</li> </ul> <p>topic: [ Command and Data System ]</p> <p>subtopics:</p> <ul> <li>Onboard Computers and Processing Units</li> <li>Software Architecture and Middleware</li> <li>Command Link and Telemetry Systems</li> <li>Interface and Bus Systems (e.g., MIL-STD-1553, SpaceWire)</li> <li>Real-Time Operating Systems (RTOS)</li> <li>Security Measures and Encryption</li> </ul> <p>topic: [ Telecommunication ]</p> <p>subtopics:</p> <ul> <li>Antenna Systems (e.g., Parabolic, Phased Array)</li> <li>Communication Transponders</li> <li>Frequency Bands and Spectrum Management</li> <li>Signal Modulation and Demodulation Techniques</li> <li>Inter-Satellite Links and Data Relays</li> <li>Error Detection and Correction</li> <li>Space Communication Protocols</li> <li>RF and Microwave Components</li> <li>Deep Space Communications</li> </ul>
Model quality assessment: Results of human ratings, prompts, models as .csv and answers of the LLM
<p>This is the dataset for the paper "<a href="https://doi.org/10.1007/978-3-031-77908-4_7" target="_blank" rel="noopener">Assessing Model Quality Using Large Language Models</a>" and contains the following files of the assessment of the model quality:</p> <ul> <li>Results of the human ratings</li> <li>Prompts</li> <li>Models as .csv</li> <li>Answers of the LLM</li> </ul> <p>The graphical representation of the models is linked under Related works.</p>
LLM Generated Synthetic Dataset of DoS Exposed Solidity Contracts
<p>This dataset provides the replication package for the paper 'Large Language Models for Synthetic Dataset<br>Generation: A Case Study on Ethereum Smart Contract DoS Vulnerabilities' accepted for publication at the 8th International Workshop on Blockchain Oriented Software Engineering. The provided sources encompass:<br>1) The synthetic contracts (Vulnerable, Exploit, and Patched contract for each use case) generated by Claude and GPT4.<br>2) The configuration files of the hardhat-based testing environment.<br>3) The test suite that showcases the vulnerabilities of the generated contracts (including mock contracts) (hardhat is required to run and test contracts).<br><br></p> <div> <p> </p> </div>
USPTO-LLM: A Large Language Model-Assisted Information-enriched Chemical Reaction Dataset
<p>USPTO-LLM is an <strong>information-enriched chemical reaction dataset</strong> that provides more side information (reaction conditions and reaction steps division) for developing new reaction prediction and retrosynthesis methods and inspires new problems, such as reaction condition prediction. It comprises over <strong>247K chemical reactions</strong> extracted from the patent documents of USPTO (United States Patent and Trademark Office), encompassing abundant information on reaction conditions. </p> <p>We employ large language models to expedite the data collection procedures automatically with a reliable quality control process. The extracted chemical reactions are organized as <strong>heterogeneous directed graphs</strong>, allowing us to formulate a series of prediction tasks, such as reaction prediction, retrosynthesis, and reaction condition prediction, in a unified graph-filling framework.</p>
SemTab 24: Semantic Table Annotations Benchmark for LLM-based approaches
<p><strong>SuperSemtab24 </strong>is a dataset for tabular data to knowledge graph matching.</p> <p>The dataset is divided into training and validation sets. The dataset includes general-purpose tables and intentionally misspelled entities to evaluate the model's robustness. Participants must annotate the entity mentions in the validation set and submit their annotations (following a target file).</p> <p>The repository contains the full version of the dataset; the ground truth (GT) of the test set will be uploaded in the future.</p>
LLM Service Outages and Incident Reports
<p>This dataset provides outage data and incident data of 3 LLM providers (OpenAI, Anthropic, Character.AI) across 8 LLM services (OpenAI API, ChatGPT, DALLE, Playground, Anthropic API, Claude, Console, Character.AI) collected until 2024-08-31.<br><br>Data sources:</p> <ul> <li>Outage: https://status.{service_provider}.com/uptime</li> <li>Incident Reports: https://status.{service_provider}.com/history</li> </ul> <p>service_provider = [openai, anthropic, character.ai]</p> <p>Documents:</p> <ul> <li>./raw_data/*: Raw data for outages and incidents, stored by service.</li> <li>./clean_data/*: Clean-up data for characterization, aggregated of all services.</li> </ul> <p> </p>
Results and log of LLM-KG-Bench runs described in article "Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering", Meyer et al. 2023
<p>Results and logs of <a href="https://github.com/AKSW/LLM-KG-Bench">LLM-KG-Bench</a> runs described in article "Developing a Scalable Benchmark for Assessing Large Language Models in Knowledge Graph Engineering", Meyer et al., to appear in <a href="https://2023-eu.semantics.cc/page/accepted_posters">SEMANTICS 2023 poster track</a> proceedings.</p>
EvalQuiz - LLM-based Automated Generation of Self-Assessment Quizzes in Software Engineering Education
<p>Self-assessment quizzes after lectures, educational videos, or chapters are a commonly used method in software engineering (SE) education to give students the opportunity to test their gained knowledge. However, the creation of these quizzes is time-consuming, cognitively exhausting, and complex, as an expert in the field needs to create the quizzes and review the lecture material for validity. Therefore, this paper presents a concept to automatically generate self-assessment quizzes based on lecture material using a large language model (LLM) to reduce lecturers' workload and simplify the general quiz creation process. The developed prototype was handed to experts, who subsequently evaluated the approach. The results show that automatic quiz generation saves time and the quizzes cover the delivered lecture material well. However, the generated quizzes often lack originality and versatility. Therefore, further prompt engineering might be required to achieve more elaborate results.</p>
LLM-SAT-Eval
<p>Exploring the theme “Do static analysis tools foresee actual bug fixes?," this poster investigates the practical utility of Static Analysis Tools (SATs) in anticipating genuine bug fixes within software code-bases. Leveraging a dataset comprising bug-fix pairs, our study scrutinizes the efficacy of SATs through quantitative analysis, revealing a significant disparity between SAT outputs and actual bug fixes, with nearly 93% of pairs rated at 0. Employing qualitative insights from Gemini-pro Large Language Model, we corroborate these findings, observing prevalent terms such as "buggy line" and "not able" in bug detection. Furthermore, correlation analysis showcases a moderate positive correlation (0.3) between SAT outputs and known bug fixes, emphasizing the importance of alignment for favorable ratings. Our findings underscore the pressing need for enhanced bug detection methodologies, prompting further research in software quality assurance.</p>
Beyond the Hype: Benchmarking LLM-Evolved Heuristics for Bin Packing - Data
<p>This repository contains the data and additional information for the paper 'Beyond the Hype: Benchmarking LLM-Evolved Heuristics for Bin Packing'. </p> <p>The following files are included:</p> <ul> <li>heatmaps.zip: heatmaps showing heuristic performance per dataset for 3 metrics;</li> <li>heuristic_perf.zip: raw results files for each datasets;</li> <li>tuning.zip: tuning log files;</li> <li>ts_features_evolved.csv and ts_features_evolved_10_features_excess_bins.csv : full and selected time-series features computed on the evolved datasets;</li> <li>BPinstances.csv : raw bin packing instances from all datasets: independent and evolved.</li> </ul>
Artifact of Search+LLM-based Testing for ARM Simulators
<h1>Abstract.</h1> <p>We present and discuss our solution to the problem of testing software simulators, using SearchSYS to test the gem5 VLSI digital circuit simulator, employed by ARM to test their systems. In particular, we focus on the simulation of the ARM silicon chip Instruction Set Architecture (ISA).</p> <p>SearchSYS can create test cases that activate bugs by combining LLMs, fuzzing, and differential testing. Using only LLM, SearchSYS identified 74 test cases that activated bugs. By incorporating fuzzing, this number increased by 93 additional bug-activating cases within 24 hours. Through differential testing, we identified 624 bugs with LLM-generated test cases and 126 with fuzzed test inputs. Out of the total number of bug-activating test cases, 4 unique bugs have been reported and acknowledged by developers. Additionally, we provided developers with a test suite and fuzzing statistics, and open-sourced SearchSYS.</p> <p>We published our artifact in Zenodo https://zenodo.org/records/14724553.</p> <p>To reproduce the evaluation in the paper, ensure your system meets the requirements, and then follow the instructions to reproduce the tables in the results. <strong>Hardware requirements:</strong> You will need an ARM machine, with 80 GB HD free space, and 72 GB RAM (for AFL++ fuzzing) or (else) 16 GB if merely running the differential testing without building gem5 with AFL++ instrumentation. </p> <p> </p> <p>The artifact documentation contains three sections:</p> <p><strong>Section 1:</strong> Software and hardware requirements;</p> <p><strong>Section 2:</strong> documentation of how to build the testing environment for ARM with full instructions on how to reproduce the four tables in our ICSE-SEIP 2025 paper; and</p> <p><strong>Section 3:</strong> a shortened version of the evaluation, to be run within a day or less.</p> <p> </p> <p>We suggest starting with the shortened evaluation, after installing AFL++, gem5 and SearchSYS (Section 1, Section 2.1 - TinyLlama only, and Section 2.2). </p> <p>==</p> <p> </p> <p><strong>ARTIFACT EVALUATION NOTE:</strong></p> <p><strong>The <a href="https://zenodo.org/records/14753604/files/README.md" target="_blank" rel="noopener">README.md</a> file in this artifact is a copy of this document. However, 2 mistakes in Section 3.3 were fixed but have not yet been pushed into the README.md file. Consequentially, it is recommended to follow the text here instead. We plan to push a new README.md file once passing the first stage of the artefact evaluation (fixing any other mistakes the committee will find).</strong></p> <p>==</p> <h1>Additional Resources.</h1> <p>SearchSYS tool can be downloaded here: <a href="https://github.com/karineek/SearchGEM5/commit/a239d3cd787c0f5cd3be49b2f21179ff90df89f3">a239d3c</a> (for this work).</p> <p>To process the fuzzed test inputs, post-fuzzing, the *.sh here can run on the output/default/queue and crash folders.</p> <p>The rest are the data collected during experiments: (1) the Excel file contains the full analysis while (2) *.tar.gz are all outputs from the fuzzing stage. The LLM data is taken from <a href="https://zenodo.org/records/13450472">here</a>. You can follow the instructions <a href="https://zenodo.org/records/13450472">here</a> to install and run SearchSYS for X86.</p> <p> </p> <p><strong>The record here details how to run SearchSYS on ARM machines (arm64-ubuntu-2204 and Red Hat-based system)</strong>.</p> <p> </p> <p>Cite the paper:</p> <table> <tbody> <tr> <td> <pre><code>@inproceedings{Even-Mendoza:2025:ICSE-SEIP, author = {Karine Even-Mendoza and Hector D. Menendez and W.B Langdon and Aidan Dakhama and Justyna Petke and Bobby R. Bruce}, title = {{Search+LLM-based} Testing for {ARM} Simulators}, booktitle = {2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)}, year = {2025}, editor = {}, volume = {}, number = {}, series = {}, pages = {}, address = {Ottawa, Canada}, publisher_address = {}, month = {April 27-May 3}, organisation = {}, publisher = {}, note = {https://conf.researchr.org/track/icse-2025/icse-2025-software-engineering-in-practice#Accepted-Papers}, keywords = {Software System Simulation, SearchSYS, gem5, AFL++, Fuzzing, Fuzz Testing, Differential Testing, Search-Based Software Testing, LLM, Language Models, Ollama, CodeLlama, TinyLlama, Phi2, Llama2, Magicoder, CodeBooga, GPT-3.5-turbo, prompt engineering}, ISBN13 = {}, url = {https://solar.cs.ucl.ac.uk/pdf/ICSE_SEIP_2025___SearchSYS_and_ARM.pdf}, code_url = {https://zenodo.org/records/13450472, https://zenodo.org/records/14724553}, size = {12 pages}, abstract = {In order to aid quality assurance of large complex hardware architectures, system simulators have been developed. However, such system simulators do not always accurately mirror what would happen on a real device. A significant challenge in testing these simulators arises from the complexity of having to model both the simulation and the infinite number of software that could be run on such a device. Our previous work introduced SearchSYS, a testing framework for software simulators. SearchSYS leverages a large language model for initial seed C code generation, which is then compiled, and the resultant binary is fed to a fuzzer. We then use differential testing by running the outputs of fuzzing on real hardware and a system simulator to identify mismatches. We present and discuss our solution to the problem of testing software simulators, using SearchSYS to test the gem5 VLSI digital circuit simulator, employed by ARM to test their systems. In particular, we focus on the simulation of the ARM silicon chip Instruction Set Architecture (ISA). SearchSYS can create test cases that activate bugs by combining LLMs, fuzzing, and differential testing. Using only LLM, SearchSYS identified 74 test cases that activated bugs. By incorporating fuzzing, this number increased by 93 additional bug-activating cases within 24 hours. Through differential testing, we identified 624 bugs with LLM-generated test cases and 126 with fuzzed test inputs. Out of the total number of bug-activating test cases, 4 unique bugs have been reported and acknowledged by developers. Additionally, we provided developers with a test suite and fuzzing statistics, and open-sourced SearchSYS for initial seeds and SearchSYS code, for adjustment to ARM8v and fuzzed seeds.}, }</code></pre> </td> </tr> </tbody> </table> <pre>==</pre> <h1>Experiments.</h1> <div> <div> <div> <div> <div> <div> <p>Ensure your system meets the requirements, then follow the instructions to reproduce the tables in Section 5 (V. Results).</p> <p><strong>Hardware requirements: </strong>You will need an ARM machine, with 80 GB HD free space, and 72 GB RAM (for AFL++ fuzzing) or 16 GB else (just differential testing without building gem5 with AFL++ instrumentation).</p> </div> </div> </div> </div> </div> </div> <h2><strong>1. Requirements:</strong></h2> <h3><strong>1.1 Hardware</strong></h3> <p><strong>Hardware requirements: </strong>You will need an ARM machine, with 80 GB HD free space, and 72 GB RAM (for AFL++ fuzzing) or 16 GB else (just differential testing without building gem5 with AFL++ instrumentation).</p> <table> <tbody> <tr> <td><strong>Requirement</strong></td> <td><strong>Details</strong></td> </tr> <tr> <td>Processor</td> <td>ARM</td> </tr> <tr> <td>HD space</td> <td>At least 80 GB of free disk space </td> </tr> <tr> <td>RAM</td> <td> <p>- 72 GB (for AFL++ fuzzing) </p> <p>- 16 GB (for differential testing without building gem5 with AFL++ instrumentation)</p> </td> </tr> <tr> <td>Operating System</td> <td>Tested on Ubuntu and Red Hat </td> </tr> </tbody> </table> <h3> </h3> <h3><strong>1.2 Software and Packages:</strong></h3> <ul> <li>GCC-11</li> <li>wget, zip, unzip (any version)</li> <li>LLVM-13+ (we tested with LLVM-13, but any newer version should be okay)</li> <li>Python3 (tested with Python 3.8 and 3.10)</li> <li>Other packages:</li> </ul> <pre><code>sudo apt-get update sudo apt-get upgrade sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get -y install gcc-11 g++-11 cpp-11 wget lsb-release gnupg software-properties-common sudo rm /usr/bin/cpp /usr/bin/gcc /usr/bin/g++ /usr/bin/gcov /usr/bin/c++ sudo rm /usr/bin/cc sudo ln -s /usr/bin/cpp-11 /usr/bin/cpp sudo ln -s /usr/bin/gcc-11 /usr/bin/gcc sudo ln -s /usr/bin/gcc-11 /usr/bin/cc sudo ln -s /usr/bin/g++-11 /usr/bin/g++ sudo ln -s /usr/bin/g++-11 /usr/bin/c++ sudo ln -s /usr/bin/gcov-11 /usr/bin/gcov wget https://apt.llvm.org/llvm.sh sudo chmod +x llvm.sh sudo ./llvm.sh 13 sudo ln -s /usr/bin/llvm-config-13 /usr/bin/llvm-config sudo apt-get install -y lld-13 llvm-13-dev clang-13 sudo apt-get install -y lld llvm llvm-dev clang sudo apt-get install -y build-essential python3-dev automake cmake git flex bison libglib2.0-dev libpixman-1-dev python3-setuptools cargo libgtk-3-dev sudo apt-get install -y ninja-build cmake sudo apt-get install -y git make cmake llvm gdb coreutils sudo apt-get install -y gcc-11-plugin-dev sudo apt -y autoremove sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-13 1300 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-13 sudo update-alternatives --install /usr/bin/llvm-config llvm-config /usr/bin/llvm-config-13 1300 LLVM_CONFIG=/usr/bin/llvm-config-13</code></pre> <p>We attached all installed packages in <a href="https://zenodo.org/records/14724553/files/installed_karl.log">installed_karl.log</a> (Red Hat-based system) and <a href="https://zenodo.org/records/14724553/files/installed_m400.log" target="_blank" rel="noopener">installed_m400.log</a> + <a href="https://zenodo.org/records/14724553/files/installed_m400_repeat.log" target="_blank" rel="noopener">installed_m400_repeat.log</a>(Ubuntu/Unix, arm64-ubuntu-2204).</p> <p> </p> <p> </p> <h2>2. Reproduce ICSE-SEIP 2025 Evaluation:</h2> <p>This section contains the full instructions for running SearchSYS on ARM. We include further explanations on how to customise the evaluation or run part of it. This is the full version, you can use the shortened version in Section 3. This evaluation should take around 40 days to reproduce: 7 initial corpora, each fuzzed with AFL++ five times (5 repeats of the fuzzing campaign), and then around 2-3 days for differential testing of the initial corpora and the same for each repetition (and we had 5 repeats of the fuzzing campaign). </p> <p>To reproduce the experiments follow these commands (Unix system):</p> <pre><code>## CREATE A NEW USER (you can also use your own user but then you will need to amend /home/debian and debian, accordingly) ## sudo adduser debian # if needed run: # sudo passwd debian # and set some simple password sudo usermod -aG sudo debian sudo mkdir -p /home/debian sudo usermod -d /home/debian debian su - debian ## GET DATA - 7 set of LLM-generated C programs ## cd /home/debian wget "https://zenodo.org/records/10999115/files/LLM_test_inputs-4FuzzExp2.zip" # Get the data unzip -n LLM_test_inputs-4FuzzExp2.zip -d /home/debian/experiment cd /home/debian/experiment/ sed -i "s:/home/ubuntu/experiment-7/:/home/debian/experiment/:g" */input/*.txt cat /home/debian/experiment/TinyLlama-cmin/input/test_input_574567374417673.txt</code></pre> <h3> </h3> <h3>YOU WILL NEED TO COPY ALL SCRIPTS OF THIS ARTIFACT INTO <code>/home/debian</code></h3> <pre><code>wget "https://zenodo.org/records/14724553/files/1-run-diff-test.sh" wget "https://zenodo.org/records/14724553/files/2-test-input.sh" wget "https://zenodo.org/records/14724553/files/run_AFL_loop_v6.sh" wget "https://zenodo.org/records/14724553/files/Experiment-ARM-24h.sh"</code></pre> <p>We will explain in the next sections when to run each. There is a shorter version of the experiments in Section 3.</p> <h3><br>2.1 Create Initial Data Seedds for ARM Machine:</h3> <pre><code>## COMPILE BINARIES - build the test binaries and tailor the paths in the seeds ## ### Model CodeBooga ### cd /home/debian/experiment/CodeBooga for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf CodeBooga-cmin/binary cp -r CodeBooga/binary CodeBooga-cmin/binary cd /home/debian/experiment/CodeBooga-cmin/input sed -i 's\/CodeBooga/\/CodeBooga-cmin/\g' *txt ### Model Llama ### cd /home/debian/experiment/Llama for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Llama-cmin/binary cp -r Llama/binary Llama-cmin/binary cd /home/debian/experiment/Llama-cmin/input sed -i 's\/Llama/\/Llama-cmin/\g' *txt ### Model Magicoder ### cd /home/debian/experiment/Magicoder for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Magicoder-cmin/binary cp -r Magicoder/binary Magicoder-cmin/binary cd /home/debian/experiment/Magicoder-cmin/input sed -i 's\/Magicoder/\/Magicoder-cmin/\g' *txt ### Model TinyLlama ### cd /home/debian/experiment/TinyLlama for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf TinyLlama-cmin/binary cp -r TinyLlama/binary TinyLlama-cmin/binary cd /home/debian/experiment/TinyLlama-cmin/input sed -i 's\/TinyLlama/\/TinyLlama-cmin/\g' *txt ### Model Phi ### cd /home/debian/experiment/Phi for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Phi-cmin/binary cp -r Phi/binary Phi-cmin/binary cd /home/debian/experiment/Phi-cmin/input sed -i 's\/Phi/\/Phi-cmin/\g' *txt ### Model gpt3.5-new ### cd /home/debian/experiment/gpt3.5-new for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf gpt3.5-new-cmin/binary cp -r gpt3.5-new/binary gpt3.5-new-cmin/binary cd /home/debian/experiment/gpt3.5-new-cmin/input sed -i 's\/gpt3.5-new/\/gpt3.5-new-cmin/\g' *txt ### Model gpt3.5-old ### cd /home/debian/experiment/gpt3.5-old for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf gpt3.5-old-cmin/binary cp -r gpt3.5-old/binary gpt3.5-old-cmin/binary cd /home/debian/experiment/gpt3.5-old-cmin/input sed -i 's\/gpt3.5-old/\/gpt3.5-old-cmin/\g' *txt</code></pre> <p>At this point all data for fuzzing is ready. You need now to get SearchSYS and build the SUT (gem5) from source.</p> <p> </p> <p> </p> <h3>2.2 Build the tools:</h3> <p>We first build AFL with some customisation:</p> <pre><code>## GET AFL++ SearchSYS ## cd /home/debian git clone https://github.com/karineek/SearchGEM5.git git clone https://github.com/AFLplusplus/AFLplusplus.git cd AFLplusplus git checkout f596a297c4de6a5e1a6fb9fbb3b4e18124a24f58 cp ../SearchGEM5/src/gem5-afl/afl-fuzz-init.c src/afl-fuzz-init.c AFL_USE_ASAN=0 make ### The compilation shall state that it is ARM architecture: ### ``` debian@node0:~/AFLplusplus$ AFL_USE_ASAN=0 make [*] Compiling AFL++ for OS Linux on ARCH aarch64 [+] ZLIB detected [!] Note: skipping x86 compilation checks (AFL_NO_X86 set). [+] shmat seems to be working. [+] Python 3.8.10 support seems to be working. [+] Everything seems to be working, ready to compile ... ```</code></pre> <p><br>then you need to check that AFL is built correctly:</p> <pre><code>```<br></code><code>Build Summary: [+] afl-fuzz and supporting tools successfully built [+] LLVM basic mode successfully built [+] LLVM mode successfully built [+] LLVM LTO mode successfully built [+] gcc_mode successfully built<br>```</code></pre> <p>Then build gem5 with AFL++ instrumentation:</p> <pre><code>cd /home/debian git clone https://github.com/BobbyRBruce/gem5-ssbse-challenge-2023.git cd /home/debian/gem5-ssbse-challenge-2023 cp ../SearchGEM5/src/gem5-afl/SConstruct . cp ../SearchGEM5/src/gem5-afl/SConscript src/ # If needed install: sudo apt install python3-pip sudo apt install scons protobuf-compiler libhdf5-dev libgoogle-perftools-dev libsystemc-dev # Then install the requirements for gem5: pip install -r requirements.txt # And build gem5 with AFL++ instrumentation:</code><code><br>CC=/home/debian/AFLplusplus/afl-cc CXX=/home/debian/AFLplusplus/afl-c++ LINK=/home/debian/AFLplusplus/afl-c++ \<br> scons build/ARM/gem5.opt -j 4</code><code> # Then test if built:<br>./build/ARM/gem5.opt -C</code></pre> <p>During the configuration, you need to see it built for ARM, hence (1) make sure you use build/ARM not X86, and check the configuration log to see that the right architecture has been properly selected for the build. Executing gem5.opt should give you the following message: (or something similar)</p> <pre><code>``` debian@node0:~/gem5-ssbse-challenge-2023$ ./build/ARM/gem5.opt -C Please see individual files for details of the license on each file. The preferred license can be found in LICENSE. All files in this distribution (other than in the ext directory) have licenses based on the BSD or MIT licenses. Some files in the ext directory are GNU LGPL. No other licenses are found in this distribution. Beyond the BSD license, some files include the following clarification of the license as required by the copyright holder: The license below extends only to copyright in the software and shall not be construed as granting a license to any other intellectual property including but not limited to intellectual property relating to a hardware implementation of the functionality of the software licensed hereunder. You may use the software subject to the license terms below provided that you ensure that this notice is replicated unmodified and in its entirety in all distributions of the software, modified or unmodified, in source code or in binary form. The copyright holders include (not counting the ext directory): Copyright (c) 2000-2011 The Regents of The University of Michigan Copyright (c) 1990,1993-1995,2007-2010 The Hewlett-Packard Development Company Copyright (c) 1999-2009,2011 Mark D. Hill and David A. Wood Copyright (c) 2009-2011 ARM Limited Copyright (c) 2008-2009 Princeton University Copyright (c) 2007 MIPS Technologies, Inc. Copyright (c) 2009-2011 Advanced Micro Devices, Inc. Copyright (c) 2009 The University of Edinburgh Copyright (c) 2007-2008 The Florida State University Copyright (c) 2010 Massachusetts Institute of Technology Copyright (c) 1990-1993 The Regents of the University of California Copyright (c) 2006-2009 Nathan Binkert Copyright (c) 2001 The NetBSD Foundation, Inc. Copyright (c) 2010-2011 Gabe Black Copyright (c) 1994 Adam Glass Copyright (c) 1990-1992 MIPS Computer Systems, Inc. Copyright (c) 2004 Richard J. Wagner Copyright (c) 2000 Computer Engineering and Communication Networks Lab Copyright (c) 2001 Eric Jackson Copyright (c) 1990 Hewlett-Packard Development Company Copyright (c) 1994-1996 Carnegie-Mellon University. Copyright (c) 1993-1994 Christopher G. Demetriou Copyright (c) 1997-2002 Makoto Matsumoto and Takuji Nishimura Copyright (c) 1998,2001 Manuel Bouyer. Copyright (c) 2016-2017 Google Inc. ```</code></pre> <p>Last step: compile the custom mutators:</p> <pre><code>cd /home/debian/SearchGEM5/src/custom_mutators/ AFL_HOME=/home/debian/AFLplusplus sed -i "s:/home/ubuntu/AFLplusplus:$AFL_HOME:g" compile_share_test.sh sed -i "s:/home/ubuntu/AFLplusplus:$AFL_HOME:g" compile_share.sh ./compile_share.sh 17 84 66 </code></pre> <p>You will get a warning from AFL++ headers, there is nothing to do about it, if you wish you can edit the AFL++ headers and fix it:</p> <pre><code>``` /home/debian/AFLplusplus/src/afl-performance.c: In function ‘get_binary_hash’: /home/debian/AFLplusplus/src/afl-performance.c:103:17: warning: pointer targets in passing argument 1 of ‘open’ differ in signedness [-Wpointer-sign] 103 | int fd = open(fn, O_RDONLY); | ^~ | | | u8 * {aka unsigned char *} In file included from /usr/aarch64-linux-gnu/include/fcntl.h:301, from /home/debian/AFLplusplus/include/afl-fuzz.h:58, from /home/debian/AFLplusplus/src/afl-performance.c:2: /usr/aarch64-linux-gnu/include/bits/fcntl2.h:41:19: note: expected ‘const char *’ but argument is of type ‘u8 *’ {aka ‘unsigned char *’} 41 | open (const char *__path, int __oflag, ...) | ~~~~~~~~~~~~^~~~~~ /home/debian/AFLplusplus/src/afl-performance.c: In function ‘sha1_hex’: /home/debian/AFLplusplus/src/afl-performance.c:407:10: warning: pointer targets in returning ‘u8 *’ {aka ‘unsigned char *’} from a function with return type ‘char *’ differ in signedness [-Wpointer-sign] 407 | return hex; | ^~~ ```</code></pre> <p>At this point, we have the tools up and the experimental data ready for fuzzing and testing.</p> <p> </p> <p> </p> <h3>2.3 Differential Testing of Input Corpus Seeds (Section V.A. & Table II):</h3> <p>Testing is usually against the most recent version. To get gem5 built from tree, run this code:</p> <pre><code>cd /home/debian git clone https://github.com/gem5/gem5.git cd gem5 pip install -r requirements.txt scons build/ARM/gem5.opt -j 4<br>cd /home/debian</code></pre> <p>To run a single test:</p> <pre><code>/home/debian/gem5/build/ARM/gem5.opt /home/debian/SearchGEM5/hello-custom-binary-Ex.py --isa ARM --input /home/debian/experiment/CodeBooga-cmin/input/test_input_645596930837886.txt</code></pre> <p>You can try different test inputs by changing the file after the --input. To run a single comparison between native and simulation, you run:</p> <pre><code>./2-test-input.sh /home/debian/experiment/CodeBooga-cmin/input/test_input_645596930837886.txt</code></pre> <p>To run the whole process for a folder, and then check the results later, you run the differential testing script:</p> <pre><code>./1-run-diff-test.sh /home/debian/experiment/CodeBooga-cmin/input > LLM-diff-testing-CodeBooga.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Llama-cmin/input > LLM-diff-testing-Llama.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Magicoder-cmin/input > LLM-diff-testing-Magicoder.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Phi-cmin/input > LLM-diff-testing-Phi.log 2>&1 & ./1-run-diff-test.sh /home/debian/experiment/TinyLlama-cmin/input > LLM-diff-testing-TinyLlama.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/gpt3.5-new-cmin/input > LLM-diff-testing-gpt3.5-new.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/gpt3.5-old-cmin/input > LLM-diff-testing-gpt3.5-old.log 2>&1</code></pre> <p><strong>==> This will give the results for Table II (page 7) after manual analysis of the logs.</strong></p> <p>These logs will later flag any issue found in ARM native vs simulation, e.g.:</p> <pre><code><strong>Simulation run: <br></strong></code><code>``` Global frequency set at 1000000000000 ticks per second warn: No dot file generated. Please install pydot to generate the dot file and pdf. src/mem/dram_interface.cc:690: warn: DRAM device capacity (8192 Mbytes) does not match the address range assigned (32 Mbytes) src/base/statistics.hh:279: warn: One of the stats is a legacy stat. Legacy stat is a stat that does not belong to any statistics::Group. Legacy stat is deprecated. src/base/remote_gdb.cc:418: warn: Sockets disabled, not accepting gdb connections gem5 Simulator System. https://www.gem5.org gem5 is copyrighted software; use the --copyright option for details. gem5 version 24.0.0.1 gem5 compiled Sep 5 2024 16:37:09 gem5 started Oct 9 2024 09:30:14 gem5 executing on karl.cs.ucl.ac.uk, pid 2528401 command line: /home/kevenmen/gem5/build/ARM/gem5.opt /home/kevenmen/SearchGEM5/hello-custom-binary-Ex.py --isa ARM --input /home/kevenmen/experiment-7/CodeBooga-cmin/output-r_5/default/queue/id:000003,time:0,execs:0,orig:id:000918,src:000589+000058,time:55938499,execs:11247,op:cm-gem5c-types.so,pos:0 info: Standard input is not a terminal, disabling listeners. src/sim/simulate.cc:199: info: Entering event queue @ 0. Starting simulation... src/arch/arm/insts/pseudo.cc:174: warn: instruction 'bti' unimplemented src/sim/mem_state.cc:448: info: Increasing stack size by one page. src/base/loader/symtab.cc:95: warn: Cannot insert a new symbol table due to name collisions. Adding prefix to each symbol's name can resolve this issue. src/sim/syscall_emul.cc:74: warn: ignoring syscall mprotect(...) src/base/loader/symtab.cc:95: warn: Cannot insert a new symbol table due to name collisions. Adding prefix to each symbol's name can resolve this issue. src/sim/syscall_emul.cc:74: warn: ignoring syscall set_robust_list(...) src/sim/syscall_emul.cc:85: warn: ignoring syscall rseq(...) (further warnings will be suppressed) src/sim/syscall_emul.cc:74: warn: ignoring syscall mprotect(...) src/sim/syscall_emul.cc:74: warn: ignoring syscall mprotect(...) src/sim/syscall_emul.cc:74: warn: ignoring syscall mprotect(...) " is neither a letter nor a digit H is a letter e is a letter l is a letter l is a letter o is a letter 1 is a digit 2 is a digit 3 is a digit " is neither a letter nor a digit Exiting @ tick 5367044250 because exiting with last active thread context. ``` <strong>vs the native run:</strong> ``` " is neither a letter nor a digit H is a letter e is a letter l is a letter l is a letter o is a letter 1 is a digit 2 is a digit 3 is a digit " is neither a letter nor a digit ```<br>In this case, but are giving the same output and hence on bug exposed.</code></pre> <h3> </h3> <p> </p> <h3>2.4 Fuzzing with AFL++ and SearchSYS Custom Mutators (Section V.B. & Table III):</h3> <p>You can run a single execution that way:</p> <pre><code># Define: cm_path="/home/debian/SearchGEM5/src/custom_mutators" corpus_folder="/home/debian/experiment/gpt3.5-old-cmin/" execgem5=/home/debian/gem5-ssbse-challenge-2023/build/ARM/gem5.opt # Run AFL AFL_NO_AFFINITY=1 AFL_SHUFFLE_QUEUE=1 AFL_CUSTOM_MUTATOR_ONLY=1 AFL_DUMB_FORKSRV=1 AFL_FAST_CAL=1 AFL_IGNORE_PROBLEMS=1 \ AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES=1 AFL_SKIP_BIN_CHECK=1 AFL_MAP_SIZE=1200000 \ AFL_CUSTOM_MUTATOR_LIBRARY="$cm_path/cm-gem5c.so;$cm_path/cm-gem5c-bin.so;$cm_path/cm-gem5c-types.so" \ /home/debian/AFLplusplus/afl-fuzz -m 50000 -t 99000 -i $corpus_folder/input/ -o $corpus_folder/output-fuzzed/ -- $execgem5 \ /home/debian/SearchGEM5/hello-custom-binary-Ex.py --isa ARM --input @@</code><br><br>You need then to check the custom mutators loaded okay:</pre> <pre><code>``` [+] Enabled environment variable AFL_IGNORE_PROBLEMS with value 1 [+] Enabled environment variable AFL_FAST_CAL with value 1 [+] Enabled environment variable AFL_CUSTOM_MUTATOR_LIBRARY with value /home/debian/SearchGEM5/src/custom_mutators/cm-gem5c.so;/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-bin.so;/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-types.so [+] Enabled environment variable AFL_SKIP_BIN_CHECK with value 1 [+] Enabled environment variable AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES with value 1 [+] Enabled environment variable AFL_DUMB_FORKSRV with value 1 [+] Enabled environment variable AFL_NO_AFFINITY with value 1 [+] Enabled environment variable AFL_CUSTOM_MUTATOR_ONLY with value 1 afl-fuzz++4.31a based on afl by Michal Zalewski and a large online community ... [*] Loading custom mutator library from '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c.so'... [+] Found 'afl_custom_mutator'. [+] Found 'afl_custom_fuzz_count'. ... [+] Custom mutator '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c.so' installed successfully. [*] Loading custom mutator library from '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-bin.so'... [+] Found 'afl_custom_mutator'. [+] Found 'afl_custom_fuzz_count'. ... [+] Custom mutator '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-bin.so' installed successfully. [*] Loading custom mutator library from '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-types.so'... [+] Found 'afl_custom_mutator'. [+] Found 'afl_custom_fuzz_count'. [*] optional symbol 'afl_custom_post_process' not found. [*] optional symbol 'afl_custom_init_trim' not found. [*] optional symbol 'afl_custom_trim' not found. [*] optional symbol 'afl_custom_post_trim' not found. [*] optional symbol 'afl_custom_havoc_mutation' not found. [*] optional symbol 'afl_custom_havoc_mutation_probability' not found. [*] optional symbol 'afl_custom_queue_get' not found. [*] optional symbol 'afl_custom_splice_optout' not found. [*] optional symbol 'afl_custom_fuzz_send' not found. [*] optional symbol 'afl_custom_post_run' not found. [*] optional symbol 'afl_custom_queue_new_entry' not found [*] optional symbol 'afl_custom_describe' not found. [+] Custom mutator '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-types.so' installed successfully. ```</code></pre> <p>and if it uploaded okay, you will see this screen in the interactive mode:</p> <pre><code>[*] Attempting dry run with 'id:000441,time:0,execs:0,orig:zero-struct-2.txt'... [!] WARNING: instability detected during calibration len = 67, map size = 41474, exec speed = 3356882 us, hash = fef2b4ed27f2d5fe Results of dry-run 0[!] WARNING: Instrumentation output varies across runs. [!] WARNING: Skipped 1 test cases (0.23%) due to timeouts or crashes. [+] All test cases processed. [!] WARNING: The target binary is pretty slow! See docs/fuzzing_in_depth.md#i-improve-the-speed [!] WARNING: Some test cases look useless. Consider using a smaller set. [!] WARNING: You probably have far too many input files! Consider trimming down. [+] Here are some useful stats: Test case count : 64 favored, 255 variable, 5 ignored, 442 total Bitmap range : 27891 to 42022 bits (average: 41556.14 bits) Exec timing : 1.21M to 44.6M us (average: 4.30M us) [*] -t option specified. We'll use an exec timeout of 99000 ms. [+] All set and ready to roll! AFL ++4.31a {default} (...ssbse-challenge-2023/build/ARM/gem5.opt) [explore] ┌─ process timing ────────────────────────────────────┬─ overall results ────┐ │ run time : 0 days, 2 hrs, 39 min, 26 sec │ cycles done : 0 │ │ last new find : 0 days, 0 hrs, 6 min, 46 sec │ corpus count : 575 │ │last saved crash : 0 days, 0 hrs, 11 min, 21 sec │saved crashes : 40 │ │ last saved hang : none seen yet │ saved hangs : 0 │ ├─ cycle progress ─────────────────────┬─ map coverage┴──────────────────────┤ │ now processing : 46.0 (8.0%) │ map density : 3.79% / 4.10% │ │ runs timed out : 0 (0.00%) │ count coverage : 1.57 bits/tuple │ ├─ stage progress ─────────────────────┼─ findings in depth ─────────────────┤ │ now trying : cm-gem5c-types.so │ favored items : 86 (14.96%) │ │ stage execs : 63/66 (95.45%) │ new edges on : 119 (20.70%) │ │ total execs : 5744 │ total crashes : 144 (40 saved) │ │ exec speed : 0.18/sec (zzzz...) │ total tmouts : 0 (0 saved) │ ├─ fuzzing strategy yields ────────────┴─────────────┬─ item geometry ───────┤ │ bit flips : disabled (custom-mutator-only mode) │ levels : 6 │ │ byte flips : disabled (custom-mutator-only mode) │ pending : 555 │ │ arithmetics : disabled (custom-mutator-only mode) │ pend fav : 77 │ │ known ints : disabled (custom-mutator-only mode) │ own finds : 133 │ │ dictionary : n/a │ imported : 0 │ │havoc/splice : 0/0, 0/0 │ stability : 100.00% │ │py/custom/rq : unused, 150/858, unused, unused ├───────────────────────┘ │ trim/eff : 5.90%/361, disabled │ [cpu: 12%] └─ strategy: explore ────────── state: in progress ──┘[!] WARNING: >>-6A Odd size of register is: 75, 8</code></pre> <p> </p> <p><strong>Non-interactive mode and full run: </strong>The results in the paper are created with batch runs. We need to run each repeat for all initial corpora to get the full results for Table III:</p> <pre><code># Repeat 1 ./Experiment-ARM-24h.sh CodeBooga-cmin 1 ./Experiment-ARM-24h.sh gpt3.5-new-cmin 1 ./Experiment-ARM-24h.sh gpt3.5-old-cmin 1 ./Experiment-ARM-24h.sh Llama-cmin 1 ./Experiment-ARM-24h.sh Magicoder-cmin 1 ./Experiment-ARM-24h.sh Phi-cmin 1 ./Experiment-ARM-24h.sh TinyLlama-cmin 1 # Repeat 3 ./Experiment-ARM-24h.sh CodeBooga-cmin 3 ./Experiment-ARM-24h.sh gpt3.5-new-cmin 3 ./Experiment-ARM-24h.sh gpt3.5-old-cmin 3 ./Experiment-ARM-24h.sh Llama-cmin 3 ./Experiment-ARM-24h.sh Magicoder-cmin 3 ./Experiment-ARM-24h.sh Phi-cmin 3 ./Experiment-ARM-24h.sh TinyLlama-cmin 3 # Repeat 4 ./Experiment-ARM-24h.sh CodeBooga-cmin 1 ./Experiment-ARM-24h.sh gpt3.5-new-cmin 1 ./Experiment-ARM-24h.sh gpt3.5-old-cmin 1 ./Experiment-ARM-24h.sh Llama-cmin 1 ./Experiment-ARM-24h.sh Magicoder-cmin 1 ./Experiment-ARM-24h.sh Phi-cmin 1 ./Experiment-ARM-24h.sh TinyLlama-cmin 1 # Repeat 4 ./Experiment-ARM-24h.sh CodeBooga-cmin 4 ./Experiment-ARM-24h.sh gpt3.5-new-cmin 4 ./Experiment-ARM-24h.sh gpt3.5-old-cmin 4 ./Experiment-ARM-24h.sh Llama-cmin 4 ./Experiment-ARM-24h.sh Magicoder-cmin 4 ./Experiment-ARM-24h.sh Phi-cmin 4 ./Experiment-ARM-24h.sh TinyLlama-cmin 4 # Repeat 5 ./Experiment-ARM-24h.sh CodeBooga-cmin 5 ./Experiment-ARM-24h.sh gpt3.5-new-cmin 5 ./Experiment-ARM-24h.sh gpt3.5-old-cmin 5 ./Experiment-ARM-24h.sh Llama-cmin 5 ./Experiment-ARM-24h.sh Magicoder-cmin 5 ./Experiment-ARM-24h.sh Phi-cmin 5 ./Experiment-ARM-24h.sh TinyLlama-cmin 5</code></pre> <p>From each output folder, we analysed the size of queue, crash and hang folders. Further statistics are in fuzzer_stats file. These scripts ensure the run for 24 hours per repeat and input corpus.</p> <p><strong>Note that, as fuzzing is random, the results in Table III can be slightly different between two repeats of the same initial corpus. </strong></p> <p> </p> <p> </p> <h3>2.5 Differential Testing of Fuzzed Corpus Seeds (Section V.B. & Table IV):</h3> <p>We use the same scripts here as in Section 2.3, but with the output fuzzed folders:</p> <pre><code># In the paper we analysed repeat 5, but you can change it to any: ./1-run-diff-test.sh /home/debian/experiment/CodeBooga-cmin/output-r_5/default/queue > Fuzzed-diff-testing-CodeBooga-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Llama-cmin/output-r_5/default/queue > Fuzzed-diff-testing-Llama-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Magicoder-cmin/output-r_5/default/queue > Fuzzed-diff-testing-Magicoder-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Phi-cmin/output-r_5/default/queue > Fuzzed-diff-testing-Phi-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/TinyLlama-cmin/output-r_5/default/queue > Fuzzed-diff-testing-TinyLlama-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/gpt3.5-new-cmin/output-r_5/default/queue > Fuzzed-diff-testing-gpt3.5-new-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/gpt3.5-old-cmin/output-r_5/default/queue > Fuzzed-diff-testing-gpt3.5-old-5.log 2>&1</code></pre> <p>The analysis here and identify the bugs are done manually and require a system simulator expert.</p> <p> </p> <p> </p> <h2>3. Reproduce ICSE-SEIP 2025 Evaluation - Shorter Version and Statistics from Dataset:</h2> <p><strong>Available</strong>: the source code is at <a href="https://github.com/karineek/SearchGEM5/tree/main">https://github.com/karineek/SearchGEM5/tree/main</a> with the extra scripts for ARM and the instructions fully documented in this Zenodo Record in Sections 1 and 2. We added code.zip to this publicly accessible archival Zenodo record to meet the availability requirements.</p> <p><strong>Functional</strong>: The artifact associated with the research are documented fully in sections 2.3, 2.4 and 2.5. Due to its time requirements, we present a shorter version in this section with additional scripts to allow easy quick reproduction of the shorter version's results. We added expected results to help validate the results and noted where the results can be different (e.g. when fuzzing for 2 hours instead of 24 hours).</p> <p><strong>Reusable</strong>: Section 2 and the documentation at <a href="https://github.com/karineek/SearchGEM5/tree/main" target="_blank" rel="noopener">https://github.com/karineek/SearchGEM5/tree/main</a> allow customisation to other targets and editing/adding to the code of SearchSYS. We attached a copy of the GitHub repository here, to ensure the documentation and code are in a single place.</p> <p> </p> <h3>3.1 Table I - Shortened Version</h3> <p>We added the tokens from the <a href="https://github.com/karineek/SearchGEM5/blob/main/src/main/java/LLMTokensOptions.java">GitHub of SearchSYS</a> into four files and filtered out duplicate tokens.</p> <p>To reproduce the counters of Table I:</p> <pre><code>wget "https://zenodo.org/records/14724553/files/TABLE-1.zip" # Get the data and scripts<br>unzip TABLE-1.zip<br>cd TABLE-1<br>./table-1.sh</code><br><br>This should give you:</pre> <pre><code>$ ./table-1.sh >> Token-1 Examples ID_1.log:1:"Scalar Optimizations", ID_1.log:2:"Dead Code Elimination", ID_1.log:3:"Constant Folding", >> Token-2 Examples ID_2.log:32:"Sema", ID_2.log:33:"Serialization", ID_2.log:34:"Parse", ID_2.log:35:"Lex", ID_2.log:36:"AST", >> Token-3 Examples ID_3.log:86:"C Program to Sort an Array using Merge Sort", ID_3.log:58:"Calendar Year in Different Formats", ID_3.log:176:"input include several arguments", >> Token-4 Examples ID_4.log: 343:AND operators, ID_4.log: 779:cimagl function, ID_4.log: 1132:EOF, ID_4.log: 1915:locale, ID_4.log: 2328:pow, ID_4.log: 2614:SCNiMAX, 26 ID_1.log 36 ID_2.log 192 ID_3.log 3480 ID_4.log 3734 total</code></pre> <h3> </h3> <h3>3.2 Table II and Table IV- Shortened Version</h3> <p>We run differential testing to only one of the sets, A, as it is the smallest (and hence should finish quickly). We can then reproduce the table for B-G by using the logs in the Excel file.</p> <p>You will need two scripts for this stage:</p> <pre><code>cd /home/debian/<br>wget "https://zenodo.org/records/14724553/files/1-run-diff-test.sh" <br>wget "https://zenodo.org/records/14724553/files/2-test-input.sh"<br>wget "https://zenodo.org/records/14724553/files/ICSE-SEIP-Bugs-READY.xlsx" ICSE-SEIP-Bugs-READY.xlsx" <br>chmod 777 1-run-diff-test.sh</code><code><br>chmod 777 2-test-input.sh</code></pre> <p>We need the newest version of gem5 and compare it to native run. If you have not yet built gem5, run:</p> <pre><code>cd /home/debian git clone https://github.com/gem5/gem5.git cd gem5 pip install -r requirements.txt scons build/ARM/gem5.opt -j 4<br>cd /home/debian</code></pre> <p>To run a single test:</p> <pre><code>/home/debian/gem5/build/ARM/gem5.opt /home/debian/SearchGEM5/hello-custom-binary-Ex.py --isa ARM --input /home/debian/experiment/CodeBooga-cmin/input/test_input_645596930837886.txt</code></pre> <p>You can try different test inputs by changing the file after the --input. To run a single comparison between native and simulation, you run:</p> <pre><code>./2-test-input.sh /home/debian/experiment/CodeBooga-cmin/input/test_input_645596930837886.txt</code></pre> <p>We start with running differential testing on set A:<code><br><br>nohup ./1-run-diff-test.sh /home/debian/experiment/TinyLlama-cmin/input > LLM-diff-testing-TinyLlama.log 2>&1 &</code></p> <p><code>nohup ./1-run-diff-test.sh /home/debian/experiment/TinyLlama-cmin/output-r_5/default/queue > AFL-diff-testing-TinyLlama.log 2>&1 &</code></p> <p>Then we analyse this manually, as not every failure to simulate a binary is a bug (it can be that the fuzzing corrupted the binary, and the expected behaviour is NOT to run the simulation successfully). However, you can "grep" the error strings, as some of the errors will likely manifest during the fuzzing.</p> <p> </p> <h3>3.3 Table III - Shortened Version</h3> <p>We will do the following: run fuzzing for 1 hour with corpus TinyLlama and check there are results in the queue to see that the fuzzing generated some files. We can reproduce the table from the logs (the full experiments here shall take 24x5x7 hours, which is beyond the expectation of artifact evaluation).</p> <pre><code># Define:<br>cd /home/debian/ cm_path="/home/debian/SearchGEM5/src/custom_mutators" corpus_folder="/home/debian/experiment/TinyLlama-cmin/" execgem5=/home/debian/gem5-ssbse-challenge-2023/build/ARM/gem5.opt # Run AFL AFL_NO_AFFINITY=1 AFL_SHUFFLE_QUEUE=1 AFL_CUSTOM_MUTATOR_ONLY=1 AFL_DUMB_FORKSRV=1 AFL_FAST_CAL=1 AFL_IGNORE_PROBLEMS=1 \ AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES=1 AFL_SKIP_BIN_CHECK=1 AFL_MAP_SIZE=1200000 \ AFL_CUSTOM_MUTATOR_LIBRARY="$cm_path/cm-gem5c.so;$cm_path/cm-gem5c-bin.so;$cm_path/cm-gem5c-types.so" \ /home/debian/AFLplusplus/afl-fuzz -m 50000 -t 99000 -i $corpus_folder/input/ -o $corpus_folder/output-fuzzed/ -- $execgem5 \ /home/debian/SearchGEM5/hello-custom-binary-Ex.py --isa ARM --input @@ </code><br>You should get the same interface as in 2.4 with version:<br><code>AFL ++4.31a {default} (...ssbse-challenge-2023/build/ARM/gem5.opt) [explore]... </code><br><br>Then to reproduce the numbers, you can mine the logs of the original fuzzing campaign. </pre> <p>You will need one script for this stage:</p> <pre><code>cd /home/debian/<br>wget "https://zenodo.org/records/14721385/files/3-run-stat-table-3.sh" <br>chmod 777 3-run-stat-table-3.sh</code></pre> <pre>and the logs:<br><code>mkdir experiments-prev<br>cd experiments-prev<br>cp /home/debian/experiment/* .<br>wget "https://zenodo.org/records/14724553/files/round-1-output.tar.gz"<br>wget "https://zenodo.org/records/14724553/files/round-2-output.tar.gz"<br>wget "https://zenodo.org/records/14724553/files/round-3-output.tar.gz"<br>wget "https://zenodo.org/records/14724553/files/round-4-output.tar.gz"<br>wget "https://zenodo.org/records/14724553/files/round-5-output.tar.gz" </code><br><br>Then run:<br><code>tar -xf round-1-output.tar.gz<br>tar -xf round-2-output.tar.gz<br>tar -xf round-3-output.tar.gz<br>tar -xf round-4-output.tar.gz<br>tar -xf round-5-output.tar.gz<br>cd ..</code><code><br>./3-run-stat-table-3.sh experiments-prev</code><br><br>This should print:</pre> <pre><code>$ ./3-stat-table-3.sh Initial Corpus Size: 612 (Directory: CodeBooga-cmin/input) Initial Corpus Size: 17 (Directory: CodeBooga-cmin/output-r_1/default/crashes) Initial Corpus Size: 17 (Directory: CodeBooga-cmin/output-r_1/default/hangs) Initial Corpus Size: 905 (Directory: CodeBooga-cmin/output-r_1/default/queue) Initial Corpus Size: 3 (Directory: CodeBooga-cmin/output-r_2/default/crashes) Initial Corpus Size: 23 (Directory: CodeBooga-cmin/output-r_2/default/hangs) Initial Corpus Size: 685 (Directory: CodeBooga-cmin/output-r_2/default/queue) Initial Corpus Size: 12 (Directory: CodeBooga-cmin/output-r_3/default/crashes) Initial Corpus Size: 3 (Directory: CodeBooga-cmin/output-r_3/default/hangs) Initial Corpus Size: 842 (Directory: CodeBooga-cmin/output-r_3/default/queue) Initial Corpus Size: 7 (Directory: CodeBooga-cmin/output-r_4/default/crashes) Initial Corpus Size: 11 (Directory: CodeBooga-cmin/output-r_4/default/hangs) Initial Corpus Size: 728 (Directory: CodeBooga-cmin/output-r_4/default/queue) Initial Corpus Size: 17 (Directory: CodeBooga-cmin/output-r_5/default/crashes) Initial Corpus Size: 12 (Directory: CodeBooga-cmin/output-r_5/default/hangs) Initial Corpus Size: 922 (Directory: CodeBooga-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 703 (Directory: gpt3.5-new-cmin/input) Initial Corpus Size: 10 (Directory: gpt3.5-new-cmin/output-r_1/default/crashes) Initial Corpus Size: 2 (Directory: gpt3.5-new-cmin/output-r_1/default/hangs) Initial Corpus Size: 839 (Directory: gpt3.5-new-cmin/output-r_1/default/queue) Initial Corpus Size: 11 (Directory: gpt3.5-new-cmin/output-r_2/default/crashes) Initial Corpus Size: 8 (Directory: gpt3.5-new-cmin/output-r_2/default/hangs) Initial Corpus Size: 898 (Directory: gpt3.5-new-cmin/output-r_2/default/queue) Initial Corpus Size: 14 (Directory: gpt3.5-new-cmin/output-r_3/default/crashes) Initial Corpus Size: 9 (Directory: gpt3.5-new-cmin/output-r_3/default/hangs) Initial Corpus Size: 877 (Directory: gpt3.5-new-cmin/output-r_3/default/queue) Initial Corpus Size: 12 (Directory: gpt3.5-new-cmin/output-r_4/default/crashes) Initial Corpus Size: 5 (Directory: gpt3.5-new-cmin/output-r_4/default/hangs) Initial Corpus Size: 855 (Directory: gpt3.5-new-cmin/output-r_4/default/queue) Initial Corpus Size: 11 (Directory: gpt3.5-new-cmin/output-r_5/default/crashes) Initial Corpus Size: 11 (Directory: gpt3.5-new-cmin/output-r_5/default/hangs) Initial Corpus Size: 886 (Directory: gpt3.5-new-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 442 (Directory: gpt3.5-old-cmin/input) Initial Corpus Size: 43 (Directory: gpt3.5-old-cmin/output-r_1/default/crashes) Initial Corpus Size: 2 (Directory: gpt3.5-old-cmin/output-r_1/default/hangs) Initial Corpus Size: 964 (Directory: gpt3.5-old-cmin/output-r_1/default/queue) Initial Corpus Size: 52 (Directory: gpt3.5-old-cmin/output-r_2/default/crashes) Initial Corpus Size: 5 (Directory: gpt3.5-old-cmin/output-r_2/default/hangs) Initial Corpus Size: 988 (Directory: gpt3.5-old-cmin/output-r_2/default/queue) Initial Corpus Size: 57 (Directory: gpt3.5-old-cmin/output-r_3/default/crashes) Initial Corpus Size: 6 (Directory: gpt3.5-old-cmin/output-r_3/default/hangs) Initial Corpus Size: 934 (Directory: gpt3.5-old-cmin/output-r_3/default/queue) Initial Corpus Size: 49 (Directory: gpt3.5-old-cmin/output-r_4/default/crashes) Initial Corpus Size: 0 (Directory: gpt3.5-old-cmin/output-r_4/default/hangs) Initial Corpus Size: 891 (Directory: gpt3.5-old-cmin/output-r_4/default/queue) Initial Corpus Size: 47 (Directory: gpt3.5-old-cmin/output-r_5/default/crashes) Initial Corpus Size: 1 (Directory: gpt3.5-old-cmin/output-r_5/default/hangs) Initial Corpus Size: 957 (Directory: gpt3.5-old-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 613 (Directory: Llama-cmin/input) Initial Corpus Size: 25 (Directory: Llama-cmin/output-r_1/default/crashes) Initial Corpus Size: 11 (Directory: Llama-cmin/output-r_1/default/hangs) Initial Corpus Size: 1014 (Directory: Llama-cmin/output-r_1/default/queue) Initial Corpus Size: 19 (Directory: Llama-cmin/output-r_2/default/crashes) Initial Corpus Size: 19 (Directory: Llama-cmin/output-r_2/default/hangs) Initial Corpus Size: 939 (Directory: Llama-cmin/output-r_2/default/queue) Initial Corpus Size: 7 (Directory: Llama-cmin/output-r_3/default/crashes) Initial Corpus Size: 14 (Directory: Llama-cmin/output-r_3/default/hangs) Initial Corpus Size: 929 (Directory: Llama-cmin/output-r_3/default/queue) Initial Corpus Size: 15 (Directory: Llama-cmin/output-r_4/default/crashes) Initial Corpus Size: 18 (Directory: Llama-cmin/output-r_4/default/hangs) Initial Corpus Size: 857 (Directory: Llama-cmin/output-r_4/default/queue) Initial Corpus Size: 16 (Directory: Llama-cmin/output-r_5/default/crashes) Initial Corpus Size: 18 (Directory: Llama-cmin/output-r_5/default/hangs) Initial Corpus Size: 909 (Directory: Llama-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 719 (Directory: Magicoder-cmin/input) Initial Corpus Size: 14 (Directory: Magicoder-cmin/output-r_1/default/crashes) Initial Corpus Size: 8 (Directory: Magicoder-cmin/output-r_1/default/hangs) Initial Corpus Size: 1012 (Directory: Magicoder-cmin/output-r_1/default/queue) Initial Corpus Size: 17 (Directory: Magicoder-cmin/output-r_2/default/crashes) Initial Corpus Size: 13 (Directory: Magicoder-cmin/output-r_2/default/hangs) Initial Corpus Size: 1010 (Directory: Magicoder-cmin/output-r_2/default/queue) Initial Corpus Size: 15 (Directory: Magicoder-cmin/output-r_3/default/crashes) Initial Corpus Size: 13 (Directory: Magicoder-cmin/output-r_3/default/hangs) Initial Corpus Size: 974 (Directory: Magicoder-cmin/output-r_3/default/queue) Initial Corpus Size: 11 (Directory: Magicoder-cmin/output-r_4/default/crashes) Initial Corpus Size: 8 (Directory: Magicoder-cmin/output-r_4/default/hangs) Initial Corpus Size: 958 (Directory: Magicoder-cmin/output-r_4/default/queue) Initial Corpus Size: 13 (Directory: Magicoder-cmin/output-r_5/default/crashes) Initial Corpus Size: 13 (Directory: Magicoder-cmin/output-r_5/default/hangs) Initial Corpus Size: 896 (Directory: Magicoder-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 366 (Directory: Phi-cmin/input) Initial Corpus Size: 18 (Directory: Phi-cmin/output-r_1/default/crashes) Initial Corpus Size: 16 (Directory: Phi-cmin/output-r_1/default/hangs) Initial Corpus Size: 707 (Directory: Phi-cmin/output-r_1/default/queue) Initial Corpus Size: 13 (Directory: Phi-cmin/output-r_2/default/crashes) Initial Corpus Size: 26 (Directory: Phi-cmin/output-r_2/default/hangs) Initial Corpus Size: 707 (Directory: Phi-cmin/output-r_2/default/queue) Initial Corpus Size: 18 (Directory: Phi-cmin/output-r_3/default/crashes) Initial Corpus Size: 18 (Directory: Phi-cmin/output-r_3/default/hangs) Initial Corpus Size: 704 (Directory: Phi-cmin/output-r_3/default/queue) Initial Corpus Size: 12 (Directory: Phi-cmin/output-r_4/default/crashes) Initial Corpus Size: 18 (Directory: Phi-cmin/output-r_4/default/hangs) Initial Corpus Size: 568 (Directory: Phi-cmin/output-r_4/default/queue) Initial Corpus Size: 16 (Directory: Phi-cmin/output-r_5/default/crashes) Initial Corpus Size: 10 (Directory: Phi-cmin/output-r_5/default/hangs) Initial Corpus Size: 721 (Directory: Phi-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 206 (Directory: TinyLlama-cmin/input) Initial Corpus Size: 24 (Directory: TinyLlama-cmin/output-r_1/default/crashes) Initial Corpus Size: 9 (Directory: TinyLlama-cmin/output-r_1/default/hangs) Initial Corpus Size: 770 (Directory: TinyLlama-cmin/output-r_1/default/queue) Initial Corpus Size: 36 (Directory: TinyLlama-cmin/output-r_2/default/crashes) Initial Corpus Size: 9 (Directory: TinyLlama-cmin/output-r_2/default/hangs) Initial Corpus Size: 802 (Directory: TinyLlama-cmin/output-r_2/default/queue) Initial Corpus Size: 34 (Directory: TinyLlama-cmin/output-r_3/default/crashes) Initial Corpus Size: 12 (Directory: TinyLlama-cmin/output-r_3/default/hangs) Initial Corpus Size: 748 (Directory: TinyLlama-cmin/output-r_3/default/queue) Initial Corpus Size: 29 (Directory: TinyLlama-cmin/output-r_4/default/crashes) Initial Corpus Size: 11 (Directory: TinyLlama-cmin/output-r_4/default/hangs) Initial Corpus Size: 783 (Directory: TinyLlama-cmin/output-r_4/default/queue) Initial Corpus Size: 31 (Directory: TinyLlama-cmin/output-r_5/default/crashes) Initial Corpus Size: 9 (Directory: TinyLlama-cmin/output-r_5/default/hangs) Initial Corpus Size: 634 (Directory: TinyLlama-cmin/output-r_5/default/queue) --------------------------------- </code></pre> <h3>Note that in this specific dataset, some of the fuzzing ran over 24 hours, hence only the files within the time frame of 24 hours were added to the final calculation in Table III. Hence, there are some discrepancies, e.g. between TinyLlama reported in the paper, queue size, which is 737, vs the result from the script, which is 747. Those sets that ran over 24-hour fuzzing were manually re-calculated, which you can see in the file: <a href="https://zenodo.org/records/14724553/files/ICSE-SEIP-Bugs-READY.xlsx" target="_blank" rel="noopener">ICSE-SEIP-Bugs-READY.xlsx.</a></h3> <h3>==</h3> <h3>3.3.1 Troubleshooting: Error in 3-run-stat-table-3.sh:</h3> <p>Sometimes script 3 may not work due to the specific structure of the folder it requires.</p> <h4><strong>Error:</strong></h4> <p>When running the script <code>./3-run-stat-table-3.sh experiments-prev</code>, you encounter errors like:</p> <p><code>$ ./3-run-stat-table-3.sh experiments-prev<br>Directory: CodeBooga-cmin/input does not exist.<br>---------------------------------<br>Directory: gpt3.5-new-cmin/input does not exist.<br>---------------------------------<br>Directory: gpt3.5-old-cmin/input does not exist.<br>---------------------------------<br>Directory: Llama-cmin/input does not exist.<br>---------------------------------<br>Directory: Magicoder-cmin/input does not exist.<br>---------------------------------<br>Directory: Phi-cmin/input does not exist.<br>---------------------------------<br>Directory: TinyLlama-cmin/input does not exist.<br>---------------------------------</code></p> <h4><strong>Cause:</strong></h4> <p>This error occurs because the required directories and files have not been properly copied into the <code>experiments-prev</code> folder.</p> <h4><strong>Solution:</strong></h4> <p>Follow these steps to fix the error.<br><br></p> <pre><strong>1. Clean Up and Restart the Setup</strong><br>Run the following commands to remove the existing <code>experiments-prev</code> directory and create a fresh copy:</pre> <pre><code>rm -rf experiments-prev # Clean up the existing directory mkdir experiments-prev cd experiments-prev</code></pre> <p><strong>2. Copy Required Files</strong></p> <p>Copy all necessary files from <code>/home/debian/experiment/</code> into the newly created <code>experiments-prev</code> directory:</p> <pre><code>cp /home/debian/experiment/* .</code></pre> <p><strong>3. Verify the Files Have Been Copied</strong> </p> <p>Run the following command to ensure the required files exist</p> <pre><code>ls -l /home/debian/experiment/CodeBooga-cmin/input</code></pre> <ul> <li><strong>Check:</strong> Confirm you can see the files you have just copied.</li> <li>If the files are not visible, ensure the source directory <code>/home/debian/experiment/</code> contains all the required files. If not return to Section 1 to fix it.</li> </ul> <p><strong>4. Download Additional Data Files</strong></p> <p>Download the necessary <code>round-*</code> output files:</p> <pre><code>wget "https://zenodo.org/records/14724553/files/round-1-output.tar.gz" wget "https://zenodo.org/records/14724553/files/round-2-output.tar.gz" wget "https://zenodo.org/records/14724553/files/round-3-output.tar.gz" wget "https://zenodo.org/records/14724553/files/round-4-output.tar.gz" wget "https://zenodo.org/records/14724553/files/round-5-output.tar.gz"</code></pre> <p><strong>5. Extract the Data Files</strong> </p> <p>Extract the contents of the downloaded <code>.tar.gz</code> files:</p> <pre><code>tar -xf round-1-output.tar.gz<br>tar -xf round-2-output.tar.gz<br>tar -xf round-3-output.tar.gz<br>tar -xf round-4-output.tar.gz<br>tar -xf round-5-output.tar.gz<br>cd ..</code><code><br>./3-run-stat-table-3.sh experiments-prev</code></pre> <h3>==</h3>
Unlocking LLM Insights: A Dataset for Automatic Model Card Generation
<p>Language models (LMs) are no longer restricted to the ML community, and instruction-following LMs have led to a rise in autonomous AI agents. As the accessibility of LMs grows, it is imperative that an understanding of their capabilities, intended usage, and development cycle also improves. Model cards are a widespread practice for documenting detailed information about an ML model. To automate model card generation, we introduce a dataset of 500 question-answer pairs for 25 LMs that cover crucial aspects of the model, such as its training configurations, datasets, biases, architecture details, and training resources. We employ annotators to extract the answers from the original paper. Further, we explore the capabilities of LMs in generating model cards by answering questions. We experiment with three configurations: zero-shot generation, retrieval-augmented generation, and fine-tuning on our dataset. The fine-tuned Llama 3 model shows an improvement of 7 points over the retrieval-augmented generation setup. This indicates that our dataset can be used to train models to automatically generate model cards from paper text and reduce the human effort in the model card curation process.</p>
Comparing LLM-Generated Tips and Expert-Created Tips in Quantum Computing Education
<p>This dataset includes the anonymized data from two studies with the goal to evaluate if LLM-generated tips can be used instead of expert-created tips to help students answer quantum computing questions. <br>For the main study (<em>main_test_anonymous.csv</em>) a between-subject design was used to quizz participants of the QUIKSTART 2024 summer school, giving them four multiple-choice quantum physics questions and one tip per question. Each participant was assigned one of four conditions represented by the combination of "creator" and "labeled_as" column in the csv file. In addition we asked to rate quality, correctness and helpfulness for each tip and to rate the perceived difficulty of the question. We removed demographics for anonymities sake.</p> <p>Additionally, we conducted a study directly comparing the LLM-generated and expert-created tips (<em>tip_eval_anonymous.csv</em>). Where we let experts and students rate the tips helpfulness, correctness, if they gave away the answer and if they pointed to relevant concepts. Furthermore, participants had to decide for each question which tip they preferred and were able to leave a comment to give their reasoning. We removed demographics for anonymities sake.<br><br>These datasets were evaluated in the paper "LLM-Generated Tips Rival Expert-Created Tips in Helping Students Answer Quantum-Computing Questions" by Lars Krupp, Jonas Bley, Isacco Gobbi, Alexander Geng, Sabine Müller, Sungho Suh, Ali Moghiseh, Arcesio Castaneda Medina, Valeria Bartsch, Artur Widera, Herwig Ott, Paul Lukowicz, Jakob Karolus, Maximilian Kiefer-Emmanouilidis</p>
Design antibody using LLM
<p>Design antibody variants using LLM</p> <p>Link to LLM: <a href="https://github.com/brianhie/efficient-evolution">GitHub - brianhie/efficient-evolution: Efficient evolution from protein language models</a></p> <p>Python code: use_llm_to_design_variants.py</p> <p>Dataset: antibody_affinity_protein_sabdab_vhvl_immunebuilder_outfiles_nomissing.fasta</p> <p>eeout,.txt: example output files</p>
Antibody dataset Kd and LLM embedding
<p>A dataset of ~500 antibodies with binding affinity Kd obtained from SAbDab via Therapeutic Data Commons. LLM embedding Ablang2</p> <p>Python code: get_antibody_llm_embedding.py</p> <p>Dataset sequence: antibody_affinity_protein_sabdab_vhvl.csv</p> <p>Dataset LLM embedding: antibody_affinity_protein_sabdab_vhvl_ablang2seqenc.csv </p>
Secure_LLM_Datasets
<p>The dataset of the paper <span>An Exploratory Study on Fine-Tuning Large </span><span>Language Models for Secure Code Generation.</span></p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.