Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
478
datasets available to search
ShareScore release 0.9.0
Dataset results
478 results for “artifact”
artifact 6 - Sharpening Tool - Sanford Museum
This is a looted, unprovenienced, artifact donated to the Sanford Museum in Sanford, Florida. This aritfact was captured in 3D using low-cost photogrammetry using Agisoft's Metashape and my personal cellphone camera, the Samsung Galaxy Note 10+. This was the first artifact I preserved using photogrammetry and is apart of a collection of looted artifacts from the Sanford Museum. Source: Objaverse 1.0 / Sketchfab
Resource Usage and Optimization Opportunities in Workflows of GitHub Actions - Artifact
<p>This package contains the data, the code used for collection and the analysis notebooks used to obtain the results presented in the paper: "Resource Usage and Optimization Opportunities in Workflows of GitHub Actions" published at ICSE2024.</p>
Artifact for "Using Evolutionary Algorithms to Find Cache-Friendly Generalized Morton Layouts for Arrays"
<p>This artifact accompanies the paper "Using Evolutionary Algorithms to Find Cache-Friendly Generalized Morton Layouts for Arrays", accepted to the International Conference on Performance Engineering (ICPE) 2024.</p> <p>Please see <code>README.md</code> for more information.</p>
Dataset for the Paper "Longevity of Artifacts in Leading Parallel and Distributed Systems Conferences: a Review of the State of the Practice in 2023"
<p>Dataset used for the paper "Longevity of Artifacts in Leading Parallel and Distributed Systems Conferences: a Review of the State of the Practice in 2023" submitted at ACM REP'24.</p> <p>The associated analysis scripts are available on Software-Heritage: https://archive.softwareheritage.org/swh:1:dir:3904795a49327b1e9b7ef1a8aff995b1f94ae6fb;origin=https://github.com/GuilloteauQ/artefact-lifetime;visit=swh:1:snp:9adc667f9fefcb6783df076520ef357231030ad4;anchor=swh:1:rev:ad4d19abb5da12fd6df708bc13e444d834c85dfa</p>
Artifact for the paper "Mutation-based Lifted Repair of Software Product Lines"
<p>This work presents a novel lifted repair algorithm for program families (Software Product Lines - SPLs) based on code mutations.<br>The inputs of our tool are an erroneous SPL and a specification given in the form of assertions. We use variability encoding to transform the given SPL into a single program, called family simulator, which is translated into a set of SMT formulas whose conjunction is satisfiable iff the simulator (i.e. the input SPL) violates an assertion. We use a predefined set of mutations applied to feature and program expressions of the given SPL. The tool repeatedly mutates the erroneous family simulator and checks if it becomes (bounded) correct. The outputs of our tool are all minimal repairs in the form of minimal number of (feature and program) expression replacements such that the repaired SPL is (bounded) correct with respect to a given set of assertions.</p> <p>We present a prototype tool for repairing #ifdef-based C programs (i.e., annotative SPLs). The experimental results show that our approach is able to successfully repair various interesting SPLs.</p>
PLDI2024 Artifact: A HAT Trick: Automatically Verifying Representation Invariants Using Symbolic Finite Automata
<p>This artifact contains:</p> <ol> <li>README.md : the artifact guide.</li> <li>marple-original-submission.pdf: the original submitted paper.</li> <li>marple:pldi-2024.tar.gz: the docker image (optional, we recommend to pull from the docker hub, see README.md).</li> <li>Dockerfile: the docker file that can reproduce the docker image (optional, we recommend to pull from the docker hub, see README.md).</li> </ol>
Jacdac: Service-based Prototyping of Embedded Systems (Artifact Evaluation)
<p>This artifact allows others to reproduce and explore the results seen in "Jacdac: Service-based Prototyping of Embedded Systems". The artifact contains a prebuilt docker image and the Dockerfile source used to produce the prebuilt docker image. Evaluators should follow the README contained in this artifact for complete instruction.</p>
Artifacts of a Controlled Experiment About the Impact of Microservice API Patterns on Understandability
<p>This repository includes the data set of responses from 66 participants and the analysis scripts of a controlled experiment about the impact of microservice API patterns on understandability.</p>
Evaluation Artifacts of RESTRuler, a Static Analysis Tool to Detect Design Rule Violations in OpenAPI Definitions
<p>All artifacts related to the empirical evaluation of the RESTRuler CLI, a Java-based tool to identify design rule violations in OpenAPI descriptions</p> <p>You can find the latest version here: <a href="https://github.com/restful-ma/rest-ruler-evaluation">https://github.com/restful-ma/rest-ruler-evaluation</a></p>
Artifacts for "Reviewing War: Unconventional User Reviews as a Side Channel to Circumvent Information Controls"
<p>List of keywords used to build the text classifier for the paper <em>Reviewing War: Unconventional User Reviews as a Side Channel to Circumvent Information Controls</em>. For more information on how these keywords were obtained, see the "Data Labeling" section of this paper.</p> <p>The provided CSV file contains 3 columns:</p> <ul> <li>"keyword": Lowercased keyword in either English, Russian, Ukranian or Polish.</li> <li>"weight": Score indicating the keyword's war-related affinity, with 3 being the most related. Negative weights are used to correct for exceptions in our classifier.</li> <li>"topic": Subject of the keyword used for topic analysis.</li> </ul>
Artifact for "Inside Bug Report Templates: An Empirical Study on Bug Report Templates in Open-Source Software"
<p>This is the artifact for the paper "Inside Bug Report Templates: An Empirical Study on Bug Report Templates in Open-Source Software".</p> <p><strong>What the artifact does:</strong><br>1) a questionnaire that we used for our online survey (PDF);<br>2) the valid responses of our online survey (CSV).</p> <p>3) the code of preprocessing (.py).</p> <p>4) the dataset of preprocessing and labeling (CSV).</p>
Artifacts for Demystifying Device-specific Compatibility Issues in Android apps
<p>This artifact is for the paper "Demystifying Device-specific Compatibility Issues in Android Apps."</p> <p>We include our repositories link, commit hash, our GitHub crawler, its output, and some representative examples for Device-specific compatibility issues in this artifact.</p> <p>Please checkout the README.pdf first. The main artifact pack is the `device-specific-artifacts.zip`.</p> <p>Other zips starting with "repo-" are repositories archives.</p>
Photographic pictures of selected artifacts from Didon´s excavations at Abri Blanchard in the collection of the Naturhistorische Gesellschaft Nürnberg, Bavaria, Germany
<p>provided by the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Institut für Ur- und Frühgeschichte. The data presented here includes photographs of the most indicative pieces from an assemblages archived in the magazine of the Naturhistorische Gesellschaft Nürnberg, Abteilung Vorgeschichte. For more information about the hosting institution, see <a href="https://www.nhg-nuernberg.de/main.php?section=Vorge">https://www.nhg-nuernberg.de/main.php?section=Vorge</a>.</p> <p>All photographic pictures made and arranged by U. Maaß (FAU).</p> <p>For further information about the context of the depicted artifacts see the following publication:</p> <p><em>Uthmeier, T., Pasda, K., Wiesner, C. M., & Lechner, C. (2020). </em><em>Aurignacian only? New evidence for a Gravettian component in the lithic and bone industry from Didon´ s excavations at Abri Blanchard: Ausschließlich Aurignacien? Neue Belege für eine Gravettien-Komponente im lithischen und organischen Fundmaterial aus den Ausgrabungen Didons im Abri Blanchard. Quartär – Internationales Jahrbuch zur Erforschung des Eiszeitalters und der Steinzeit 67, 23-43.</em></p> <p>The full-text is available under <a href="https://doi.org/10.7485/qu.2020.67.88914">https://doi.org/10.7485/qu.2020.67.88914</a></p> <p><strong>Abstract</strong> - The article presents for the first time the collection from Didon´s excavations at Abri Blanchard hosted in the Naturhistorische Gesellschaft Nürnberg. Bought by the Gesellschaft in 1912 through the agency of Hugo Obermaier, the material consists of lithic artifacts, artifacts from bone, antler and ivory, personal ornament, faunal remains and pieces of the brecciated archaeological layer(s). The analysis shows that most of the lithic and osseous artifacts fit well into the variability of the Aurignacian. The bone points can formally be attributed to an early Aurignacian, characterized by split-based points, and a later Aurignacian, indicated by the presence of massive-based points. Although Gravettian artifacts have already been identified in one of the collections hosted in other museums, these were restricted to isolated pieces among the osseous industry. The present study for the first time identified not only additional Gravettian pieces among the osseous industry, but at the same time also a small series of Noailles-type burins, several Gravette points, and a bidirectional core alongside a Pavlovian smoother (“Geweihstecher”). The article discusses how these new findings compare to existing hypothesis about the origin of a non-Aurignacian component at the site.</p> <p><strong>Zusammenfassung</strong> - In dem vorliegenden Artikel werden erstmals Artefakte und Faunenreste aus den Ausgrabungen Didons im Abri Blanchard beschrieben, die sich im Archiv der Naturhistorischen Gesellschaft Nürnberg befinden. Das Inventar wurde 1912 nach Vermittlung von Hugo Obermaier angekauft und enthält neben Steinartefakten und Artefakten aus Knochen, Geweih und Elfenbein auch Faunenreste, Schmuckanhänger und Brekzien-Bruchstücke aus der Fundschicht. Die meisten Steinartefakte und Artefakte aus Knochen, Geweih und Elfenbein können in das Aurignacien gestellt werden. Die Geschossspitzen erlauben bei Anwendung der klassischen französischen Gliederung die Unterscheidung eines frühen Aurignacien mit Geschossspitzen mit gespaltener Basis einerseits, und eines entwickelten Aurignacien mit Geschossspitzen mit massiver Basis andererseits. Wie in bereits publizierten Analysen an Altbeständen in Museen in Europa und den Vereinigten Staaten, so konnte auch in dem Nürnberger Inventar eine Gravettien-Komponente erkannt werden. Im Unterschied zu existierenden Untersuchungen liegen hier jedoch nicht nur isolierte Stücke vor, sondern eine Serie von Noailles-Sticheln und Gravette-Spitzen sowie ein großer bidirektionaler Kern. In dem Artikel werden die neuen Erkenntnisse mit bereits vorliegenden Ergebnissen aus älteren Studien verglichen sowie verschiedene Hypothesen zum Vorliegen einer Komponente, die nicht in das Aurignacien gestellt werden kann, diskutiert.</p>
Artifact for ACM CSUR article: "A Meta-Study of Software-Change Intentions"
<p>This archive includes the CSV files with the raw and aggregated data we used for our meta study:</p> <p>"A Meta-Study of Software-Change Intentions" published at the ACM Computing Surveys journal.</p> <p> </p> <p>Please refer to the readme for details on the files.</p>
IRCA: ndss2025-artifact-release
<p>We have submitted two ROS2 bag files that capture recorded fault scenarios, which have been verified by developers from the Autoware project:</p> <ol> <li> <p><strong>File:</strong> <code>rosbag2_2024_05_07-22_10_34_0</code><br><strong>Related Issue:</strong> <a href="https://github.com/autowarefoundation/autoware.universe/issues/6938" target="_blank" rel="noopener">Issue #6938</a><br><strong>Description:</strong> This bag file contains data pertinent to the issue where the obstacle pointcloud-based validator node incorrectly removes small detected objects, such as traffic cones, identified by the CenterPoint node.</p> </li> <li> <p><strong>File:</strong> <code>rosbag2_2024_06_18-17_11_08</code><br><strong>Related Issue:</strong> <a href="https://github.com/autowarefoundation/autoware.universe/issues/7563" target="_blank" rel="noopener">Issue #7563</a><br><strong>Description:</strong> This file is related to the issue where the perception system fails to detect and track pedestrians when they are close to the vehicle, recognizing them only after they have moved away.</p> </li> </ol> <p>These files are intended to assist in reproducing the reported issues and aid in their resolution.</p>
Electron Beam-Induced Artifacts in SEI Characterization: Evidence from Controlled-Dose Diffraction Studies (datasets)
<p>The paper has been published, and the data were updated following the revision. The dataset is now publicly available. All data are in DM4 format, and we recommend using Py4DSTEM or other tools after processing them to h5 format.</p>
Artifact of Search+LLM-based Testing for ARM Simulators
<h1>Abstract.</h1> <p>We present and discuss our solution to the problem of testing software simulators, using SearchSYS to test the gem5 VLSI digital circuit simulator, employed by ARM to test their systems. In particular, we focus on the simulation of the ARM silicon chip Instruction Set Architecture (ISA).</p> <p>SearchSYS can create test cases that activate bugs by combining LLMs, fuzzing, and differential testing. Using only LLM, SearchSYS identified 74 test cases that activated bugs. By incorporating fuzzing, this number increased by 93 additional bug-activating cases within 24 hours. Through differential testing, we identified 624 bugs with LLM-generated test cases and 126 with fuzzed test inputs. Out of the total number of bug-activating test cases, 4 unique bugs have been reported and acknowledged by developers. Additionally, we provided developers with a test suite and fuzzing statistics, and open-sourced SearchSYS.</p> <p>We published our artifact in Zenodo https://zenodo.org/records/14724553.</p> <p>To reproduce the evaluation in the paper, ensure your system meets the requirements, and then follow the instructions to reproduce the tables in the results. <strong>Hardware requirements:</strong> You will need an ARM machine, with 80 GB HD free space, and 72 GB RAM (for AFL++ fuzzing) or (else) 16 GB if merely running the differential testing without building gem5 with AFL++ instrumentation. </p> <p> </p> <p>The artifact documentation contains three sections:</p> <p><strong>Section 1:</strong> Software and hardware requirements;</p> <p><strong>Section 2:</strong> documentation of how to build the testing environment for ARM with full instructions on how to reproduce the four tables in our ICSE-SEIP 2025 paper; and</p> <p><strong>Section 3:</strong> a shortened version of the evaluation, to be run within a day or less.</p> <p> </p> <p>We suggest starting with the shortened evaluation, after installing AFL++, gem5 and SearchSYS (Section 1, Section 2.1 - TinyLlama only, and Section 2.2). </p> <p>==</p> <p> </p> <p><strong>ARTIFACT EVALUATION NOTE:</strong></p> <p><strong>The <a href="https://zenodo.org/records/14753604/files/README.md" target="_blank" rel="noopener">README.md</a> file in this artifact is a copy of this document. However, 2 mistakes in Section 3.3 were fixed but have not yet been pushed into the README.md file. Consequentially, it is recommended to follow the text here instead. We plan to push a new README.md file once passing the first stage of the artefact evaluation (fixing any other mistakes the committee will find).</strong></p> <p>==</p> <h1>Additional Resources.</h1> <p>SearchSYS tool can be downloaded here: <a href="https://github.com/karineek/SearchGEM5/commit/a239d3cd787c0f5cd3be49b2f21179ff90df89f3">a239d3c</a> (for this work).</p> <p>To process the fuzzed test inputs, post-fuzzing, the *.sh here can run on the output/default/queue and crash folders.</p> <p>The rest are the data collected during experiments: (1) the Excel file contains the full analysis while (2) *.tar.gz are all outputs from the fuzzing stage. The LLM data is taken from <a href="https://zenodo.org/records/13450472">here</a>. You can follow the instructions <a href="https://zenodo.org/records/13450472">here</a> to install and run SearchSYS for X86.</p> <p> </p> <p><strong>The record here details how to run SearchSYS on ARM machines (arm64-ubuntu-2204 and Red Hat-based system)</strong>.</p> <p> </p> <p>Cite the paper:</p> <table> <tbody> <tr> <td> <pre><code>@inproceedings{Even-Mendoza:2025:ICSE-SEIP, author = {Karine Even-Mendoza and Hector D. Menendez and W.B Langdon and Aidan Dakhama and Justyna Petke and Bobby R. Bruce}, title = {{Search+LLM-based} Testing for {ARM} Simulators}, booktitle = {2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)}, year = {2025}, editor = {}, volume = {}, number = {}, series = {}, pages = {}, address = {Ottawa, Canada}, publisher_address = {}, month = {April 27-May 3}, organisation = {}, publisher = {}, note = {https://conf.researchr.org/track/icse-2025/icse-2025-software-engineering-in-practice#Accepted-Papers}, keywords = {Software System Simulation, SearchSYS, gem5, AFL++, Fuzzing, Fuzz Testing, Differential Testing, Search-Based Software Testing, LLM, Language Models, Ollama, CodeLlama, TinyLlama, Phi2, Llama2, Magicoder, CodeBooga, GPT-3.5-turbo, prompt engineering}, ISBN13 = {}, url = {https://solar.cs.ucl.ac.uk/pdf/ICSE_SEIP_2025___SearchSYS_and_ARM.pdf}, code_url = {https://zenodo.org/records/13450472, https://zenodo.org/records/14724553}, size = {12 pages}, abstract = {In order to aid quality assurance of large complex hardware architectures, system simulators have been developed. However, such system simulators do not always accurately mirror what would happen on a real device. A significant challenge in testing these simulators arises from the complexity of having to model both the simulation and the infinite number of software that could be run on such a device. Our previous work introduced SearchSYS, a testing framework for software simulators. SearchSYS leverages a large language model for initial seed C code generation, which is then compiled, and the resultant binary is fed to a fuzzer. We then use differential testing by running the outputs of fuzzing on real hardware and a system simulator to identify mismatches. We present and discuss our solution to the problem of testing software simulators, using SearchSYS to test the gem5 VLSI digital circuit simulator, employed by ARM to test their systems. In particular, we focus on the simulation of the ARM silicon chip Instruction Set Architecture (ISA). SearchSYS can create test cases that activate bugs by combining LLMs, fuzzing, and differential testing. Using only LLM, SearchSYS identified 74 test cases that activated bugs. By incorporating fuzzing, this number increased by 93 additional bug-activating cases within 24 hours. Through differential testing, we identified 624 bugs with LLM-generated test cases and 126 with fuzzed test inputs. Out of the total number of bug-activating test cases, 4 unique bugs have been reported and acknowledged by developers. Additionally, we provided developers with a test suite and fuzzing statistics, and open-sourced SearchSYS for initial seeds and SearchSYS code, for adjustment to ARM8v and fuzzed seeds.}, }</code></pre> </td> </tr> </tbody> </table> <pre>==</pre> <h1>Experiments.</h1> <div> <div> <div> <div> <div> <div> <p>Ensure your system meets the requirements, then follow the instructions to reproduce the tables in Section 5 (V. Results).</p> <p><strong>Hardware requirements: </strong>You will need an ARM machine, with 80 GB HD free space, and 72 GB RAM (for AFL++ fuzzing) or 16 GB else (just differential testing without building gem5 with AFL++ instrumentation).</p> </div> </div> </div> </div> </div> </div> <h2><strong>1. Requirements:</strong></h2> <h3><strong>1.1 Hardware</strong></h3> <p><strong>Hardware requirements: </strong>You will need an ARM machine, with 80 GB HD free space, and 72 GB RAM (for AFL++ fuzzing) or 16 GB else (just differential testing without building gem5 with AFL++ instrumentation).</p> <table> <tbody> <tr> <td><strong>Requirement</strong></td> <td><strong>Details</strong></td> </tr> <tr> <td>Processor</td> <td>ARM</td> </tr> <tr> <td>HD space</td> <td>At least 80 GB of free disk space </td> </tr> <tr> <td>RAM</td> <td> <p>- 72 GB (for AFL++ fuzzing) </p> <p>- 16 GB (for differential testing without building gem5 with AFL++ instrumentation)</p> </td> </tr> <tr> <td>Operating System</td> <td>Tested on Ubuntu and Red Hat </td> </tr> </tbody> </table> <h3> </h3> <h3><strong>1.2 Software and Packages:</strong></h3> <ul> <li>GCC-11</li> <li>wget, zip, unzip (any version)</li> <li>LLVM-13+ (we tested with LLVM-13, but any newer version should be okay)</li> <li>Python3 (tested with Python 3.8 and 3.10)</li> <li>Other packages:</li> </ul> <pre><code>sudo apt-get update sudo apt-get upgrade sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get -y install gcc-11 g++-11 cpp-11 wget lsb-release gnupg software-properties-common sudo rm /usr/bin/cpp /usr/bin/gcc /usr/bin/g++ /usr/bin/gcov /usr/bin/c++ sudo rm /usr/bin/cc sudo ln -s /usr/bin/cpp-11 /usr/bin/cpp sudo ln -s /usr/bin/gcc-11 /usr/bin/gcc sudo ln -s /usr/bin/gcc-11 /usr/bin/cc sudo ln -s /usr/bin/g++-11 /usr/bin/g++ sudo ln -s /usr/bin/g++-11 /usr/bin/c++ sudo ln -s /usr/bin/gcov-11 /usr/bin/gcov wget https://apt.llvm.org/llvm.sh sudo chmod +x llvm.sh sudo ./llvm.sh 13 sudo ln -s /usr/bin/llvm-config-13 /usr/bin/llvm-config sudo apt-get install -y lld-13 llvm-13-dev clang-13 sudo apt-get install -y lld llvm llvm-dev clang sudo apt-get install -y build-essential python3-dev automake cmake git flex bison libglib2.0-dev libpixman-1-dev python3-setuptools cargo libgtk-3-dev sudo apt-get install -y ninja-build cmake sudo apt-get install -y git make cmake llvm gdb coreutils sudo apt-get install -y gcc-11-plugin-dev sudo apt -y autoremove sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-13 1300 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-13 sudo update-alternatives --install /usr/bin/llvm-config llvm-config /usr/bin/llvm-config-13 1300 LLVM_CONFIG=/usr/bin/llvm-config-13</code></pre> <p>We attached all installed packages in <a href="https://zenodo.org/records/14724553/files/installed_karl.log">installed_karl.log</a> (Red Hat-based system) and <a href="https://zenodo.org/records/14724553/files/installed_m400.log" target="_blank" rel="noopener">installed_m400.log</a> + <a href="https://zenodo.org/records/14724553/files/installed_m400_repeat.log" target="_blank" rel="noopener">installed_m400_repeat.log</a>(Ubuntu/Unix, arm64-ubuntu-2204).</p> <p> </p> <p> </p> <h2>2. Reproduce ICSE-SEIP 2025 Evaluation:</h2> <p>This section contains the full instructions for running SearchSYS on ARM. We include further explanations on how to customise the evaluation or run part of it. This is the full version, you can use the shortened version in Section 3. This evaluation should take around 40 days to reproduce: 7 initial corpora, each fuzzed with AFL++ five times (5 repeats of the fuzzing campaign), and then around 2-3 days for differential testing of the initial corpora and the same for each repetition (and we had 5 repeats of the fuzzing campaign). </p> <p>To reproduce the experiments follow these commands (Unix system):</p> <pre><code>## CREATE A NEW USER (you can also use your own user but then you will need to amend /home/debian and debian, accordingly) ## sudo adduser debian # if needed run: # sudo passwd debian # and set some simple password sudo usermod -aG sudo debian sudo mkdir -p /home/debian sudo usermod -d /home/debian debian su - debian ## GET DATA - 7 set of LLM-generated C programs ## cd /home/debian wget "https://zenodo.org/records/10999115/files/LLM_test_inputs-4FuzzExp2.zip" # Get the data unzip -n LLM_test_inputs-4FuzzExp2.zip -d /home/debian/experiment cd /home/debian/experiment/ sed -i "s:/home/ubuntu/experiment-7/:/home/debian/experiment/:g" */input/*.txt cat /home/debian/experiment/TinyLlama-cmin/input/test_input_574567374417673.txt</code></pre> <h3> </h3> <h3>YOU WILL NEED TO COPY ALL SCRIPTS OF THIS ARTIFACT INTO <code>/home/debian</code></h3> <pre><code>wget "https://zenodo.org/records/14724553/files/1-run-diff-test.sh" wget "https://zenodo.org/records/14724553/files/2-test-input.sh" wget "https://zenodo.org/records/14724553/files/run_AFL_loop_v6.sh" wget "https://zenodo.org/records/14724553/files/Experiment-ARM-24h.sh"</code></pre> <p>We will explain in the next sections when to run each. There is a shorter version of the experiments in Section 3.</p> <h3><br>2.1 Create Initial Data Seedds for ARM Machine:</h3> <pre><code>## COMPILE BINARIES - build the test binaries and tailor the paths in the seeds ## ### Model CodeBooga ### cd /home/debian/experiment/CodeBooga for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf CodeBooga-cmin/binary cp -r CodeBooga/binary CodeBooga-cmin/binary cd /home/debian/experiment/CodeBooga-cmin/input sed -i 's\/CodeBooga/\/CodeBooga-cmin/\g' *txt ### Model Llama ### cd /home/debian/experiment/Llama for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Llama-cmin/binary cp -r Llama/binary Llama-cmin/binary cd /home/debian/experiment/Llama-cmin/input sed -i 's\/Llama/\/Llama-cmin/\g' *txt ### Model Magicoder ### cd /home/debian/experiment/Magicoder for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Magicoder-cmin/binary cp -r Magicoder/binary Magicoder-cmin/binary cd /home/debian/experiment/Magicoder-cmin/input sed -i 's\/Magicoder/\/Magicoder-cmin/\g' *txt ### Model TinyLlama ### cd /home/debian/experiment/TinyLlama for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf TinyLlama-cmin/binary cp -r TinyLlama/binary TinyLlama-cmin/binary cd /home/debian/experiment/TinyLlama-cmin/input sed -i 's\/TinyLlama/\/TinyLlama-cmin/\g' *txt ### Model Phi ### cd /home/debian/experiment/Phi for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf Phi-cmin/binary cp -r Phi/binary Phi-cmin/binary cd /home/debian/experiment/Phi-cmin/input sed -i 's\/Phi/\/Phi-cmin/\g' *txt ### Model gpt3.5-new ### cd /home/debian/experiment/gpt3.5-new for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf gpt3.5-new-cmin/binary cp -r gpt3.5-new/binary gpt3.5-new-cmin/binary cd /home/debian/experiment/gpt3.5-new-cmin/input sed -i 's\/gpt3.5-new/\/gpt3.5-new-cmin/\g' *txt ### Model gpt3.5-old ### cd /home/debian/experiment/gpt3.5-old for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done rm binary/fuzz_* || true rm .DS_Store || true cd /home/debian/experiment/ rm -rf gpt3.5-old-cmin/binary cp -r gpt3.5-old/binary gpt3.5-old-cmin/binary cd /home/debian/experiment/gpt3.5-old-cmin/input sed -i 's\/gpt3.5-old/\/gpt3.5-old-cmin/\g' *txt</code></pre> <p>At this point all data for fuzzing is ready. You need now to get SearchSYS and build the SUT (gem5) from source.</p> <p> </p> <p> </p> <h3>2.2 Build the tools:</h3> <p>We first build AFL with some customisation:</p> <pre><code>## GET AFL++ SearchSYS ## cd /home/debian git clone https://github.com/karineek/SearchGEM5.git git clone https://github.com/AFLplusplus/AFLplusplus.git cd AFLplusplus git checkout f596a297c4de6a5e1a6fb9fbb3b4e18124a24f58 cp ../SearchGEM5/src/gem5-afl/afl-fuzz-init.c src/afl-fuzz-init.c AFL_USE_ASAN=0 make ### The compilation shall state that it is ARM architecture: ### ``` debian@node0:~/AFLplusplus$ AFL_USE_ASAN=0 make [*] Compiling AFL++ for OS Linux on ARCH aarch64 [+] ZLIB detected [!] Note: skipping x86 compilation checks (AFL_NO_X86 set). [+] shmat seems to be working. [+] Python 3.8.10 support seems to be working. [+] Everything seems to be working, ready to compile ... ```</code></pre> <p><br>then you need to check that AFL is built correctly:</p> <pre><code>```<br></code><code>Build Summary: [+] afl-fuzz and supporting tools successfully built [+] LLVM basic mode successfully built [+] LLVM mode successfully built [+] LLVM LTO mode successfully built [+] gcc_mode successfully built<br>```</code></pre> <p>Then build gem5 with AFL++ instrumentation:</p> <pre><code>cd /home/debian git clone https://github.com/BobbyRBruce/gem5-ssbse-challenge-2023.git cd /home/debian/gem5-ssbse-challenge-2023 cp ../SearchGEM5/src/gem5-afl/SConstruct . cp ../SearchGEM5/src/gem5-afl/SConscript src/ # If needed install: sudo apt install python3-pip sudo apt install scons protobuf-compiler libhdf5-dev libgoogle-perftools-dev libsystemc-dev # Then install the requirements for gem5: pip install -r requirements.txt # And build gem5 with AFL++ instrumentation:</code><code><br>CC=/home/debian/AFLplusplus/afl-cc CXX=/home/debian/AFLplusplus/afl-c++ LINK=/home/debian/AFLplusplus/afl-c++ \<br> scons build/ARM/gem5.opt -j 4</code><code> # Then test if built:<br>./build/ARM/gem5.opt -C</code></pre> <p>During the configuration, you need to see it built for ARM, hence (1) make sure you use build/ARM not X86, and check the configuration log to see that the right architecture has been properly selected for the build. Executing gem5.opt should give you the following message: (or something similar)</p> <pre><code>``` debian@node0:~/gem5-ssbse-challenge-2023$ ./build/ARM/gem5.opt -C Please see individual files for details of the license on each file. The preferred license can be found in LICENSE. All files in this distribution (other than in the ext directory) have licenses based on the BSD or MIT licenses. Some files in the ext directory are GNU LGPL. No other licenses are found in this distribution. Beyond the BSD license, some files include the following clarification of the license as required by the copyright holder: The license below extends only to copyright in the software and shall not be construed as granting a license to any other intellectual property including but not limited to intellectual property relating to a hardware implementation of the functionality of the software licensed hereunder. You may use the software subject to the license terms below provided that you ensure that this notice is replicated unmodified and in its entirety in all distributions of the software, modified or unmodified, in source code or in binary form. The copyright holders include (not counting the ext directory): Copyright (c) 2000-2011 The Regents of The University of Michigan Copyright (c) 1990,1993-1995,2007-2010 The Hewlett-Packard Development Company Copyright (c) 1999-2009,2011 Mark D. Hill and David A. Wood Copyright (c) 2009-2011 ARM Limited Copyright (c) 2008-2009 Princeton University Copyright (c) 2007 MIPS Technologies, Inc. Copyright (c) 2009-2011 Advanced Micro Devices, Inc. Copyright (c) 2009 The University of Edinburgh Copyright (c) 2007-2008 The Florida State University Copyright (c) 2010 Massachusetts Institute of Technology Copyright (c) 1990-1993 The Regents of the University of California Copyright (c) 2006-2009 Nathan Binkert Copyright (c) 2001 The NetBSD Foundation, Inc. Copyright (c) 2010-2011 Gabe Black Copyright (c) 1994 Adam Glass Copyright (c) 1990-1992 MIPS Computer Systems, Inc. Copyright (c) 2004 Richard J. Wagner Copyright (c) 2000 Computer Engineering and Communication Networks Lab Copyright (c) 2001 Eric Jackson Copyright (c) 1990 Hewlett-Packard Development Company Copyright (c) 1994-1996 Carnegie-Mellon University. Copyright (c) 1993-1994 Christopher G. Demetriou Copyright (c) 1997-2002 Makoto Matsumoto and Takuji Nishimura Copyright (c) 1998,2001 Manuel Bouyer. Copyright (c) 2016-2017 Google Inc. ```</code></pre> <p>Last step: compile the custom mutators:</p> <pre><code>cd /home/debian/SearchGEM5/src/custom_mutators/ AFL_HOME=/home/debian/AFLplusplus sed -i "s:/home/ubuntu/AFLplusplus:$AFL_HOME:g" compile_share_test.sh sed -i "s:/home/ubuntu/AFLplusplus:$AFL_HOME:g" compile_share.sh ./compile_share.sh 17 84 66 </code></pre> <p>You will get a warning from AFL++ headers, there is nothing to do about it, if you wish you can edit the AFL++ headers and fix it:</p> <pre><code>``` /home/debian/AFLplusplus/src/afl-performance.c: In function ‘get_binary_hash’: /home/debian/AFLplusplus/src/afl-performance.c:103:17: warning: pointer targets in passing argument 1 of ‘open’ differ in signedness [-Wpointer-sign] 103 | int fd = open(fn, O_RDONLY); | ^~ | | | u8 * {aka unsigned char *} In file included from /usr/aarch64-linux-gnu/include/fcntl.h:301, from /home/debian/AFLplusplus/include/afl-fuzz.h:58, from /home/debian/AFLplusplus/src/afl-performance.c:2: /usr/aarch64-linux-gnu/include/bits/fcntl2.h:41:19: note: expected ‘const char *’ but argument is of type ‘u8 *’ {aka ‘unsigned char *’} 41 | open (const char *__path, int __oflag, ...) | ~~~~~~~~~~~~^~~~~~ /home/debian/AFLplusplus/src/afl-performance.c: In function ‘sha1_hex’: /home/debian/AFLplusplus/src/afl-performance.c:407:10: warning: pointer targets in returning ‘u8 *’ {aka ‘unsigned char *’} from a function with return type ‘char *’ differ in signedness [-Wpointer-sign] 407 | return hex; | ^~~ ```</code></pre> <p>At this point, we have the tools up and the experimental data ready for fuzzing and testing.</p> <p> </p> <p> </p> <h3>2.3 Differential Testing of Input Corpus Seeds (Section V.A. & Table II):</h3> <p>Testing is usually against the most recent version. To get gem5 built from tree, run this code:</p> <pre><code>cd /home/debian git clone https://github.com/gem5/gem5.git cd gem5 pip install -r requirements.txt scons build/ARM/gem5.opt -j 4<br>cd /home/debian</code></pre> <p>To run a single test:</p> <pre><code>/home/debian/gem5/build/ARM/gem5.opt /home/debian/SearchGEM5/hello-custom-binary-Ex.py --isa ARM --input /home/debian/experiment/CodeBooga-cmin/input/test_input_645596930837886.txt</code></pre> <p>You can try different test inputs by changing the file after the --input. To run a single comparison between native and simulation, you run:</p> <pre><code>./2-test-input.sh /home/debian/experiment/CodeBooga-cmin/input/test_input_645596930837886.txt</code></pre> <p>To run the whole process for a folder, and then check the results later, you run the differential testing script:</p> <pre><code>./1-run-diff-test.sh /home/debian/experiment/CodeBooga-cmin/input > LLM-diff-testing-CodeBooga.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Llama-cmin/input > LLM-diff-testing-Llama.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Magicoder-cmin/input > LLM-diff-testing-Magicoder.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Phi-cmin/input > LLM-diff-testing-Phi.log 2>&1 & ./1-run-diff-test.sh /home/debian/experiment/TinyLlama-cmin/input > LLM-diff-testing-TinyLlama.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/gpt3.5-new-cmin/input > LLM-diff-testing-gpt3.5-new.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/gpt3.5-old-cmin/input > LLM-diff-testing-gpt3.5-old.log 2>&1</code></pre> <p><strong>==> This will give the results for Table II (page 7) after manual analysis of the logs.</strong></p> <p>These logs will later flag any issue found in ARM native vs simulation, e.g.:</p> <pre><code><strong>Simulation run: <br></strong></code><code>``` Global frequency set at 1000000000000 ticks per second warn: No dot file generated. Please install pydot to generate the dot file and pdf. src/mem/dram_interface.cc:690: warn: DRAM device capacity (8192 Mbytes) does not match the address range assigned (32 Mbytes) src/base/statistics.hh:279: warn: One of the stats is a legacy stat. Legacy stat is a stat that does not belong to any statistics::Group. Legacy stat is deprecated. src/base/remote_gdb.cc:418: warn: Sockets disabled, not accepting gdb connections gem5 Simulator System. https://www.gem5.org gem5 is copyrighted software; use the --copyright option for details. gem5 version 24.0.0.1 gem5 compiled Sep 5 2024 16:37:09 gem5 started Oct 9 2024 09:30:14 gem5 executing on karl.cs.ucl.ac.uk, pid 2528401 command line: /home/kevenmen/gem5/build/ARM/gem5.opt /home/kevenmen/SearchGEM5/hello-custom-binary-Ex.py --isa ARM --input /home/kevenmen/experiment-7/CodeBooga-cmin/output-r_5/default/queue/id:000003,time:0,execs:0,orig:id:000918,src:000589+000058,time:55938499,execs:11247,op:cm-gem5c-types.so,pos:0 info: Standard input is not a terminal, disabling listeners. src/sim/simulate.cc:199: info: Entering event queue @ 0. Starting simulation... src/arch/arm/insts/pseudo.cc:174: warn: instruction 'bti' unimplemented src/sim/mem_state.cc:448: info: Increasing stack size by one page. src/base/loader/symtab.cc:95: warn: Cannot insert a new symbol table due to name collisions. Adding prefix to each symbol's name can resolve this issue. src/sim/syscall_emul.cc:74: warn: ignoring syscall mprotect(...) src/base/loader/symtab.cc:95: warn: Cannot insert a new symbol table due to name collisions. Adding prefix to each symbol's name can resolve this issue. src/sim/syscall_emul.cc:74: warn: ignoring syscall set_robust_list(...) src/sim/syscall_emul.cc:85: warn: ignoring syscall rseq(...) (further warnings will be suppressed) src/sim/syscall_emul.cc:74: warn: ignoring syscall mprotect(...) src/sim/syscall_emul.cc:74: warn: ignoring syscall mprotect(...) src/sim/syscall_emul.cc:74: warn: ignoring syscall mprotect(...) " is neither a letter nor a digit H is a letter e is a letter l is a letter l is a letter o is a letter 1 is a digit 2 is a digit 3 is a digit " is neither a letter nor a digit Exiting @ tick 5367044250 because exiting with last active thread context. ``` <strong>vs the native run:</strong> ``` " is neither a letter nor a digit H is a letter e is a letter l is a letter l is a letter o is a letter 1 is a digit 2 is a digit 3 is a digit " is neither a letter nor a digit ```<br>In this case, but are giving the same output and hence on bug exposed.</code></pre> <h3> </h3> <p> </p> <h3>2.4 Fuzzing with AFL++ and SearchSYS Custom Mutators (Section V.B. & Table III):</h3> <p>You can run a single execution that way:</p> <pre><code># Define: cm_path="/home/debian/SearchGEM5/src/custom_mutators" corpus_folder="/home/debian/experiment/gpt3.5-old-cmin/" execgem5=/home/debian/gem5-ssbse-challenge-2023/build/ARM/gem5.opt # Run AFL AFL_NO_AFFINITY=1 AFL_SHUFFLE_QUEUE=1 AFL_CUSTOM_MUTATOR_ONLY=1 AFL_DUMB_FORKSRV=1 AFL_FAST_CAL=1 AFL_IGNORE_PROBLEMS=1 \ AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES=1 AFL_SKIP_BIN_CHECK=1 AFL_MAP_SIZE=1200000 \ AFL_CUSTOM_MUTATOR_LIBRARY="$cm_path/cm-gem5c.so;$cm_path/cm-gem5c-bin.so;$cm_path/cm-gem5c-types.so" \ /home/debian/AFLplusplus/afl-fuzz -m 50000 -t 99000 -i $corpus_folder/input/ -o $corpus_folder/output-fuzzed/ -- $execgem5 \ /home/debian/SearchGEM5/hello-custom-binary-Ex.py --isa ARM --input @@</code><br><br>You need then to check the custom mutators loaded okay:</pre> <pre><code>``` [+] Enabled environment variable AFL_IGNORE_PROBLEMS with value 1 [+] Enabled environment variable AFL_FAST_CAL with value 1 [+] Enabled environment variable AFL_CUSTOM_MUTATOR_LIBRARY with value /home/debian/SearchGEM5/src/custom_mutators/cm-gem5c.so;/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-bin.so;/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-types.so [+] Enabled environment variable AFL_SKIP_BIN_CHECK with value 1 [+] Enabled environment variable AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES with value 1 [+] Enabled environment variable AFL_DUMB_FORKSRV with value 1 [+] Enabled environment variable AFL_NO_AFFINITY with value 1 [+] Enabled environment variable AFL_CUSTOM_MUTATOR_ONLY with value 1 afl-fuzz++4.31a based on afl by Michal Zalewski and a large online community ... [*] Loading custom mutator library from '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c.so'... [+] Found 'afl_custom_mutator'. [+] Found 'afl_custom_fuzz_count'. ... [+] Custom mutator '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c.so' installed successfully. [*] Loading custom mutator library from '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-bin.so'... [+] Found 'afl_custom_mutator'. [+] Found 'afl_custom_fuzz_count'. ... [+] Custom mutator '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-bin.so' installed successfully. [*] Loading custom mutator library from '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-types.so'... [+] Found 'afl_custom_mutator'. [+] Found 'afl_custom_fuzz_count'. [*] optional symbol 'afl_custom_post_process' not found. [*] optional symbol 'afl_custom_init_trim' not found. [*] optional symbol 'afl_custom_trim' not found. [*] optional symbol 'afl_custom_post_trim' not found. [*] optional symbol 'afl_custom_havoc_mutation' not found. [*] optional symbol 'afl_custom_havoc_mutation_probability' not found. [*] optional symbol 'afl_custom_queue_get' not found. [*] optional symbol 'afl_custom_splice_optout' not found. [*] optional symbol 'afl_custom_fuzz_send' not found. [*] optional symbol 'afl_custom_post_run' not found. [*] optional symbol 'afl_custom_queue_new_entry' not found [*] optional symbol 'afl_custom_describe' not found. [+] Custom mutator '/home/debian/SearchGEM5/src/custom_mutators/cm-gem5c-types.so' installed successfully. ```</code></pre> <p>and if it uploaded okay, you will see this screen in the interactive mode:</p> <pre><code>[*] Attempting dry run with 'id:000441,time:0,execs:0,orig:zero-struct-2.txt'... [!] WARNING: instability detected during calibration len = 67, map size = 41474, exec speed = 3356882 us, hash = fef2b4ed27f2d5fe Results of dry-run 0[!] WARNING: Instrumentation output varies across runs. [!] WARNING: Skipped 1 test cases (0.23%) due to timeouts or crashes. [+] All test cases processed. [!] WARNING: The target binary is pretty slow! See docs/fuzzing_in_depth.md#i-improve-the-speed [!] WARNING: Some test cases look useless. Consider using a smaller set. [!] WARNING: You probably have far too many input files! Consider trimming down. [+] Here are some useful stats: Test case count : 64 favored, 255 variable, 5 ignored, 442 total Bitmap range : 27891 to 42022 bits (average: 41556.14 bits) Exec timing : 1.21M to 44.6M us (average: 4.30M us) [*] -t option specified. We'll use an exec timeout of 99000 ms. [+] All set and ready to roll! AFL ++4.31a {default} (...ssbse-challenge-2023/build/ARM/gem5.opt) [explore] ┌─ process timing ────────────────────────────────────┬─ overall results ────┐ │ run time : 0 days, 2 hrs, 39 min, 26 sec │ cycles done : 0 │ │ last new find : 0 days, 0 hrs, 6 min, 46 sec │ corpus count : 575 │ │last saved crash : 0 days, 0 hrs, 11 min, 21 sec │saved crashes : 40 │ │ last saved hang : none seen yet │ saved hangs : 0 │ ├─ cycle progress ─────────────────────┬─ map coverage┴──────────────────────┤ │ now processing : 46.0 (8.0%) │ map density : 3.79% / 4.10% │ │ runs timed out : 0 (0.00%) │ count coverage : 1.57 bits/tuple │ ├─ stage progress ─────────────────────┼─ findings in depth ─────────────────┤ │ now trying : cm-gem5c-types.so │ favored items : 86 (14.96%) │ │ stage execs : 63/66 (95.45%) │ new edges on : 119 (20.70%) │ │ total execs : 5744 │ total crashes : 144 (40 saved) │ │ exec speed : 0.18/sec (zzzz...) │ total tmouts : 0 (0 saved) │ ├─ fuzzing strategy yields ────────────┴─────────────┬─ item geometry ───────┤ │ bit flips : disabled (custom-mutator-only mode) │ levels : 6 │ │ byte flips : disabled (custom-mutator-only mode) │ pending : 555 │ │ arithmetics : disabled (custom-mutator-only mode) │ pend fav : 77 │ │ known ints : disabled (custom-mutator-only mode) │ own finds : 133 │ │ dictionary : n/a │ imported : 0 │ │havoc/splice : 0/0, 0/0 │ stability : 100.00% │ │py/custom/rq : unused, 150/858, unused, unused ├───────────────────────┘ │ trim/eff : 5.90%/361, disabled │ [cpu: 12%] └─ strategy: explore ────────── state: in progress ──┘[!] WARNING: >>-6A Odd size of register is: 75, 8</code></pre> <p> </p> <p><strong>Non-interactive mode and full run: </strong>The results in the paper are created with batch runs. We need to run each repeat for all initial corpora to get the full results for Table III:</p> <pre><code># Repeat 1 ./Experiment-ARM-24h.sh CodeBooga-cmin 1 ./Experiment-ARM-24h.sh gpt3.5-new-cmin 1 ./Experiment-ARM-24h.sh gpt3.5-old-cmin 1 ./Experiment-ARM-24h.sh Llama-cmin 1 ./Experiment-ARM-24h.sh Magicoder-cmin 1 ./Experiment-ARM-24h.sh Phi-cmin 1 ./Experiment-ARM-24h.sh TinyLlama-cmin 1 # Repeat 3 ./Experiment-ARM-24h.sh CodeBooga-cmin 3 ./Experiment-ARM-24h.sh gpt3.5-new-cmin 3 ./Experiment-ARM-24h.sh gpt3.5-old-cmin 3 ./Experiment-ARM-24h.sh Llama-cmin 3 ./Experiment-ARM-24h.sh Magicoder-cmin 3 ./Experiment-ARM-24h.sh Phi-cmin 3 ./Experiment-ARM-24h.sh TinyLlama-cmin 3 # Repeat 4 ./Experiment-ARM-24h.sh CodeBooga-cmin 1 ./Experiment-ARM-24h.sh gpt3.5-new-cmin 1 ./Experiment-ARM-24h.sh gpt3.5-old-cmin 1 ./Experiment-ARM-24h.sh Llama-cmin 1 ./Experiment-ARM-24h.sh Magicoder-cmin 1 ./Experiment-ARM-24h.sh Phi-cmin 1 ./Experiment-ARM-24h.sh TinyLlama-cmin 1 # Repeat 4 ./Experiment-ARM-24h.sh CodeBooga-cmin 4 ./Experiment-ARM-24h.sh gpt3.5-new-cmin 4 ./Experiment-ARM-24h.sh gpt3.5-old-cmin 4 ./Experiment-ARM-24h.sh Llama-cmin 4 ./Experiment-ARM-24h.sh Magicoder-cmin 4 ./Experiment-ARM-24h.sh Phi-cmin 4 ./Experiment-ARM-24h.sh TinyLlama-cmin 4 # Repeat 5 ./Experiment-ARM-24h.sh CodeBooga-cmin 5 ./Experiment-ARM-24h.sh gpt3.5-new-cmin 5 ./Experiment-ARM-24h.sh gpt3.5-old-cmin 5 ./Experiment-ARM-24h.sh Llama-cmin 5 ./Experiment-ARM-24h.sh Magicoder-cmin 5 ./Experiment-ARM-24h.sh Phi-cmin 5 ./Experiment-ARM-24h.sh TinyLlama-cmin 5</code></pre> <p>From each output folder, we analysed the size of queue, crash and hang folders. Further statistics are in fuzzer_stats file. These scripts ensure the run for 24 hours per repeat and input corpus.</p> <p><strong>Note that, as fuzzing is random, the results in Table III can be slightly different between two repeats of the same initial corpus. </strong></p> <p> </p> <p> </p> <h3>2.5 Differential Testing of Fuzzed Corpus Seeds (Section V.B. & Table IV):</h3> <p>We use the same scripts here as in Section 2.3, but with the output fuzzed folders:</p> <pre><code># In the paper we analysed repeat 5, but you can change it to any: ./1-run-diff-test.sh /home/debian/experiment/CodeBooga-cmin/output-r_5/default/queue > Fuzzed-diff-testing-CodeBooga-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Llama-cmin/output-r_5/default/queue > Fuzzed-diff-testing-Llama-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Magicoder-cmin/output-r_5/default/queue > Fuzzed-diff-testing-Magicoder-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/Phi-cmin/output-r_5/default/queue > Fuzzed-diff-testing-Phi-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/TinyLlama-cmin/output-r_5/default/queue > Fuzzed-diff-testing-TinyLlama-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/gpt3.5-new-cmin/output-r_5/default/queue > Fuzzed-diff-testing-gpt3.5-new-5.log 2>&1 ./1-run-diff-test.sh /home/debian/experiment/gpt3.5-old-cmin/output-r_5/default/queue > Fuzzed-diff-testing-gpt3.5-old-5.log 2>&1</code></pre> <p>The analysis here and identify the bugs are done manually and require a system simulator expert.</p> <p> </p> <p> </p> <h2>3. Reproduce ICSE-SEIP 2025 Evaluation - Shorter Version and Statistics from Dataset:</h2> <p><strong>Available</strong>: the source code is at <a href="https://github.com/karineek/SearchGEM5/tree/main">https://github.com/karineek/SearchGEM5/tree/main</a> with the extra scripts for ARM and the instructions fully documented in this Zenodo Record in Sections 1 and 2. We added code.zip to this publicly accessible archival Zenodo record to meet the availability requirements.</p> <p><strong>Functional</strong>: The artifact associated with the research are documented fully in sections 2.3, 2.4 and 2.5. Due to its time requirements, we present a shorter version in this section with additional scripts to allow easy quick reproduction of the shorter version's results. We added expected results to help validate the results and noted where the results can be different (e.g. when fuzzing for 2 hours instead of 24 hours).</p> <p><strong>Reusable</strong>: Section 2 and the documentation at <a href="https://github.com/karineek/SearchGEM5/tree/main" target="_blank" rel="noopener">https://github.com/karineek/SearchGEM5/tree/main</a> allow customisation to other targets and editing/adding to the code of SearchSYS. We attached a copy of the GitHub repository here, to ensure the documentation and code are in a single place.</p> <p> </p> <h3>3.1 Table I - Shortened Version</h3> <p>We added the tokens from the <a href="https://github.com/karineek/SearchGEM5/blob/main/src/main/java/LLMTokensOptions.java">GitHub of SearchSYS</a> into four files and filtered out duplicate tokens.</p> <p>To reproduce the counters of Table I:</p> <pre><code>wget "https://zenodo.org/records/14724553/files/TABLE-1.zip" # Get the data and scripts<br>unzip TABLE-1.zip<br>cd TABLE-1<br>./table-1.sh</code><br><br>This should give you:</pre> <pre><code>$ ./table-1.sh >> Token-1 Examples ID_1.log:1:"Scalar Optimizations", ID_1.log:2:"Dead Code Elimination", ID_1.log:3:"Constant Folding", >> Token-2 Examples ID_2.log:32:"Sema", ID_2.log:33:"Serialization", ID_2.log:34:"Parse", ID_2.log:35:"Lex", ID_2.log:36:"AST", >> Token-3 Examples ID_3.log:86:"C Program to Sort an Array using Merge Sort", ID_3.log:58:"Calendar Year in Different Formats", ID_3.log:176:"input include several arguments", >> Token-4 Examples ID_4.log: 343:AND operators, ID_4.log: 779:cimagl function, ID_4.log: 1132:EOF, ID_4.log: 1915:locale, ID_4.log: 2328:pow, ID_4.log: 2614:SCNiMAX, 26 ID_1.log 36 ID_2.log 192 ID_3.log 3480 ID_4.log 3734 total</code></pre> <h3> </h3> <h3>3.2 Table II and Table IV- Shortened Version</h3> <p>We run differential testing to only one of the sets, A, as it is the smallest (and hence should finish quickly). We can then reproduce the table for B-G by using the logs in the Excel file.</p> <p>You will need two scripts for this stage:</p> <pre><code>cd /home/debian/<br>wget "https://zenodo.org/records/14724553/files/1-run-diff-test.sh" <br>wget "https://zenodo.org/records/14724553/files/2-test-input.sh"<br>wget "https://zenodo.org/records/14724553/files/ICSE-SEIP-Bugs-READY.xlsx" ICSE-SEIP-Bugs-READY.xlsx" <br>chmod 777 1-run-diff-test.sh</code><code><br>chmod 777 2-test-input.sh</code></pre> <p>We need the newest version of gem5 and compare it to native run. If you have not yet built gem5, run:</p> <pre><code>cd /home/debian git clone https://github.com/gem5/gem5.git cd gem5 pip install -r requirements.txt scons build/ARM/gem5.opt -j 4<br>cd /home/debian</code></pre> <p>To run a single test:</p> <pre><code>/home/debian/gem5/build/ARM/gem5.opt /home/debian/SearchGEM5/hello-custom-binary-Ex.py --isa ARM --input /home/debian/experiment/CodeBooga-cmin/input/test_input_645596930837886.txt</code></pre> <p>You can try different test inputs by changing the file after the --input. To run a single comparison between native and simulation, you run:</p> <pre><code>./2-test-input.sh /home/debian/experiment/CodeBooga-cmin/input/test_input_645596930837886.txt</code></pre> <p>We start with running differential testing on set A:<code><br><br>nohup ./1-run-diff-test.sh /home/debian/experiment/TinyLlama-cmin/input > LLM-diff-testing-TinyLlama.log 2>&1 &</code></p> <p><code>nohup ./1-run-diff-test.sh /home/debian/experiment/TinyLlama-cmin/output-r_5/default/queue > AFL-diff-testing-TinyLlama.log 2>&1 &</code></p> <p>Then we analyse this manually, as not every failure to simulate a binary is a bug (it can be that the fuzzing corrupted the binary, and the expected behaviour is NOT to run the simulation successfully). However, you can "grep" the error strings, as some of the errors will likely manifest during the fuzzing.</p> <p> </p> <h3>3.3 Table III - Shortened Version</h3> <p>We will do the following: run fuzzing for 1 hour with corpus TinyLlama and check there are results in the queue to see that the fuzzing generated some files. We can reproduce the table from the logs (the full experiments here shall take 24x5x7 hours, which is beyond the expectation of artifact evaluation).</p> <pre><code># Define:<br>cd /home/debian/ cm_path="/home/debian/SearchGEM5/src/custom_mutators" corpus_folder="/home/debian/experiment/TinyLlama-cmin/" execgem5=/home/debian/gem5-ssbse-challenge-2023/build/ARM/gem5.opt # Run AFL AFL_NO_AFFINITY=1 AFL_SHUFFLE_QUEUE=1 AFL_CUSTOM_MUTATOR_ONLY=1 AFL_DUMB_FORKSRV=1 AFL_FAST_CAL=1 AFL_IGNORE_PROBLEMS=1 \ AFL_I_DONT_CARE_ABOUT_MISSING_CRASHES=1 AFL_SKIP_BIN_CHECK=1 AFL_MAP_SIZE=1200000 \ AFL_CUSTOM_MUTATOR_LIBRARY="$cm_path/cm-gem5c.so;$cm_path/cm-gem5c-bin.so;$cm_path/cm-gem5c-types.so" \ /home/debian/AFLplusplus/afl-fuzz -m 50000 -t 99000 -i $corpus_folder/input/ -o $corpus_folder/output-fuzzed/ -- $execgem5 \ /home/debian/SearchGEM5/hello-custom-binary-Ex.py --isa ARM --input @@ </code><br>You should get the same interface as in 2.4 with version:<br><code>AFL ++4.31a {default} (...ssbse-challenge-2023/build/ARM/gem5.opt) [explore]... </code><br><br>Then to reproduce the numbers, you can mine the logs of the original fuzzing campaign. </pre> <p>You will need one script for this stage:</p> <pre><code>cd /home/debian/<br>wget "https://zenodo.org/records/14721385/files/3-run-stat-table-3.sh" <br>chmod 777 3-run-stat-table-3.sh</code></pre> <pre>and the logs:<br><code>mkdir experiments-prev<br>cd experiments-prev<br>cp /home/debian/experiment/* .<br>wget "https://zenodo.org/records/14724553/files/round-1-output.tar.gz"<br>wget "https://zenodo.org/records/14724553/files/round-2-output.tar.gz"<br>wget "https://zenodo.org/records/14724553/files/round-3-output.tar.gz"<br>wget "https://zenodo.org/records/14724553/files/round-4-output.tar.gz"<br>wget "https://zenodo.org/records/14724553/files/round-5-output.tar.gz" </code><br><br>Then run:<br><code>tar -xf round-1-output.tar.gz<br>tar -xf round-2-output.tar.gz<br>tar -xf round-3-output.tar.gz<br>tar -xf round-4-output.tar.gz<br>tar -xf round-5-output.tar.gz<br>cd ..</code><code><br>./3-run-stat-table-3.sh experiments-prev</code><br><br>This should print:</pre> <pre><code>$ ./3-stat-table-3.sh Initial Corpus Size: 612 (Directory: CodeBooga-cmin/input) Initial Corpus Size: 17 (Directory: CodeBooga-cmin/output-r_1/default/crashes) Initial Corpus Size: 17 (Directory: CodeBooga-cmin/output-r_1/default/hangs) Initial Corpus Size: 905 (Directory: CodeBooga-cmin/output-r_1/default/queue) Initial Corpus Size: 3 (Directory: CodeBooga-cmin/output-r_2/default/crashes) Initial Corpus Size: 23 (Directory: CodeBooga-cmin/output-r_2/default/hangs) Initial Corpus Size: 685 (Directory: CodeBooga-cmin/output-r_2/default/queue) Initial Corpus Size: 12 (Directory: CodeBooga-cmin/output-r_3/default/crashes) Initial Corpus Size: 3 (Directory: CodeBooga-cmin/output-r_3/default/hangs) Initial Corpus Size: 842 (Directory: CodeBooga-cmin/output-r_3/default/queue) Initial Corpus Size: 7 (Directory: CodeBooga-cmin/output-r_4/default/crashes) Initial Corpus Size: 11 (Directory: CodeBooga-cmin/output-r_4/default/hangs) Initial Corpus Size: 728 (Directory: CodeBooga-cmin/output-r_4/default/queue) Initial Corpus Size: 17 (Directory: CodeBooga-cmin/output-r_5/default/crashes) Initial Corpus Size: 12 (Directory: CodeBooga-cmin/output-r_5/default/hangs) Initial Corpus Size: 922 (Directory: CodeBooga-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 703 (Directory: gpt3.5-new-cmin/input) Initial Corpus Size: 10 (Directory: gpt3.5-new-cmin/output-r_1/default/crashes) Initial Corpus Size: 2 (Directory: gpt3.5-new-cmin/output-r_1/default/hangs) Initial Corpus Size: 839 (Directory: gpt3.5-new-cmin/output-r_1/default/queue) Initial Corpus Size: 11 (Directory: gpt3.5-new-cmin/output-r_2/default/crashes) Initial Corpus Size: 8 (Directory: gpt3.5-new-cmin/output-r_2/default/hangs) Initial Corpus Size: 898 (Directory: gpt3.5-new-cmin/output-r_2/default/queue) Initial Corpus Size: 14 (Directory: gpt3.5-new-cmin/output-r_3/default/crashes) Initial Corpus Size: 9 (Directory: gpt3.5-new-cmin/output-r_3/default/hangs) Initial Corpus Size: 877 (Directory: gpt3.5-new-cmin/output-r_3/default/queue) Initial Corpus Size: 12 (Directory: gpt3.5-new-cmin/output-r_4/default/crashes) Initial Corpus Size: 5 (Directory: gpt3.5-new-cmin/output-r_4/default/hangs) Initial Corpus Size: 855 (Directory: gpt3.5-new-cmin/output-r_4/default/queue) Initial Corpus Size: 11 (Directory: gpt3.5-new-cmin/output-r_5/default/crashes) Initial Corpus Size: 11 (Directory: gpt3.5-new-cmin/output-r_5/default/hangs) Initial Corpus Size: 886 (Directory: gpt3.5-new-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 442 (Directory: gpt3.5-old-cmin/input) Initial Corpus Size: 43 (Directory: gpt3.5-old-cmin/output-r_1/default/crashes) Initial Corpus Size: 2 (Directory: gpt3.5-old-cmin/output-r_1/default/hangs) Initial Corpus Size: 964 (Directory: gpt3.5-old-cmin/output-r_1/default/queue) Initial Corpus Size: 52 (Directory: gpt3.5-old-cmin/output-r_2/default/crashes) Initial Corpus Size: 5 (Directory: gpt3.5-old-cmin/output-r_2/default/hangs) Initial Corpus Size: 988 (Directory: gpt3.5-old-cmin/output-r_2/default/queue) Initial Corpus Size: 57 (Directory: gpt3.5-old-cmin/output-r_3/default/crashes) Initial Corpus Size: 6 (Directory: gpt3.5-old-cmin/output-r_3/default/hangs) Initial Corpus Size: 934 (Directory: gpt3.5-old-cmin/output-r_3/default/queue) Initial Corpus Size: 49 (Directory: gpt3.5-old-cmin/output-r_4/default/crashes) Initial Corpus Size: 0 (Directory: gpt3.5-old-cmin/output-r_4/default/hangs) Initial Corpus Size: 891 (Directory: gpt3.5-old-cmin/output-r_4/default/queue) Initial Corpus Size: 47 (Directory: gpt3.5-old-cmin/output-r_5/default/crashes) Initial Corpus Size: 1 (Directory: gpt3.5-old-cmin/output-r_5/default/hangs) Initial Corpus Size: 957 (Directory: gpt3.5-old-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 613 (Directory: Llama-cmin/input) Initial Corpus Size: 25 (Directory: Llama-cmin/output-r_1/default/crashes) Initial Corpus Size: 11 (Directory: Llama-cmin/output-r_1/default/hangs) Initial Corpus Size: 1014 (Directory: Llama-cmin/output-r_1/default/queue) Initial Corpus Size: 19 (Directory: Llama-cmin/output-r_2/default/crashes) Initial Corpus Size: 19 (Directory: Llama-cmin/output-r_2/default/hangs) Initial Corpus Size: 939 (Directory: Llama-cmin/output-r_2/default/queue) Initial Corpus Size: 7 (Directory: Llama-cmin/output-r_3/default/crashes) Initial Corpus Size: 14 (Directory: Llama-cmin/output-r_3/default/hangs) Initial Corpus Size: 929 (Directory: Llama-cmin/output-r_3/default/queue) Initial Corpus Size: 15 (Directory: Llama-cmin/output-r_4/default/crashes) Initial Corpus Size: 18 (Directory: Llama-cmin/output-r_4/default/hangs) Initial Corpus Size: 857 (Directory: Llama-cmin/output-r_4/default/queue) Initial Corpus Size: 16 (Directory: Llama-cmin/output-r_5/default/crashes) Initial Corpus Size: 18 (Directory: Llama-cmin/output-r_5/default/hangs) Initial Corpus Size: 909 (Directory: Llama-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 719 (Directory: Magicoder-cmin/input) Initial Corpus Size: 14 (Directory: Magicoder-cmin/output-r_1/default/crashes) Initial Corpus Size: 8 (Directory: Magicoder-cmin/output-r_1/default/hangs) Initial Corpus Size: 1012 (Directory: Magicoder-cmin/output-r_1/default/queue) Initial Corpus Size: 17 (Directory: Magicoder-cmin/output-r_2/default/crashes) Initial Corpus Size: 13 (Directory: Magicoder-cmin/output-r_2/default/hangs) Initial Corpus Size: 1010 (Directory: Magicoder-cmin/output-r_2/default/queue) Initial Corpus Size: 15 (Directory: Magicoder-cmin/output-r_3/default/crashes) Initial Corpus Size: 13 (Directory: Magicoder-cmin/output-r_3/default/hangs) Initial Corpus Size: 974 (Directory: Magicoder-cmin/output-r_3/default/queue) Initial Corpus Size: 11 (Directory: Magicoder-cmin/output-r_4/default/crashes) Initial Corpus Size: 8 (Directory: Magicoder-cmin/output-r_4/default/hangs) Initial Corpus Size: 958 (Directory: Magicoder-cmin/output-r_4/default/queue) Initial Corpus Size: 13 (Directory: Magicoder-cmin/output-r_5/default/crashes) Initial Corpus Size: 13 (Directory: Magicoder-cmin/output-r_5/default/hangs) Initial Corpus Size: 896 (Directory: Magicoder-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 366 (Directory: Phi-cmin/input) Initial Corpus Size: 18 (Directory: Phi-cmin/output-r_1/default/crashes) Initial Corpus Size: 16 (Directory: Phi-cmin/output-r_1/default/hangs) Initial Corpus Size: 707 (Directory: Phi-cmin/output-r_1/default/queue) Initial Corpus Size: 13 (Directory: Phi-cmin/output-r_2/default/crashes) Initial Corpus Size: 26 (Directory: Phi-cmin/output-r_2/default/hangs) Initial Corpus Size: 707 (Directory: Phi-cmin/output-r_2/default/queue) Initial Corpus Size: 18 (Directory: Phi-cmin/output-r_3/default/crashes) Initial Corpus Size: 18 (Directory: Phi-cmin/output-r_3/default/hangs) Initial Corpus Size: 704 (Directory: Phi-cmin/output-r_3/default/queue) Initial Corpus Size: 12 (Directory: Phi-cmin/output-r_4/default/crashes) Initial Corpus Size: 18 (Directory: Phi-cmin/output-r_4/default/hangs) Initial Corpus Size: 568 (Directory: Phi-cmin/output-r_4/default/queue) Initial Corpus Size: 16 (Directory: Phi-cmin/output-r_5/default/crashes) Initial Corpus Size: 10 (Directory: Phi-cmin/output-r_5/default/hangs) Initial Corpus Size: 721 (Directory: Phi-cmin/output-r_5/default/queue) --------------------------------- Initial Corpus Size: 206 (Directory: TinyLlama-cmin/input) Initial Corpus Size: 24 (Directory: TinyLlama-cmin/output-r_1/default/crashes) Initial Corpus Size: 9 (Directory: TinyLlama-cmin/output-r_1/default/hangs) Initial Corpus Size: 770 (Directory: TinyLlama-cmin/output-r_1/default/queue) Initial Corpus Size: 36 (Directory: TinyLlama-cmin/output-r_2/default/crashes) Initial Corpus Size: 9 (Directory: TinyLlama-cmin/output-r_2/default/hangs) Initial Corpus Size: 802 (Directory: TinyLlama-cmin/output-r_2/default/queue) Initial Corpus Size: 34 (Directory: TinyLlama-cmin/output-r_3/default/crashes) Initial Corpus Size: 12 (Directory: TinyLlama-cmin/output-r_3/default/hangs) Initial Corpus Size: 748 (Directory: TinyLlama-cmin/output-r_3/default/queue) Initial Corpus Size: 29 (Directory: TinyLlama-cmin/output-r_4/default/crashes) Initial Corpus Size: 11 (Directory: TinyLlama-cmin/output-r_4/default/hangs) Initial Corpus Size: 783 (Directory: TinyLlama-cmin/output-r_4/default/queue) Initial Corpus Size: 31 (Directory: TinyLlama-cmin/output-r_5/default/crashes) Initial Corpus Size: 9 (Directory: TinyLlama-cmin/output-r_5/default/hangs) Initial Corpus Size: 634 (Directory: TinyLlama-cmin/output-r_5/default/queue) --------------------------------- </code></pre> <h3>Note that in this specific dataset, some of the fuzzing ran over 24 hours, hence only the files within the time frame of 24 hours were added to the final calculation in Table III. Hence, there are some discrepancies, e.g. between TinyLlama reported in the paper, queue size, which is 737, vs the result from the script, which is 747. Those sets that ran over 24-hour fuzzing were manually re-calculated, which you can see in the file: <a href="https://zenodo.org/records/14724553/files/ICSE-SEIP-Bugs-READY.xlsx" target="_blank" rel="noopener">ICSE-SEIP-Bugs-READY.xlsx.</a></h3> <h3>==</h3> <h3>3.3.1 Troubleshooting: Error in 3-run-stat-table-3.sh:</h3> <p>Sometimes script 3 may not work due to the specific structure of the folder it requires.</p> <h4><strong>Error:</strong></h4> <p>When running the script <code>./3-run-stat-table-3.sh experiments-prev</code>, you encounter errors like:</p> <p><code>$ ./3-run-stat-table-3.sh experiments-prev<br>Directory: CodeBooga-cmin/input does not exist.<br>---------------------------------<br>Directory: gpt3.5-new-cmin/input does not exist.<br>---------------------------------<br>Directory: gpt3.5-old-cmin/input does not exist.<br>---------------------------------<br>Directory: Llama-cmin/input does not exist.<br>---------------------------------<br>Directory: Magicoder-cmin/input does not exist.<br>---------------------------------<br>Directory: Phi-cmin/input does not exist.<br>---------------------------------<br>Directory: TinyLlama-cmin/input does not exist.<br>---------------------------------</code></p> <h4><strong>Cause:</strong></h4> <p>This error occurs because the required directories and files have not been properly copied into the <code>experiments-prev</code> folder.</p> <h4><strong>Solution:</strong></h4> <p>Follow these steps to fix the error.<br><br></p> <pre><strong>1. Clean Up and Restart the Setup</strong><br>Run the following commands to remove the existing <code>experiments-prev</code> directory and create a fresh copy:</pre> <pre><code>rm -rf experiments-prev # Clean up the existing directory mkdir experiments-prev cd experiments-prev</code></pre> <p><strong>2. Copy Required Files</strong></p> <p>Copy all necessary files from <code>/home/debian/experiment/</code> into the newly created <code>experiments-prev</code> directory:</p> <pre><code>cp /home/debian/experiment/* .</code></pre> <p><strong>3. Verify the Files Have Been Copied</strong> </p> <p>Run the following command to ensure the required files exist</p> <pre><code>ls -l /home/debian/experiment/CodeBooga-cmin/input</code></pre> <ul> <li><strong>Check:</strong> Confirm you can see the files you have just copied.</li> <li>If the files are not visible, ensure the source directory <code>/home/debian/experiment/</code> contains all the required files. If not return to Section 1 to fix it.</li> </ul> <p><strong>4. Download Additional Data Files</strong></p> <p>Download the necessary <code>round-*</code> output files:</p> <pre><code>wget "https://zenodo.org/records/14724553/files/round-1-output.tar.gz" wget "https://zenodo.org/records/14724553/files/round-2-output.tar.gz" wget "https://zenodo.org/records/14724553/files/round-3-output.tar.gz" wget "https://zenodo.org/records/14724553/files/round-4-output.tar.gz" wget "https://zenodo.org/records/14724553/files/round-5-output.tar.gz"</code></pre> <p><strong>5. Extract the Data Files</strong> </p> <p>Extract the contents of the downloaded <code>.tar.gz</code> files:</p> <pre><code>tar -xf round-1-output.tar.gz<br>tar -xf round-2-output.tar.gz<br>tar -xf round-3-output.tar.gz<br>tar -xf round-4-output.tar.gz<br>tar -xf round-5-output.tar.gz<br>cd ..</code><code><br>./3-run-stat-table-3.sh experiments-prev</code></pre> <h3>==</h3>
ReSym Artifact
<pre>Our main artifact repository: <a href="https://github.com/lt-asset/resym">https://github.com/lt-asset/resym</a><br><br>We are activaly maintaining/updating our artifacts. Please make sure you downloaded the latest version. <br><br>[Update log]<br>- 04/05/2025: update prediction results in ReSym_data.zip. Results updated in fielddecoder_pred.jsonl. Only metadata changes in vardecoder_pred.jsonl. </pre>
Artifact Group Frequency at Lubrza 10
<p>Raw data analyzed in the article "“DARK MATTER” OF HUNTER-GATHERERS ARCHAEOLOGY: HOW MANY ARTIFACTS DOES A SITE LACK?" by Aleksandr Diachenko and Iwona Sobkowiak-Tabaka (to appear in Archaeometry). </p>
A Stochastic Extension of Stateflow - Artifact
<p>Artifact for reproducing the results Vizualised in Figure 5 of the paper "A stochastic extension of Stateflow" for the conference the 13th ACM/SPEC International Conference on Performance Engineering (ICPE). </p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.