Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
1,549
datasets available to search
ShareScore release 0.9.0
Dataset results
1,549 results for “benchmarks”
MELA Dataset: A Benchmark for Mediastinal Lesion Analysis (Test Set)
<p>MELA dataset is a benchmark for developing algorithms on mediastinal lesion analysis. We hope this large-scale dataset could facilitate the research and application of automatic mediastinal lesion detection and diagnosis. </p> <p>MELA dataset contains 1100 CT scans collected from patients with one or more lesions in the mediastinum. The MELA dataset is split into a subset of 770 CT scans for training, a subset of 110 CT scans for validation, and a test set of 220 CT scans for evaluation.</p> <p>This is the Test Set of MELA dataset, including 220 CTs. Files include:</p> <ol> <li>Test1.zip: 110 CTs in NII format (nii.gz).</li> <li>Test2.zip: 110 CTs in NII format (nii.gz).</li> </ol>
PIGSFLI Benchmarking Dataset
<p>This repository contains all raw unprocessed quantum Monte Carlo data utilized in testing and benchmarking the <code>pigsfli</code> code available at <a href="https://github.com/DelMaestroGroup/pigsfli ">https://github.com/DelMaestroGroup/pigsfli</a>.</p> <p><strong>Directory Names</strong></p> <p>Directory names are encoded according to the following rule:</p> <pre>{dimension}D_{linear_size}_{total_particles}_{partition size}_{interactionpotential}_{tunneling parameter}_{beta}_{number of bins}</pre> <p><strong>File Names</strong></p> <ol> <li>Contains the state of the RNG: <pre>1D_16_16_8_7.071100_1.000000_12.000000_10001_rng-state_0_square_2.dat {dimension}D_{linear_size}_{total_particles}_{partition size}_{interaction potential}_{tunneling parameter}_{beta}_{number of bins}_rng-state_{seed}_{geometry of subregion}_{number_of_replicas}.dat </pre> </li> <li>Contains the state of the system: <pre>1D_16_16_8_7.071100_1.000000_12.000000_10001_system-state_0_square_2.dat {dimension}D_{linear_size}_{total_particles}_{partition size}_{interaction potential}_{tunneling parameter}_{beta}_{number of bins}_system-state_{seed}_{geometry of subregion}_{number_of_replicas}.dat</pre> </li> <li>Number of times each possible number of swapped sites was measured (each column is a number of swaps ranging from 0 to ℓ): <pre>1D_16_16_8_7.071100_1.000000_12.000000_10001_SWAP_137_square.dat {dimension}D_{linear_size}_{total_particles}_{partition size}_{interaction potential}_{tunneling parameter}_{beta}_{number of bins}_SWAP_{seed}_{geometry of subregion}.dat</pre> </li> <li>For fixed number of swapped sites (mA), how many times each possible local particle number was measured (columns range from n=0,...,N): <pre>1D_8_8_4_3.300000_1.000000_0.600000_10000_SWAPn-mA4_42_square.dat {dimension}D_{linear_size}_{total_particles}_{partition size}_{interaction potential}_{tunneling parameter}_{beta}_{number of bins}_SWAPn-mA{number of swapped sites}_{seed}_{geometry of subregion}.dat</pre> </li> <li>For fixed number of subregion sites (mA), how many times each possible local particle number was measured (columns range from n=0,...,N): <pre>1D_8_8_4_3.300000_1.000000_0.600000_10000_Pn-mA4_42_square.dat {dimension}D_{linear_size}_{total_particles}_{partition size}_{interaction potential}_{tunneling parameter}_{beta}_{number of bins}_Pn-mA{number of swapped sites}_{seed}_{geometry of subregion}.dat </pre> </li> <li>For fixed number of subregion sites (mA), how many times each possible local particle number was measured simultaneously on both replicas when there were no swapped sited (columns range from n=0,...,N): <pre> 1D_8_8_4_3.300000_1.000000_0.600000_10000_PnSquared-mA4_42_square.dat {dimension}D_{linear_size}_{total_particles}_{partition size}_{interaction potential}_{tunneling parameter}_{beta}_{number of bins}_PnSquared-mA{number of swapped sites}_{seed}_{geometry of subregion}.dat</pre> </li> <li>Kinetic energy: <pre> 1D_8_8_4_3.300000_1.000000_2.000000_10000_K_93_square.dat {dimension}D_{linear_size}_{total_particles}_{partition size}_{interaction potential}_{tunneling parameter}_{beta}_{number of bins}_K_{seed}_{geometry of subregion}.dat </pre> </li> <li>Potential energy: <pre>1D_8_8_4_3.300000_1.000000_2.000000_10000_V_93_square.dat {dimension}D_{linear_size}_{total_particles}_{partition size}_{interaction potential}_{tunneling parameter}_{beta}_{number of bins}_V_{seed}_{geometry of subregion}.dat</pre> </li> </ol>
CloudButton Serverless Benchmarks results (June 2022)
<p>Results of the <strong>CloudButton Serverless Benchmark</strong>. It includes plots with results of the <strong>Flops benchmark</strong> and the <strong>Storage benchmark</strong>. The first one evaluates compute power and scalability of cloud providers by running multiple compute intensive tasks concurrently. The second one evaluates the throughput of read and write operations to the object storage services of different cloud providers by running such operations in many tasks and concurrently.</p> <p>The benchmark has been run on 6 FaaS services (IBM Cloud Functions, AWS Lambda, Microsoft Azure Functions, Google Cloud Functions, Google Cloud Run, and Alibaba Aliyun Function Compute) and 5 Object Storage services (IBM Cloud Object Storage, AWS S3, Microsoft Azure Blob, Google Storage, Alibaba Aliyun Object Storage Service).</p>
Replication Package for "TriggerBench: A Performance Benchmark for Serverless Function Triggers"
<p>This replication package contains the code (`aws-triggers` and `azure-trigger`), data analysis scripts (`data-analysis`), and dataset (`data`) of the TriggerBench cross-provider serverless benchmark.</p> <p>It also bundles a customized extension of the `serverless-benchmarker` tool to automate and analyze serverless performance experiments.</p> <p><strong>TriggerBench</strong></p> <p>The Github repository <a href="https://github.com/joe4dev/trigger-bench/">joe4dev/trigger-bench</a> contains the last version of TriggerBench. This replication package describes the version for the paper "TriggerBench: A Performance Benchmark for Serverless Function Triggers".</p> <p>TriggerBench currently supports three triggers on AWS and eight triggers on Microsoft Azure.</p> <p><strong>Dataset</strong></p> <p>The `data/aws` and `data/azure` directories contain data from benchmark executions from April 2022.</p> <p>Each execution is a separate directory with a timestamp in the format `yyyy-mm-dd-HH-MM-SS` (e.g., `2022-04-15_21-58-52`) and contains the following files:</p> <ul> <li>`k6_metrics.csv`: Load generator HTTP client logs in CSV format (see [K6 docs](https://k6.io/docs/results-visualization/csv/))</li> <li>`sb_config.yml`: serverless benchmarker execution configuration including experiment label.</li> <li>`trigger.csv`: analyzer output CSV per trace. <ul> <li>`root_trace_id`: The trace id created by k6 and adopted by the invoker function</li> <li>`child_trace_id`: The trace id newly created by the receiver function if trace propagation is not supported (this is the case for most asynchronous triggers)</li> <li>`t1`-`t4`: Timestamps following the trace model (see paper)</li> <li>`t5`-`t9`: Additional timestamps for measuring timestamping overhead</li> <li>`coldstart_f1=True|False`: coldstart status for invoker (f1) and receiver (f2) functions</li> <li>`trace_ids.txt`: text file with each pair of `root_trace_id` and `child_trace_id` on a new line.</li> </ul> </li> <li>`traces.json`: raw trace JSON representation as retrieved from the provider tracing service. For AWS, see [X-Ray segment docs](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html). For Azure, see [Application Insights telemetry data model](https://docs.microsoft.com/en-us/azure/azure-monitor/app/data-model).</li> <li>`workload_options.json`: [K6 load scenario](https://k6.io/docs/using-k6/scenarios/) configuration.</li> </ul> <p><strong>Replicate Data Analysis</strong></p> <p><strong>Installation</strong></p> <p>1. Install [Python](https://www.python.org/downloads/) 3.10+</p> <p>2. Install Python dependencies `pip install -r requirements.txt`</p> <p><strong>Create Plots</strong></p> <p>1. Run `python plots.py` generates the plots and the statistical summaries presented in the paper.</p> <p>By default, the plots will be saved into a `plots` sub-directory.<br> An alternative output directory can be configured through the environment variable `PLOTS_PATH`.</p> <p>> Hint: For interactive development, we recommend the VSCode [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python) in [interactive mode](https://youtu.be/lwN4-W1WR84?t=107).</p> <p> </p> <p><strong>Replicate Cloud Experiments</strong></p> <p>The following experiment plan automates benchmarking experiments with different types workloads (constant and bursty).</p> <p>This generates a new dataset in the same format as described above.</p> <p> </p> <ol> <li> Set up a load generator as vantage point following the description in [LOADGENERATOR](./serverless-benchmarker/docs/LOADGENERATOR.md).</li> <li> Choose the `PROVIDER` (aws or azure) in the [constant.py](./experiment-plans/constant.py) experiment plan</li> <li> Run the [constant.py](./experiment-plans/constant.py) experiment plan <ol> <li> Open tmux</li> <li> Activate virtualenv `source sb-env/bin/activate`</li> <li> Run `./constant.py 2>&1 | tee -a constant.log`</li> </ol> </li> </ol> <p><strong>Contributors</strong></p> <p>The initial trigger implementations for AWS and Azure are based on two master thesis projects at Chalmers University of Technology in Sweden supervised by Joel:</p> <ul> <li>AWS + Azure: <a href="https://odr.chalmers.se/handle/20.500.12380/302822">Performance Comparison of Function-as- a-Service Triggers: A Cross-Platform Performance Study of Function Triggers in Function-as-a-Service</a> by Marcus Bertilsson and Oskar Grönqvist, 2021.</li> <li>Azure Extension: <a href="https://odr.chalmers.se/handle/20.500.12380/305138">Serverless Function Triggers in Azure: An Analysis of Latency and Reliability</a> by Henrik Lagergren and Henrik Tao, 2022.</li> </ul> <p>Joel contributed many improvements to their original source code as documented in the import commits <a href="https://github.com/joe4dev/trigger-bench/commit/a00b67a1dd8476ca77d026e59adf2674c7807e68">a00b67a</a> and <a href="https://github.com/joe4dev/trigger-bench/commit/6d2f5ef8bda0596b3f295cb6c6cbeba212c6ef43">6d2f5ef</a> and developed TriggerBench as an integrated benchmark suite (see commit history for detailed changelog).</p>
Application-Oriented Performance Benchmarks for Quantum Computing
<p>Complete dataset and Jupyter Notebook used to produce image files for the paper at</p> <p> https://arxiv.org/abs/2110.03137.</p> <p>To execute the notebook, copy the .ipynb file and the _data directory to the top level of the repository at:</p> <p> https://github.com/SRI-International/QC-App-Oriented-Benchmarks</p> <p> </p>
LRGB: Long Range Graph Benchmark
<p>Graph Neural Networks (GNNs) that are based on the message passing (MP) paradigm exchange information between 1-hop neighbors to build node representations at each layer. In principle, such networks are not able to capture long-range interactions (LRI) that may be desired or necessary for learning a given task on graphs. Recently, there has been an increasing interest in development of Transformer-based methods for graphs that can consider full node connectivity beyond the original sparse structure, thus enabling the modeling of LRI. However, MP-GNNs that simply rely on 1-hop message passing often fare better in several existing graph benchmarks when combined with positional feature representations, among other innovations, hence limiting the perceived utility and ranking of Transformer-like architectures. Here, we present the Long Range Graph Benchmark (LRGB) with 5 graph learning datasets: PascalVOC-SP, COCO-SP, PCQM-Contact, Peptides-func and Peptides-struct that arguably require LRI reasoning to achieve strong performance in a given task. We benchmark both baseline GNNs and Graph Transformer networks to verify that the models which capture long-range dependencies perform significantly better on these tasks. Therefore, these datasets are suitable for benchmarking and exploration of MP-GNNs and Graph Transformer architectures that are intended to capture LRI.</p> <p><a href="https://arxiv.org/abs/2206.08164">arXiv</a>, <a href="https://paperswithcode.com/paper/long-range-graph-benchmark">Papers with Code</a></p>
WaterBench-Iowa: A Large-scale Benchmark Dataset for Data-Driven Streamflow Forecasting
<p>WaterBench-Iowa is a comprehensive benchmark dataset for streamflow forecasting. It follows FAIR data principles that are prepared with a focus on convenience for utilizing in data-driven and machine learning studies and provides benchmark performance for state-of-art deep learning architectures on the dataset for comparative analysis. By aggregating the datasets of streamflow, precipitation, watershed area, slope, soil types, and evapotranspiration from federal agencies and state organizations (i.e., NASA, NOAA, USGS, and Iowa Flood Center), we provided the WaterBench for hourly streamflow forecast studies. This dataset has a high temporal and spatial resolution with rich metadata and relational information, which can be used for varieties of deep learning and machine learning research. To some extent, WaterBench makes up for the lack of a unified benchmark in earth science research. We highly encourage researchers to use the WaterBench for deep learning research in hydrology.</p>
Re-Training Extension of the Benchmark for Automatic Glottis Segmentation (BAGLS-RT)
<p>BAGLS-RT is an extension of the BAGLS dataset (DOI 10.5281/zenodo.3762320) intended for (re-)training glottis segmentation models.</p>
Supplemental materials for "Benchmarking magnetized three-wave coupling for laser backscattering: Analytic solutions and kinetic simulations"
<p>Place the unzipped Data and Programs folders in the same directory. The contents of these folders are as follows:</p> <ul> <li>Data<br> Post processed data underlying each figure in the paper. The data files are .txt files with self-contained explanations. The files are organized in subfolders according to their purposes.<br> </li> <li>Programs <ul> <li>./PlotFigures<br> Contains python scripts for reading and plotting Data</li> <li>input.deck<br> Example input for EPOCH PIC code that generates raw data </li> <li>setup_batch.csh<br> Linux/Unix shell script for setting up batch simulations</li> <li>submit_batch<br> Slurm script for submitting jobs on computing clusters</li> </ul> </li> </ul>
SemTab 24: Semantic Table Annotations Benchmark for LLM-based approaches
<p><strong>SuperSemtab24 </strong>is a dataset for tabular data to knowledge graph matching.</p> <p>The dataset is divided into training and validation sets. The dataset includes general-purpose tables and intentionally misspelled entities to evaluate the model's robustness. Participants must annotate the entity mentions in the validation set and submit their annotations (following a target file).</p> <p>The repository contains the full version of the dataset; the ground truth (GT) of the test set will be uploaded in the future.</p>
Historical time-series reconstruction benchmark dataset of Landsat bi-monthly aggregates from GLAD ARD-2 at 30-m resolution with stratified sampling based on ESA CCI
<h2>Description</h2> <p>Historical time-series reconstruction benchmark dataset presented here is designed for evaluating and comparing the performance of time series reconstruction methods in the context of land cover change detection. The dataset is based on the European Space Agency Climate Change Initiative (ESA CCI) land cover dataset, which has been aggregated into 18 classes to facilitate analysis. The dataset includes information on land cover dynamics from 2000 to 2020, focusing on identifying and characterizing changes in land cover over time.</p> <h3><strong>Data Collection and Processing:</strong></h3> <p>The dataset is derived from the ESA CCI land cover dataset, which provides information on land cover classes at a global scale. The original dataset, containing 37 land cover classes, was aggregated into 18 classes based on similarity. Pixels with stable land cover over the study period and pixels with one or multiple land cover changes were identified and grouped into strata for sampling purposes.</p> <p>Sampling points were selected using a stratified sampling design, ensuring representation across different land cover classes and change scenarios. Approximately 2600 points were selected from each stratum, resulting in a total of 51,978 sampling points. The selected points were uniformly distributed along the strata, with spatial variations accounted for.</p> <p>Bimonthly time series data were extracted for each sampling point from 1997 to 2022, capturing temporal dynamics in land cover. Artificial gaps were introduced into the time series data to simulate real-world data loss, allowing for the evaluation of time series reconstruction methods under varying gap densities.</p> <p>The time series values were extracted from Landsat GLAD imagery using the specified spectral bands, including blue, green, red, NIR, SWIR1, SWIR2, and thermal bands. Additionally, a clear quality band was also extracted.</p> <h3>Data Details</h3> <ul> <li><strong>Time Period:</strong> 1997-01-01 to 2022-12-31</li> <li><strong>Type of Data: </strong>R data frame / Geopackage points.</li> <li><strong>Collection/Derivation:</strong> Derived from Landsat ARD v2, processed with Scikit-map.</li> <li><strong>Coordinate Reference System:</strong> EPSG:4326</li> <li><strong>Bounding Box:</strong> All the globe</li> <li><strong>File Format:</strong> RDS</li> </ul> <p> </p> <h3><strong>Reclassified Classes of ESA CCI Land Cover Dataset</strong></h3> <table> <tbody> <tr> <td> <div> <div> <p><strong>Aggregated Class Code</strong></p> </div> </div> </td> <td> <div> <div> <p><strong>Aggregated Class Label</strong></p> </div> </div> </td> <td> <div> <div> <p><strong>Original ESA CCI Classes</strong></p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>10</p> </div> </div> </td> <td> <div> <div> <p>Cropland rainfed</p> </div> </div> </td> <td> <div> <div> <p>10, 11, 12</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>30</p> </div> </div> </td> <td> <div> <div> <p>Mosaic cropland | natural vegetation</p> </div> </div> </td> <td> <div> <div> <p>30, 40</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>50</p> </div> </div> </td> <td> <div> <div> <p>Tree cover broadleaved evergreen</p> </div> </div> </td> <td> <div> <div> <p>50</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>60</p> </div> </div> </td> <td> <div> <div> <p>Tree cover broadleaved deciduous</p> </div> </div> </td> <td> <div> <div> <p>60, 61, 62</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>70</p> </div> </div> </td> <td> <div> <div> <p>Tree cover needleleaved evergreen</p> </div> </div> </td> <td> <div> <div> <p>70, 71, 72</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>80</p> </div> </div> </td> <td> <div> <div> <p>Tree cover needleleaved deciduous</p> </div> </div> </td> <td> <div> <div> <p>80, 81, 82</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>90</p> </div> </div> </td> <td> <div> <div> <p>Tree cover mixed leaf type</p> </div> </div> </td> <td> <div> <div> <p>90</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>100</p> </div> </div> </td> <td> <div> <div> <p>Mosaic tree and shrub | herbaceous cover</p> </div> </div> </td> <td> <div> <div> <p>100, 110</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>120</p> </div> </div> </td> <td> <div> <div> <p>Shrubland</p> </div> </div> </td> <td> <div> <div> <p>120, 121, 122</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>150</p> </div> </div> </td> <td> <div> <div> <p>Sparse vegetation</p> </div> </div> </td> <td> <div> <div> <p>150, 151, 152, 153</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>160</p> </div> </div> </td> <td> <div> <div> <p>Tree cover flooded</p> </div> </div> </td> <td> <div> <div> <p>160, 170</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>180</p> </div> </div> </td> <td> <div> <div> <p>Shrub or herbaceous cover flooded</p> </div> </div> </td> <td> <div> <div> <p>180</p> </div> </div> </td> </tr> <tr> <td> <div> <div> <p>200</p> </div> </div> </td> <td> <div> <div> <p>Bare areas</p> </div> </div> </td> <td> <div> <div> <p>200, 201, 202</p> </div> </div> </td> </tr> </tbody> </table> <p>In the table, each row represents a reclassified land cover class, identified by a unique code. The 'Original ESA CCI Classes' column lists the specific land cover classes from the European Space Agency Climate Change Initiative dataset that are grouped together to form each broader category. Note that land cover classes not listed in this table were retained in their original value and were not reclassified.</p> <h3><strong>File Format</strong></h3> <p>The dataset comprises observations spanning from January 1997 to November 2022, capturing data for 51,978 samples.</p> <ul> <li>blue.rds: Time series data for the blue spectral band.</li> <li>green.rds: Time series data for the green spectral band.</li> <li>red.rds: Time series data for the red spectral band.</li> <li>nir.rds: Time series data for the near-infrared (NIR) spectral band.</li> <li>swir1.rds: Time series data for the shortwave infrared 1 (SWIR1) spectral band.</li> <li>swir2.rds: Time series data for the shortwave infrared 2 (SWIR2) spectral band.</li> <li>thermal.rds: Time series data for the thermal infrared band.</li> <li>clear.rds: Time series data for the clear quality band, used for masking out cloudy observations.</li> </ul> <p>How open the files in R:</p> <p><code>blue <- readRDS("blue.rds")</code></p> <p>To open the files in Python, you need to the <code>pyreadr</code> library:</p> <p><code>import pyreadr</code><br><code>blue = pyreadr.read_r('blue.rds')</code></p> <p> </p>
segmentation results and ground truth for ComSeg benchmark
<p>This folder displays segmentation results and ground truth cell masks used in the benchmark of the paper: A point cloud segmentation framework for image-based spatial transcriptomics, Defard et al. </p> <p><br>The mouse iluem dataset raw images can be downloaded from the study : Petukhov, V., Xu, R.J., Soldatov, R.A. et al. Cell segmentation in imaging-based spatial transcriptomics. Nat Biotechnol 40, 345–354 (2022). https://doi.org/10.1038/s41587-021-01044-w</p> <p> </p> <p>the Vizgen MERFISH breast cancer dataset was sampled from MERSCOPE FFPE Human Immuno-oncology, https://vizgen.com/data-release-program/ : Breast cancer</p> <p> </p> <p> </p>
Lexical Semantic Change Cause-Type-Definitions Benchmark
<p>The Lexical Semantic Change Cause-Type-Definitions (LSC-CTD) Benchmark is a digitised dataset that builds on and extends the Blank's seminal 1997 taxonomy of semantic change. This collection categorises 657 instances of linguistic evolution across the vocabulary of the Romance languages, with additional entries of German and English instances. Each entry is accompanied by a new pair (Old and New Meaning) of english definitions, manually curated by a historical linguist.</p> <p>The dataset includes a detailed classification of causes of change such as semantic wear, lexical gap, orphaned word, lexical complexity, atypical actant structure, frame, socio-cultural change, abstract concept, atypical part of speech, new concept, taboo, expressivity and prototype. It also includes types of semantic shift as classified by Blank, i.e. specialisation, generalisation, co-hyponymous transfer, auto-antonym, metaphor, antiphrasis, metonymy, auto-converse, ellipsis, folk etymology, analogy, meaning dilution, meaning reinforcement and doubtful cases. </p> <p><br><strong>Reference</strong></p> <p>The accompanying paper where this resource is described in detail will be published at ACL 2024.<br><br><span>Pierluigi Cassotti, Stefano De Pascale, and Nina Tahmasebi. 2024. <a href="https://aclanthology.org/2024.acl-long.249">Using Synchronic Definitions and Semantic Relations to Classify Semantic Change Types</a>. In <em>Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>, pages 4539–4553, Bangkok, Thailand. Association for Computational Linguistics.</span></p>
Dataset for 'The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track'
<p>This packages comprises of analyses and evaluations of 60 datasets from the NeurIPS Datasets and Benchmarks track. It is part of a paper currently under review at the 2024 the NeurIPS Datasets and Benchmarks track, titled, "The State of Data Curation at NeurIPS: An Assessment of Dataset Development Practices in the Datasets and Benchmarks Track". </p>
A Benchmark Suite for Systematically Evaluating Reasoning Shortcuts
<p><strong>Codebase</strong> [<a href="https://unitn-sml.github.io/rsbench/">Github</a>] | <strong>Dataset</strong> [<a href="doi.org/10.5281/zenodo.11612556">Zenodo</a>]</p> <p> </p> <p><strong>Abstract</strong></p> <p>The advent of powerful neural classifiers has increased interest in problems that require both learning and reasoning. These problems are critical for understanding important properties of models, such as trustworthiness, generalization, interpretability, and compliance to safety and structural constraints. However, recent research observed that tasks requiring both learning and reasoning on background knowledge often suffer from <em>reasoning shortcuts</em> (RSs): predictors can solve the downstream reasoning task without associating the correct concepts to the high-dimensional data. To address this issue, we introduce <strong>rsbench</strong>, a comprehensive benchmark suite designed to systematically evaluate the impact of RSs on models by providing easy access to highly customizable tasks affected by RSs. Furthermore, rsbench implements common metrics for evaluating concept quality and introduces novel formal verification procedures for assessing the presence of RSs in learning tasks. Using rsbench, we highlight that obtaining high quality concepts in both purely neural and neuro-symbolic models is a far-from-solved problem. rsbench is available on <a href="https://unitn-sml.github.io/rsbench">Github</a>.</p> <p> </p> <p><strong>Usage</strong></p> <p>We recommend visiting the official code <a href="https://unitn-sml.github.io/rsbench/">website</a> for instructions on how to use the dataset and accompaying software code.</p> <p> </p> <p><strong>License</strong></p> <p>All ready-made data sets and generated datasets are distributed under the <a href="https://creativecommons.org/licenses/by-sa/4.0/">CC-BY-SA 4.0</a> license, with the exception of <code>Kand-Logic</code>, which is derived from <code>Kandinsky-patterns</code> and as such is distributed under the <a href="https://www.gnu.org/licenses/gpl-3.0.en.html">GPL-3.0</a> license.</p> <p> </p> <p><strong>Datasets Overview</strong></p> <ul> <li><strong>CLIP-embeddings. </strong>This folder contains the saved activations from a pretrained CLIP model applied to the tested dataset. It includes embeddings that represent the dataset in a format suitable for further analysis and experimentation.</li> <li><strong>BDD_OIA-original-dataset</strong>. This directory holds the original files from the X-OIA project by Xu et al. [1]. These datasets have been made publicly available for ease of access and further research. If you are going to use it, please consider citing the original authors.</li> <li><strong>kand-logic-3k</strong>. This folder contains all images generated for the Kand-Logic project. Each image is accompanied by annotations for both concepts and labels.</li> <li><strong>bbox-kand-logic-3k</strong>. In this directory, you will find images from the Kand-Logic project that have undergone a preprocessing step. These images are extracted based on bounding boxes, rescaled, and include annotations for concepts and labels.</li> <li><strong>sdd-oia</strong>. This folder includes all images and labels generated using rsbench.</li> <li><strong>sdd-oia-embeddings</strong>. This directory contains 512-dimensional embeddings extracted from a pretrained ResNet18 model on ImageNet. The embeddings are derived from the sdd-oia`dataset.</li> <li><strong>BDD-OIA-preprocessed</strong>. Here you will find preprocessed data that follow the methodology outlined by Sawada and Nakamura [2]. The folder contains 2048-dimensional embeddings extracted from a pretrained Faster-RCNN model on the BDD-100k dataset.</li> </ul> <p>The original BDD datasets can be downloaded from the following Google Drive link: [<a href="https://drive.google.com/file/d/1WFiwRi_sMA_McZnkbEjh8Rnl-Im7_9Mk/view">Download BDD Dataset</a>].</p> <p> </p> <p><strong>References</strong></p> <p>[1] Xu et al., *Explainable Object-Induced Action Decision for Autonomous Vehicles*, CVPR 2020.</p> <p>[2] Sawada and Nakamura, *Concept Bottleneck Model With Additional Unsupervised Concepts*, IEEE 2022.</p> <p> </p>
CausalBench A Comprehensive Benchmark for Evaluating Causal Reasoning Capabilities of Large Language Models
<p>CausalBench is a comprehensive benchmark dataset designed to evaluate the causal reasoning capabilities of large language models. The primary uses of this dataset include, but are not limited to:</p> <p>- Testing the performance of large language models on causal reasoning tasks</p> <p>- Serving as a benchmark dataset for causal reasoning research</p> <p>- Improving and developing new causal reasoning algorithms and models</p>
Benchmark datasets for RDF load time evaluation (RiverBench)
<div> <p>Datasets to be used for reproducing the RDF load time benchmark, using the code here: <a href="https://github.com/Ostrzyciel/rdf4led-riverbench">https://github.com/Ostrzyciel/rdf4led-riverbench</a></p> <p>The datasets were obtained from <a href="https://w3id.org/riverbench/v/2.0.1/profiles/flat-triples" rel="nofollow">RiverBench profile <code>flat-triples</code> version 2.0.1</a>. <strong>The detailed licensing and authorship information for each individual dataset is available on <a href="https://w3id.org/riverbench/v/2.0.1/datasets" rel="nofollow">RiverBench's website</a>.</strong> The most restrictive license that applies to any of the datasets is CC BY-SA.</p> <p>Benchmark results: <a href="https://doi.org/10.5281/zenodo.12087112">https://doi.org/10.5281/zenodo.12087112</a></p> </div>
BSBM v3.1 SPARQL benchmark pre-generated files
<p>Data files generated from the <a href="http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/20110607/">Berlin SPARQL Benchmark v3.1</a>.</p> <p>The following files are available:</p> <ul> <li>dataset-X.nt.bz2 are the RDF files containing the benchmark dataset. They are built with the generator provided with the benchmark and all parameters left to their default values except the number of products (-pc option) set to X.</li> <li>explore-X.csv.bz2, exploreAndUpdate-X.csv.bz2 and businessIntelligence-X.csv.bz2 are the various query use cases to be run on the corresponding dataset-X.nt.bz2. They are generated with all parameters left to their default values. The first 1250 queries for the explore and exploreAndUpdate use cases and the first 750 queries of the businessIntelligence use case are warm-up queries. The queries are encoded in CSV files, with an "id" column containing the query identifier (1, 2, 3...), a "kind" column being "query" or "update" (for SPARQL updates in the update use case), and a "content" column with the query itself.</li> </ul> <p>All files are compressed using bzip2.</p> <p>The script used to generate the file is <a href="https://github.com/Tpt/bsbm-tools/blob/da7c78d8c6f4eeb19149b9848c24123a97f31e2f/generate_bsbm.sh">available on GitHub</a>.</p>
Database of Benchmark Molecules with Functional Groups
<p>A dataset was created by calculating 156 molecules, each with around 800 conformers. To store the dataset, a database was designed that supports CRUD (Create, Read, Update, Delete) operations, allowing efficient management and retrieval of the data. Each molecule is associated with a unique identifier (SMILES), and each conformer has an ID as well. The molecules also have their corresponding functional groups, which they can be identified by, as well. A PDF file representing the functional groups is included, ensuring comprehensive documentation and easy access to the molecular data.<br>The python script, can be accessed via the repository link.</p>
LeanDojo Benchmark 4
<p>Lean 4 version of the dataset in the paper:</p> <p><a href="https://leandojo.org/">LeanDojo: Theorem Proving with Retrieval-Augmented Language Models</a><br> <a href="https://yangky11.github.io/">Kaiyu Yang</a>, <a href="https://aidanswope.com/about">Aidan Swope</a>, <a href="https://minimario.github.io/">Alex Gu</a>, <a href="https://www.linkedin.com/in/rchalamala">Rahul Chalamala</a>, <a href="https://www.linkedin.com/in/peiyang-song-3279b3251/">Peiyang Song</a>, <a href="https://billysx.github.io/">Shixing Yu</a>, <a href="https://www.linkedin.com/in/saad-godil-9728353/">Saad Godil</a>, <a href="https://www.linkedin.com/in/ryan-prenger-18797ba1/">Ryan Prenger</a>, <a href="http://tensorlab.cms.caltech.edu/users/anima/">Anima Anandkumar</a></p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.