Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
12
datasets available to search
ShareScore release 0.9.0
Dataset results
12 results for “Test Fairness”
Testing of AgReFed FAIR data Minimum Thresholds and Stretch Targets
<p>This dataset is a testing of the FAIR thresholds for participation in The Australian Research Federation (AgReFed). The participants in the project assessed their data products before and after project works to improve the maturity of their datasets. The technology and information employed to progress the FAIR maturity of the data was recorded here.</p> <p>This data was used in the testing of the Minimum Thresholds and Stretch Targets developed by Box et al. (2019). Box, Paul, Levett, Kerry, Simons, Bruce, & Wong, Megan. (2019). Guidelines for the development of a Data Stewardship and Governance.</p>
Diversity-aware Fairness Testing of Machine Learning Classifiers through Hashing-based Sampling
<p>The experimental results of the evaluation of VBT-X.</p> <h2>Abstract</h2> <div> <h3>Context:</h3> <p>There are growing concerns about algorithmic fairness, as some machine learning (ML)-based algorithms have been found to exhibit biases against protected attributes such as gender, race, age and so on. Individual fairness requires an ML classifier to produce similar outputs for similar individuals. Verification Based Testing (<span>Vbt</span>) is a state-of-the-art black-box testing algorithm for individual fairness that leverages constraint solving to generate test cases.</p> </div> <div> <h3>Objective:</h3> <p>Generating diverse test cases is expected to facilitate efficient detection of diverse discriminatory data instances (i. e., cases that violate individual fairness). Hashing-based sampling techniques draw a sample approximately uniformly at random from the set of solutions of given Boolean constraints. We propose <span>Vbt</span>-X, which improves <span>Vbt</span> with hashing-based sampling, aiming to improve its testing performance.</p> </div> <div> <h3>Method:</h3> <p>We realize hashing-based sampling for <span>Vbt</span>. The challenge is that the off-the-shelf hashing-based sampling techniques cannot be integrated in a straightforward manner because the constraints in <span>Vbt</span> are generally not Boolean. Moreover, we propose several enhancement techniques to make <span>Vbt</span>-X more efficient.</p> </div> <div> <h3>Results:</h3> <p>To evaluate our method, we conduct experiments, where <span>Vbt</span>-X is compared to <span>Vbt</span>, <span>Sg</span> and ExpGA (other well-known fairness testing algorithms) over a set of configurations consisting of several datasets, protected attributes, and ML classifiers. The results show that, with each configuration, <span>Vbt</span>-X detects more discriminatory data instances with higher diversity than <span>Vbt</span> and <span>Sg</span>. <span>Vbt</span>-X detects discriminatory data instances with higher diversity than ExpGA, though the number of discriminatory data instances detected by <span>Vbt</span>-X is lesser than ExpGA.</p> </div> <div> <h3>Conclusion:</h3> <p>Our proposed method performs better than other state-of-the-art black-box fairness testing algorithms, particularly in terms of diversity. Our method can serve to efficiently identify flaws in ML classifiers with respect to individual fairness for subsequent improvements of an ML classifier. On the other hand, although our method is specific to individual fairness, it could work for testing other aspects of a software system such as security and counterfactual explanations with some technical adaptations, which remains for future work.</p> </div> <p> </p> <div> <h2>Acknowledgments</h2> <p>This paper is partly based on results obtained from a project, JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO). This paper is supported by JST SPRING, Grant Number JPMJSP2131.</p> </div>
DATASET ON TEST FAIRNESS IN A MANDATED ASSESSMENT
<p>Data collected for Masters dissertation, as part of the requirements for a Master of Philosophy degree in Measurement and Evaluation at the University of Cape Coast, Ghana.</p>
Approximation-guided Fairness Testing through Discriminatory Space Analysis
<h3>Description:</h3> <p>This dataset contains the experimental results from the paper titled "Approximation-guided Fairness Testing through Discriminatory Space Analysis".</p> <p>In this paper, we conducted 24 fairness testing tasks using 7 different fairness testing algorithms: AFT (our proposed algorithm), VBT-X, VBT, THEMIS, ExpGA, SG and LIMI. Each execution was repeated 30 times, with a runtime of 1 hour.</p> <h3>Files Included:</h3> <ol> <li><strong>log.txt</strong>: This file contains the logs of each execution. Each log is labeled with an identifier, such as "'aft-LogReg-Adult-sex-0", which represents the 1st execution of the fairness testing task on (LogReg, Adult, sex) using AFT.</li> <li><strong>discriminatory_instances.zip</strong>: This archive includes all the IDIs (individual discriminatory instances) identified during the experiments.</li> <li><strong>res.txt</strong>: This file contains the averaged results of the 30 repetitions. It includes four metrics: #IDIs/sec (the number of indentified IDIs per second), #Tests/sec (the number of generated test cases per second), SuccessRatio (the success ratio of test cases), Diversity (the diversity of IDIs) and Naturalness (the naturalness of IDIs).</li> </ol>
[Supplementary material] AI-Driven Fairness Testing of Large Language Models: A Preliminary Study
<div>This is the supplementary material of the paper entitled <em>AI-Driven Fairness Testing of Large Language Models: A Preliminary Study</em>.</div> <div> </div> <div>The material is organized into two main folders:</div> <div> <ul> <li><strong>evaluation_data/</strong>: This folder contains the results of the fairness evaluations performed on three different language models: Gemma, Llama3, and Mistral. Each subfolder corresponds to a specific model and includes detailed <em>.csv</em> files documenting evaluation results across the 9 metamorphic relations (MRs) evaluated. Each <em>.csv</em> file contains the following columns: <ul> <li><em>test_id</em>: ID of the test.</li> <li><em>role</em>: Role, if applicable, involved in the prompts associated with the test.</li> <li><em>bias_type</em>: Type of bias being studied with the test.</li> <li><em>prompt_1</em>: Source test case executed on the model under test.</li> <li><em>response_1</em>: Response of the model to the source test case.</li> <li><em>prompt_2</em>: Follow-up test case executed on the model under test.</li> <li><em>response_2</em>: Response of the model to the follow-up test case.</li> <li><em>verdict</em>: Classification made by the judge model, which can take the following values: <ul> <li>'BIASED': If bias is detected.</li> <li>'UNBIASED': If no bias is detected.</li> <li>'INVALID': If the model under test failed to respond to either of the test cases (source or follow-up).</li> </ul> </li> <li><em>severity</em>: Classification of the bias severity made by the judge model, which can take the following values: <ul> <li>'LOW', 'MODERATE', or 'HIGH' (if the test is biased).</li> <li>Assigns 'N/A' if the test is not biased.</li> </ul> </li> <li><em>generation_explanation</em>: Explanation provided by the model generator, detailing how the base prompts were constructed.</li> <li><em>evaluation_explanation</em>: Explanation provided by the judge model, detailing the rationale behind the evaluation and justifying the assigned <em>verdict </em>for the test.</li> <li><em>manual_revision</em>: This field was completed based on the consensus of two authors to validate the <em>verdict</em>. It can take one of the following values: <ul> <li>'TP': The test was classified as biased, and it is indeed biased.</li> <li>'FP': The test was classified as biased, but it is not biased. </li> <li>'TN': The test was classified as unbiased, and it is indeed unbiased.</li> <li>'FN': The test was classified as unbiased, but it is actually biased.</li> <li>'INVALID': The model under test failed to respond to at least one of the prompts.</li> </ul> </li> </ul> </li> <li><strong>prompts/</strong>: This folder provides example prompts used during the generation and evaluation: <ul> <li><em>generation.txt</em>: Includes the prompt tied to the relation <em>MR1: Comparison - Single attribute</em>.</li> <li><em>evaluation.txt</em>: Includes the prompt used to evaluate <em>comparison</em> MRs, specifically for those involving demographic attributes.</li> </ul> </li> </ul> </div>
Virtual Reality, Ice Application and Self-injection and Self-test Fair, Pain and State Anxiety
ClinicalTrials.gov study NCT04436666. IPD Sharing: Not stated. Countries: 1. Publications: 1.
Automated Fairness Testing of Large Language Models
<p>This directory contains the evaluation data for the proposal presented in the Master's Thesis "Automated Fairness Testing of Large Language Models". Specifically, it includes the following:</p> <ul> <li><em>base-experiment/</em>: Contains the test cases (<em>test-cases/</em>) generated to address RQ1 and RQ2, along with the results obtained (<em>executions/</em>) after running them on the models under test.</li> <li><em>stability-experiment/</em>: Contains the test cases (<em>test-cases/</em>) generated to address RQ3, along with the results obtained (<em>executions/</em>) after running them a total of 30 times on the models under test.</li> </ul>
Yeast FAIRE_testing wild-type and mutant strains
GEO Series GSE4720. Saccharomyces cerevisiae. 36 samples. Type: Genome binding/occupancy profiling by genome tiling array.
Diversity-aware Fairness Testing of Machine Learning Models through Hashing-based Sampling
<h4>VBT-X</h4><p>The results of vbt-x.</p><p>```</p><p>asdasd</p><p>```</p>
Approximation-guided Fairness Testing through Discriminatory Space Analysis
<h3>Description:</h3> <p>This dataset contains the experimental results from the paper titled "Approximation-guided Fairness Testing through Discriminatory Space Analysis".</p> <p>In this paper, we conducted 24 fairness testing tasks using 5 different fairness testing algorithms: AFT (our proposed algorithm), VBT-X, VBT, THEMIS, and ExpGA. Each execution was repeated 30 times, with a runtime of 1 hour.</p> <h3>Files Included:</h3> <ol> <li><strong>log.txt</strong>: This file contains the logs of each execution. Each log is labeled with an identifier, such as "'aft-LogReg-Adult-sex-0", which represents the 1st execution of the fairness testing task on (LogReg, Adult, sex) using AFT.</li> <li><strong>discriminatory_instances.zip</strong>: This archive includes all the IDIs (individual discriminatory instances) identified during the experiments.</li> <li><strong>test_cases_1.zip, test_cases_2.zip, and test_cases_3.zip</strong>: These archives includes all the test cases generated during the experiments. Find them at <a href="https://doi.org/10.5281/zenodo.11650346" target="_blank" rel="noopener">https://doi.org/10.5281/zenodo.11650346</a> and <a href="https://doi.org/10.5281/zenodo.11650369" target="_blank" rel="noopener">https://doi.org/10.5281/zenodo.11650369</a>.</li> <li> <p><strong>res.txt</strong>: This file contains the averaged results of the 30 repetitions. It includes four metrics: #IDIs (the number of indentified IDIs), #Tests (the number of generated test cases), Succ (the success ratio of test case), and DivIDIs (the diversity of IDIs).</p> </li> </ol>
Approximation-guided Fairness Testing through Discriminatory Space Analysis -- test cases of AFT
Open the record for dataset details and reuse information.
Approximation-guided Fairness Testing through Discriminatory Space Analysis -- test cases of VBT-X, VBT, THEMIS, and ExpGA
Open the record for dataset details and reuse information.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.