[Supplementary material] AI-Driven Fairness Testing of Large Language Models: A Preliminary Study

<div>This is the supplementary material of the paper entitled <em>AI-Driven Fairness Testing of Large Language Models: A Preliminary Study</em>.</div> <div>&nbsp;</div> <div>The material is organized into two main folders:</div> <div> <ul> <li><strong>evaluation_data/</strong>: This folder contains the results of the fairness evaluations performed on three different language models: Gemma, Llama3, and Mistral. Each subfolder corresponds to a specific model and includes detailed&nbsp;<em>.csv</em> files documenting evaluation results across the 9 metamorphic relations (MRs) evaluated. Each&nbsp;<em>.csv</em>&nbsp;file contains the following columns: <ul> <li><em>test_id</em>: ID of the test.</li> <li><em>role</em>: Role, if applicable, involved in the prompts associated with the test.</li> <li><em>bias_type</em>: Type of bias being studied with the test.</li> <li><em>prompt_1</em>: Source test case executed on the model under test.</li> <li><em>response_1</em>: Response of the model to the source test case.</li> <li><em>prompt_2</em>: Follow-up test case executed on the model under test.</li> <li><em>response_2</em>: Response of the model to the follow-up test case.</li> <li><em>verdict</em>: Classification made by the judge model, which can take the following values: <ul> <li>'BIASED': If bias is detected.</li> <li>'UNBIASED': If no bias is detected.</li> <li>'INVALID': If the model under test failed to respond to either of the test cases (source or follow-up).</li> </ul> </li> <li><em>severity</em>: Classification of the bias severity made by the judge model, which can take the following values: <ul> <li>'LOW', 'MODERATE', or 'HIGH' (if the test is biased).</li> <li>Assigns 'N/A' if the test is not biased.</li> </ul> </li> <li><em>generation_explanation</em>: Explanation provided by the model generator, detailing how the base prompts were constructed.</li> <li><em>evaluation_explanation</em>: Explanation provided by the judge model, detailing the rationale behind the evaluation and justifying the assigned <em>verdict </em>for the test.</li> <li><em>manual_revision</em>: This field was completed based on the consensus of two authors to validate the <em>verdict</em>. It can take one of the following values: <ul> <li>'TP': The test was classified as biased, and it is indeed biased.</li> <li>'FP': The test was classified as biased, but it is not biased.&nbsp;</li> <li>'TN': The test was classified as unbiased, and it is indeed unbiased.</li> <li>'FN': The test was classified as unbiased, but it is actually biased.</li> <li>'INVALID': The model under test failed to respond to at least one of the prompts.</li> </ul> </li> </ul> </li> <li><strong>prompts/</strong>: This folder provides example prompts used during the generation and evaluation: <ul> <li><em>generation.txt</em>: Includes the prompt tied to the relation&nbsp;<em>MR1: Comparison - Single attribute</em>.</li> <li><em>evaluation.txt</em>: Includes the prompt used to evaluate&nbsp;<em>comparison</em> MRs, specifically for those involving demographic attributes.</li> </ul> </li> </ul> </div>