LocalizeAgent Reproducibility Package

<h1>Replication Package: An LLM-Based Agent-Oriented Approach for Automated Code Design Issue Localization</h1> <p>This replication package contains the necessary code and instructions to run the experiments described in the paper, "An LLM-Based Agent-Oriented Approach for Automated Code Design Issue Localization".</p> <h2>Prerequisites</h2> <ul> <li>Python 3.12.2</li> <li>Required Python packages: <ul> <li>pip install anthropic</li> <li>pip install openai</li> <li>pip install google-generativeai</li> <li>pip install tqdm</li> <li>pip install joblib</li> <li>pip install javalang</li> <li>pip install numpy</li> </ul> </li> </ul> <h2>Installation</h2> <p>1. Clone the repository or extract the replication package.<br>2. Install the required Python packages</p> <h2>Running the Experiments</h2> <h3>Generating LLM Responses</h3> <p>To generate responses from language models (LLMs) like Claude, OpenAI, or Gemini, use the&nbsp;<code>run.py</code>&nbsp;script with the appropriate arguments.</p> <p><code>&nbsp;python3 run.py --MODEL_NAME --output ./openai/ --path PATH_TO_EVALUATION_DATASET</code></p> <p>For example,</p> <p><code>&nbsp;python3 run.py --openai --output ./openai/move_m --path EvaluationDataset/move_m/single</code></p> <ul> <li><code>--openai</code>: Use the OpenAI API (alternatively, use&nbsp;<code>--claude</code>&nbsp;or&nbsp;<code>--gemini</code>&nbsp;for other LLMs).</li> <li><code>--output</code>: Specify the directory to save the LLM responses.</li> <li><code>--path</code>: Provide the path to the directory containing the Java files.</li> </ul> <p>You can also use additional arguments like&nbsp;<code>--limit</code>&nbsp;to limit the number of files to process,&nbsp;<code>--context</code>&nbsp;to add context to the prompt, and&nbsp;<code>--log_file</code>&nbsp;to specify a log file path.</p> <h3>Evaluating LLM Responses</h3> <p>To evaluate the LLM responses by comparing function names extracted from file IDs and response texts, use the&nbsp;<code>eval_llms.py</code>&nbsp;script.</p> <p><code>python3 run.py --eval --path /openai/move_m/llm_responses.json</code></p> <ul> <li><code>--eval</code>: Enable evaluation mode.</li> <li><code>--path</code>: Provide the path to the JSON file containing the LLM responses.</li> </ul> <h3>Running the Evaluation Script</h3> <p>The&nbsp;<code>run_eval.sh</code>&nbsp;script is provided to automate the evaluation process for multiple LLM responses. It loops through a predefined set of paths and executes the evaluation command for each path.</p> <p><code>bash ./run_eval.sh</code></p> <h2>Customization</h2> <p>You can modify the source code files (<code>eval_llms.py</code>,&nbsp;<code>llm_interfaces.py</code>,&nbsp;<code>prompt_generation.py</code>, and&nbsp;<code>utils.py</code>) to customize the behavior or add new features as needed.</p> <h2>Additional Notes</h2> <ul> <li>The "EvaluationDataset" directory is already included in the replication package.</li> <li>The output directory specified with the&nbsp;<code>--output</code> argument will be created if it doesn't already exist.</li> </ul>