zenodoopen

Replication package of the paper "Do LLMs Provide Links to Code Similar to what they Generate? A Study with Gemini and Bing CoPilot"

<h1>Replication Package</h1> This replication package contains the necessary tools, data, and scripts for reproducing the results of our paper: "Do LLMs Provide Links to Code Similar to what they Generate? A Study with Gemini and Bing CoPilot". Below is a detailed description of the directory structure and the contents of this package. <h2>Contents</h2> The replication package is organized into two main directories: <ul> <li> <code>assets</code>: This directory contains all .csv files used as input for the script and the outputted .csv file used to perform the manual and automated analyses for RQ1 and RQ2. </li> <li> <code>script</code>: This directory contains all scripts for RQ1 and RQ2. </li> </ul> In the following, we describe the content of each directory: <h2><code>assets</code></h2> This directory contains the tools and resources required for our study. <h3><code>dataset</code>: Contains the main datasets used in the study.</h3> <ul> <li> <code>annotationStore.csv</code>: Input dataset for our analyses, originating from the CODESEARCHNET dataset. </li> <li> <code>queries.csv</code>: .csv file containing the queries used for the experiments filtered from the CODESEARCHNETdataset. This file contains the following columns: <ul> <li>Language: Programming language of the query</li> <li>Query: Query used for the experiment</li> <li>GitHubUrl: GitHub URL related to a snippet that addresses the query</li> <li>Relevance: Relevance of the linked GitHub snippet to the query</li> </ul> </li> </ul> <h3><code>data</code>: Contains the datasets and results of all analyses.</h3> <ul> <li> <code>queries.csv</code>: General input queries. This file contains the following columns: <ul> <li>Language: Programming language of the query</li> <li>Query: Query used for the snippet generation</li> <li>Prompt: LLM prompt generated for the query as: You are a Senior <code><Language></code> developer. Then give me a <code><Language></code> code snippet about: <code><Query></code></li> </ul> </li> <li> <code>queries_filled.csv</code>: Similar to the previous file, but also containing the output produced by the LLM-based assistants. This file contains the following columns: <ul> <li>Language: Programming language of the query</li> <li>Query: Query used for the snippet generation</li> <li>Prompt: LLM prompt generated for the query as: You are a Senior <code><Language></code> developer. Then give me a <code><Language></code> code snippet about: <code><Query></code></li> <li>Notes: General notes that provide additional context or information about the query or prompt.</li> <li>Gemini_Answer(n): The generated code snippets by Gemini.</li> <li>Gemini(n): The external links provided by Gemini.</li> <li>Prompt (repeated)</li> <li>Note: Notes that provide additional context or information about the query or prompt.</li> <li>Copilot_Answer(n): The generated code snippets by Bing-Copilot.</li> <li>Copilot_Bing(n): The external links provided by Bing-Copilot.</li> </ul> </li> </ul> <h4><code>copilot</code> || <code>gemini</code>: Contains the data related to the specific LLM. These two subdirectories have the same internal structure.</h4> <ul> <li><code>queries.csv</code>: The <code>queries_filled.csv</code> file, filtered for the specific LLM.</li> <li><code>queries_noTrivial.csv</code>: Contains only the queries with at least one nontrivial generated snippet.</li> <li> <code>external_links.csv</code>: External links extracted from the LLMs output. </li> <li> <code>external_links_filled.csv</code>: Snippets extracted from the external links. <ul> <li>index: Query ID</li> <li>source: Snippet ID</li> <li>url: Link URL</li> <li>note: Notes that provide additional context or information about the query or prompt</li> <li>code(n): The n-th code snippet extracted from the source</li> </ul> </li> </ul> <h4><code>manual_analysis</code>: Manual analysis results.</h4> <ul> <li><code>manual_analysis.csv</code>: <ul> <li>index: Query ID</li> <li>query: Query used for the snippet generation</li> <li>generatedsnippet(n): The n-th code snippet generated by the LLM-based assistant</li> <li>trivial_1: Manual analysis of whether or not the snippet was trivial (validator 1)</li> <li>trivial_2: Manual analysis of whether or not the snippet was trivial (validator 2)</li> <li>trivial_final: Manual analysis of whether or not the snippet was trivial (final classification if there is a disagreement)</li> <li>source: URL to analyze</li> <li>sourcetype1: Type of the source (validator 1)</li> <li>sourcetype2: Type of the source (validator 2)</li> <li>sourcetypefinal: Type of the source (final classification if there is a disagreement)</li> <li>relatedtoquery_1: Relevance of the link to the query (validator 1)</li> <li>relatedtoquery_2: Relevance of the link to the query (validator 2)</li> <li>relatedtoquery_final: Relevance of the link to the query (final classification if there is a disagreement)</li> <li>relatedtosnippets_1: Relevance of the generated snippet to those in the link (validator 1)</li> <li>relatedtosnippets_2: Relevance of the generated snippet to those in the link (validator 2)</li> <li>relatedtosnippets_final: Relevance of the generated snippet to those in the link (final classification if there is a disagreement)</li> </ul> </li> <li><code>manual_analysis_noTrivial.csv</code>: As in the previous file, but only the queries with at least one nontrivial generated code snippet.</li> </ul> <h4><code>clone_detector</code>: Output and intermediate files for clone detection with Copilot data.</h4> <ul> <li><code>copilot_tokens || gemini_tokens</code>: Contains the output the tokenization of the generated code snippets and the code snippets extracted from the external links.</li> <li><code>merged_llm_ext_link.csv</code>: All possible pairs (Cartesian product) (code snippet extracted from the external links, generated code snippet). This file is the input of the clone detection tool. <ul> <li>ID_query: Query ID</li> <li>query: Query used for the snippet generation</li> <li>language: Programming language of the query</li> <li>generated_snippet: The generated code snippet by the LLM-based assistant</li> <li>IDgensnippet: The index of the generated code snippet</li> <li>LOCgensnippet: The number of lines of code of the generated code snippet</li> <li>ID_source: Source ID</li> <li>source: Source URL</li> <li>source_snippet: Code snippet extracted from the source</li> <li>IDsourcesnippet: ID of the code snippet extracted from the source</li> <li>LOCsourcesnippet: The number of lines of code of the code snippet extracted from the source</li> <li>note: Notes that provide additional context or information about the query or prompt</li> </ul> </li> <li><code>clone_detection_output.csv</code>: Contains the clone detection results. <ul> <li>ID_query: The index of the query</li> <li>query: Query used for the snippet generation</li> <li>language: The programming language of the query</li> <li>generated_snippet: The generated code snippet by the LLM-based assistant</li> <li>IDgensnippet: The index of the generated code snippet</li> <li>LOCgensnippet: The number of lines of code of the generated code snippet</li> <li>ID_source: Source ID</li> <li>source: Source URL</li> <li>source_snippet: Code snippet extracted from the source</li> <li>IDsourcesnippet: ID of the code snippet extracted from the source</li> <li>LOCsourcesnippet: The number of lines of code of the code snippet extracted from the source</li> <li>note: Notes that provide additional context or information about the query or prompt</li> <li>clone_detected: bBolean value that indicates whether a clone has been detected (1 = detected, 0 = not detected)</li> <li>cloning_ratio: Ratio of the number of lines of code of the generated code snippet has been detected as a clone in the code snippet extracted from the source</li> <li>cloned_lines: The number of lines of code of the generated code snippet that has been detected as a clone in the code snippet extracted from the source</li> </ul> </li> </ul> <h4><code>cosine_sim</code>: Cosine similarity results.</h4> <ul> <li><code>cosine_sim_output.csv</code>: Contains the cosine similarity results <ul> <li>query_id: Query ID</li> <li>snippet_id:ID the generated code snippet</li> <li>source_id: ID of the source</li> <li>sourcesnippetid: ID of the code snippet extracted from the source <ul> <li>cosine_similarity: The cosine similarity between the generated code snippet and the code snippet extracted from the source</li> </ul> </li> </ul> </li> </ul> <h4><code>quant_analysis</code>: Quantitative analysis results.</h4> <ul> <li><code>topN_links_se.csv</code>: Contains the top-N links extracted from the search engine. <ul> <li>id: Query ID</li> <li>query: The query</li> <li>url: Link URL</li> </ul> </li> <li><code>merged_clone_cosine.csv</code>: Contains the merged results of the clone detection and cosine similarity. <ul> <li>ID_query: Query ID</li> <li>query: The query</li> <li>language: The programming language of the query</li> <li>generated_snippet: The generated code snippet by the LLM-based assistant</li> <li>IDgensnippet: The ID of the generated code snippet</li> <li>LOCgensnippet: The number of lines of code of the generated code snippet</li> <li>ID_source: The index of the source</li> <li>source: The source URL</li> <li>source_snippet: The code snippet extracted from the source</li> <li>IDsourcesnippet: The index of the code snippet extracted from the source</li> <li>LOCsourcesnippet: The number of lines of code of the code snippet extracted from the source</li> <li>note: Notes that provide additional context or information about the query or prompt</li> <li>clone_detected: Boolean value that indicates if a clone has been detected(1 = detected, 0 = not detected)</li> <li>cloning_ratio: The ratio of the number of lines of code of the generated code snippet has been detected as a clone in the code snippet extracted from the source</li> <li>cloned_lines: The number of lines of code of the generated code snippet that has been detected as a clone in the code snippet extracted from the source</li> <li>cosine_similarity: The cosine similarity between the generated code snippet and the code snippet extracted from the source</li> </ul> </li> </ul> <h4><code>other_analysis</code>: Contains more performed analysis.</h4> <ul> <li><code>sample_queries.csv</code>: Contains a sample of five queries for language used for perform the chain of thought experiment. <ul> <li>Language: Programming language of the query</li> <li>Query: Query used for the snippet generation</li> <li>Prompt: LLM prompt generated for the query as: You are a Senior <code><Language></code> developer. Then give me a <code><Language></code> code snippet about: <code><Query></code></li> </ul> </li> <li><code>chain_of_thought.csv</code>: <ul> <li>Language: Programming language of the query</li> <li>Query: Query used for the snippet generation</li> <li>Prompt: LLM prompt generated for the query as: You are a Senior <code><Language></code> developer. Then give me a <code><Language></code> code snippet about: <code><Query></code></li> <li>Clone_fonud: Boolean value that indicates if a clone has been detected (Yes = detected, No = not detected)</li> <li>Note: Notes that provide additional context or information about the performed analysis</li> </ul> </li> <li><code>data_check.csv</code>: <ul> <li>Link: URL of the source provided by the LLM</li> <li>Post_date: Indicates if the date of the post is before/after the date of training of the LLM (before 2023, after 2023, not provided)</li> <li>Note: Notes that provide additional context or information about the performed analysis</li> </ul> </li> </ul> <h4><code>results</code>: Final analysis results.</h4> <ul> <li><code>jaccard_analysis.csv</code>: Contains the results of the Jaccard analysis comparing the provided external links by the LLMs with the top-N links extracted from the corresponding search engine. <ul> <li>id: Query ID</li> <li>language: The programming language of the query</li> <li>llm_link: The external links provided by the LLM</li> <li>llmlinksize: The number of external links provided by the LLM</li> <li>overlap_links: The overlapping links between the LLM and the search engine</li> <li>overlap_size: The number of overlapping links between the LLM and the search engine</li> <li>nonoverlaplinks: The non-overlapping links between the LLM and the search engine</li> <li>union_size: The size of the union set links between the LLM and the search engine</li> <li>jaccard: The Jaccard similarity between the LLM and the search engine</li> </ul> </li> <li><code>merged_analysis.csv</code>: Contains the merged results of the manual and quantitative analyses. <ul> <li>id: The index of the query</li> <li>query: The query used for the experiment</li> <li>trivial_final(n): The final assignment for the triviality of the n-th generated code snippet</li> <li>source: The URL of the source</li> <li>sourcetypefinal: The final assignment for the type of the source</li> <li>relatedtoquery_final: The final assignment for the relevance of the generated code snippet to the query</li> <li>relatedtosnippets_final: The final assignment for the relevance of the generated code snippet to the source</li> <li>cloning_ratio: The maximum cloning ratio between the generated code snippet and all the code snippets extracted from the source</li> <li>cosine_similarity: The cosine similarity related to the snippets with maximum cloning ratio</li> </ul> </li> </ul> <h3><code>cccfindersw-configuration-files</code>:</h3> Contains additional configuration files for the CCFinderSW clone detection tool. The files are <code>javascript_comment.txt</code> and <code>javascript_reserved.txt</code>. They must be placed in the tool's <code>comment/</code> and <code>reserved/</code> directories. <h3><code>appendix.tex</code>: The appendix of the paper containing:</h3> <ul> <li>Table 1: Number of links of different types provided by Gemini and Bing CoPilot</li> </ul> <h3><code>appendix.pdf</code>: The appendix of the paper in PDF format.</h3> <h2><code>script</code></h2> This directory contains our scripts (mostly Python, an R script and an Applescript) to preprocess data and run the clone detection analyses. <ul> <li><code>1_dateset_filtering.py</code>: Script to filter the dataset. The input of this script is the <code>annotationStore.csv</code> file, and the output is the <code>queries.csv</code> file.</li> <li><code>2_prompt_generation.py</code>: Script to generate prompts for the LLM-based assistants. The input of this script is the <code>queries.csv</code> file, and the output is the <code>queries_filled.csv</code> file.</li> <li><code>3_gen_sheet_sources_extraction.py</code>: Script to split the external links provided by the LLM, one for each row. The input of this script is the <code>queries_filled.csv</code> file, and the output is the <code>external_link.csv</code> file.</li> <li><code>4_ext_link_snippet_extraction.py</code>: Script to extract the snippets from Web URLs. It only works for the most popular domains. The input of this script is the <code>external_links.csv</code> file, and the output is the <code>external_links_filled.csv</code> file.</li> <li><code>5_top_n_link_SearchEngine.py</code>: Script to perform top-N link search using the corresponding search engines (Google Search and Bing). The input of this script is the <code>queries_filled.csv</code> file. It executes the <code>browser_bot.scpt</code>. The output is the <code>topN_links_se.csv</code> file. <ul> <li><code>browser_bot.scpt</code>: Script for browser automation (AppleScript).</li> </ul> </li> <li><code>6_se_vs_llm.py</code>: Script to compare (using the Jaccard metric) the links returned by the corresponding search engines with those provided by the LLM-based assistants. The input of this script is the <code>external_links.csv</code> file and the <code>topN_links_se.csv</code> file. The output is the <code>jaccaard_analysis.csv</code> file.</li> <li><code>7_results_manual_analysis.py</code>: Script to extract results and statistical analyses performed on the manual analysis and reported in the tables in the paper.</li> <li><code>8_merge_gen_source_snippets.py</code>: This script takes as input: <code>{llm}/queries.csv</code> and <code>{llm}/external_link_filled.csv</code> to merge them and generates an expanded one, i.e., one in which we have on each line a snippet extracted from the source, this will be the input of our final script for clone detection. The output is the <code>merged_llm_ext_link.csv</code> file.</li> <li><code>9_clone_detection.py</code>: Script to perform clone detection. The input of this script is the <code>merged_llm_ext_link.csv</code> file, and the output is the <code>clone_detection_output.csv</code> file.</li> <li><code>10_cosine_sim_check.py</code>: Script to compute the code snippets' cosine similarity. The script takes as input the tokenized files from the <code>{llm}_tokens</code> directory. The output is the <code>cosine_sim_output.csv</code>file.</li> <li><code>11_merger_clone_cosine.py</code>: Script to merge clone detection and cosine similarity results. The input of this script is the <code>clone_detection_output.csv</code> and the <code>cosine_sim_output.csv</code> files, and the output is the <code>merged_clone_cosine.csv</code> file.</li> <li><code>12_merge_manual_quantitative_analysis.py</code>: Script to merge manual and quantitative analysis results. The inputs of this script are the <code>manual_analysis.csv</code> and the <code>merged_clone_cosine.csv</code>files, and the output is the <code>merged_analysis.csv</code> file.</li> <li><code>13_sample_for_COT_analysis.py</code>: Script to collect the sample of queries on which we perform the chain of thought analysis, the output is the <code>sample_queries.csv</code> file.</li> <li><code>14_llm_vs_csn.py</code>: Script to check the overlap between the links provided by the LLM and the one associated with the related query in the CodeSearchNet dataset. The input of this script is the <code>queries.csv</code>and <code>queries_noTrivial.csv</code> file.</li> <li><code>cloningGraph.R</code>: R script to generate the cloning graph. The input of this script is the <code>merged_analysis.csv</code> file.</li> </ul>

ShareScore

32/100

Overall dataset sharing score

Score breakdown

These five areas show where the dataset supports — or may limit — practical reuse.

Stewardship: 4
Harmonization: 4
Access: 16
Reuse readiness: 8
Engagement: 0