Skip to main content
ConductScienceScienceDex
zenodoopen

The Artifact of the ESEC/FSE 2023 Paper Titled "Natural Language to Code: How Far are We?"

<p>In this online repository, we release the source code of each of the selected techniques as well as the experiment results from each technique (which are stored in the <em>Results.zip</em> file). For each technique, we also provide our scripts to fine this approach on the CodeSearchNet-Python dataset. For example,&nbsp;<em>finetune.sh/inference.sh</em> are used to finetune/evaluate CodeBERT and they are under &quot;CodeBERT/CodeBERT&quot;.</p> <p>&nbsp;</p> <p>Our evaluation dataset CodeSearchNet is a well-known benchmark and it can be downloaded on its <a href="http://github.com/github/CodeSearchNet">official webpage</a>.</p> <p>&nbsp;</p> <p>The code to calculate the evaluation metrics are reused from <a href="https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/code-to-code-trans/evaluator/CodeBLEU">CodeBLEU</a>.</p> <p>&nbsp;</p> <p>Below is a piece of code generated by CodeT5. In this case, CodeT5&nbsp;generates a statement&nbsp;recurrently, which leads to the syntactic error. Despite that, the code itself fulfills certain functionalities, and that is why it can achieve a CodeBLEU of 24.9%.</p> <pre><code class="language-python">def makeMimiLocal(filename): try: with open(filename, 'rb') as f: data = f.read() except IOError: data = b'' data = data.decode('utf-8') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\x00', b'\x00') data = data.replace(b'\ </code></pre> <p>&nbsp;</p> <p>We also release the 100 randomly-selected queries as well as the&nbsp;code generated by ChatGPT&nbsp;in the <em>chatGPT.jsonl</em>.</p>

ShareScore

36/100

Overall dataset sharing score

Score breakdown

These five areas show where the dataset supports — or may limit — practical reuse.

Stewardship
4
Harmonization
4
Access
20
Reuse readiness
8
Engagement
0

Dataset details

Repository
zenodo
Published
August 17, 2023
License
cc-by-4.0
DOI
10.5281/zenodo.7546357

Scored with release 0.9.0

ScienceDex for faculty

Your academic CV and NIH biosketch

Make your CV in your school's format and your NIH biosketch from it, then download both as Word files.