Understanding API Usage and Testing: An Empricial Study of C Libraries (Artifact)

<h1>LibProbe Artifact</h1> <p>For the sake of the evaluation we preprocessed the CCScanner data to identify all clients of the libraries used in our evaluation and included those in our&nbsp;<code>MongoDB</code>&nbsp;database. This artifact will start by importing this pre-processed dependency information into a docker image which is then used for the evaluation.</p> <h2>Running the docker image</h2> <ul> <li> <p>First load the docker image by running</p> <ul> <li> <div> <div><code>gunzip -c libprobe_v2.2.tar.gz | sudo docker import - libprobe:latest </code></div> </div> </li> </ul> <p>this will load the image in your local docker images.</p> </li> <li> <p>Run a container from the image:&nbsp;<code>sudo docker run -it --device /dev/snd --privileged libprobe:latest /bin/bash</code>&nbsp;This will run a docker container which maps the pulseaudio and alsa configurations from your local host to the docker image. This is necessary to get some clients for some target libraries to build correctly. You will need an ubuntu host machine which has pulseaudio and alsa installed.</p> </li> <li>Run&nbsp;<code>mongod --fork --logpath /var/log/mongodb/mongod.log</code>&nbsp;followed by&nbsp;<code>mongorestore --drop --db apiusage /tmp/libprobe/database/apiusage</code>&nbsp;to import the results saved in the artifact.&nbsp;</li> </ul> <h2>Validating analysis results</h2> <ul> <li>The docker image provided does not contain any clients due to size limitations on sharing. The Mongo database contains all the results of running this evaluation.</li> <li>We provide a script <code>clone_clients.py</code> with a <code>client_repos.json </code>file in <code>/tmp/libprobe/extra</code> which can be used to clone the clients in the clients directory.&nbsp;</li> <li>To get the results it's possible to run&nbsp;<code>python3 libprobe.py analyse all -n</code>&nbsp;from&nbsp;<code>/tmp/libprobe</code>. This will overwrite the JSON files in the&nbsp;<code>json_files</code>&nbsp;directory and overwrite the graphs in the&nbsp;<code>graphs</code>&nbsp;directory.</li> </ul> <h2>Running the evaluation for one library (vorbis)</h2> <ul> <li>Download clients: Go to&nbsp;<code>/tmp/libprobe</code>&nbsp;and run&nbsp;<code>python3 libprobe.py download vorbis</code></li> <li>Prepare the library: <ul> <li>In /tmp/data/libraries/xiph@@vorbis run make clean then make and make check and make install.</li> <li>Copy all C files from /tmp/data/libraries/xiph@@vorbis/lib to /tmp/data/libraries/xiph@@vorbis/lib/.libs to collect accurate API coverage information.</li> </ul> </li> <li>Process the library to get the APIs and the coverage information : <code>python3 libprobe.py processlib vorbis</code></li> <li>Prepare clients for excluding sub directories that might contain vorbis library code:&nbsp;<code>python3 libprobe.py prepclients vorbis</code></li> <li>Get client usages:&nbsp;<code>python3 libprobe.py fetchusages vorbis</code></li> <li>Analyse:&nbsp;<code>python3 libprobe.py analyse vorbis -n</code></li> <li>(optional) Measure differential coverage for improved coverage libs:&nbsp;<code>python3 libprobe.py coverage vorbis</code></li> </ul> <h2>Running the evaluation for all libraries (this requires at least 300GB of disk space)</h2> <ul> <li>Download clients: Go to&nbsp;<code>\tmp\libprobe</code>&nbsp;and run&nbsp;<code>python3 libprobe.py download all</code></li> <li>Process the libraries:&nbsp;<code>python3 libprobe.py processlib all</code></li> <li>Prepare clients:&nbsp;<code>python3 libprobe.py prepclients all</code></li> <li>Get usages:&nbsp;<code>python3 libprobe.py fetchusages all</code></li> <li>Analyse:&nbsp;<code>python3 libprobe.py analyse all -n</code></li> <li>(optional) Measure differential coverage for improved coverage libs:&nbsp;<code>python3 libprobe.py coverage &lt;library&gt;</code></li> </ul> <h2>Getting baseline coverage for libraries</h2> <p>All libraries are cloned in&nbsp;<code>/tmp/data/libraries</code>&nbsp;and clients are cloned in&nbsp;<code>/tmp/data/clients</code>.</p> <div>&nbsp;</div> <ul> <li>MBedtls: Copy the script <code>coverage_mbedtls.sh</code> from <code>/tmp/libprobe/extra</code> into the the build directory of Mbedtls <code>/tmp/data/libraries/Mbed-TLS@@mbedtls/build</code> and run <code>./coverage_mbedtls.sh</code> followed by <code>genhtml baseline.info --output-directory</code> out this will calculate the baseline coverage for mbedtls.</li> <li>FFTW: Copy the script coverage.sh from /tmp/libprobe/extra into the the root dir of FFTW and run ./coverage.sh baseline this will calculate the baseline coverage for fftw</li> <li>HDF5: Copy the script coverage_hdf.sh from /tmp/libprobe/extra into the the root dir of HDF and run&nbsp;<code>./coverage_hdf.sh baseline</code>&nbsp;this will calculate the baseline coverage for HDF.</li> <li>LMDB: Copy the script coverage_lmdb.sh from /tmp/libprobe/extra into the&nbsp;<code>/tmp/data/libraries/LMDB@@lmdb/libraries/liblmdb</code>&nbsp;and run&nbsp;<code>./coverage_lmdb.sh baseline</code>&nbsp;this will calculate the baseline coverage for LMDB.</li> <li>Zip: Copy the script coverage_zip.sh from /tmp/libprobe/extra into&nbsp;<code>/tmp/data/libraries/kuba--@@zip/build/CMakeFiles/zip.dir/sr</code>c and run&nbsp;<code>./coverage_zip.sh baseline</code>&nbsp;this will calculate the baseline coverage for zip.</li> <li>Vorbis: Copy the script cal_cov.py from /tmp/libprobe/extra to&nbsp;<code>/tmp/data/libraries/xiph@@vorbis/lib/.libs</code>&nbsp;and then copy all source files in the&nbsp;<code>.libs</code>&nbsp;folder by running&nbsp;<code>cp ../*.c .</code>&nbsp;from the&nbsp;<code>.libs</code>&nbsp;folder. Finally run&nbsp;<code>python3 cal_cov.py .</code>.</li> <li>XXhash: Copy the script coverage.sh from /tmp/libprobe/extra into /tmp/data/libraries/Cyan4973@@xxHash and run ./coverage.sh baseline this will calculate the baseline coverage for xxhash</li> </ul> <h2>Reproducing increased coverage using clients</h2> <ul> <li> <p>LMDB: The client we will use is Knot DNS.</p> <ul> <li>Change directory to&nbsp;<code>/tmp/data/clients/CZ-NIC@@knot</code>&nbsp;and run&nbsp;<code>autogen.sh</code>.</li> <li>Run&nbsp;<code>./configure --with-lmdb=/usr/local</code>.</li> <li>Then&nbsp;<code>make &amp;&amp; make check</code>.</li> </ul> <p>Now go back to the LMDB directory and run</p> <ul> <li> <p>Run&nbsp;<code>./coverage_lmdb.sh after_knot</code>.</p> </li> <li> <p>Now go /tmp/libprobe and run&nbsp;<code>python3 libprobe.py coverage lmdb</code></p> </li> </ul> </li> <li> <p>VORBIS: The client we will use in SFML.</p> <p>Go to the vorbis library dir&nbsp;<code>/tmp/data/libraries/xiph@@vorbis</code>&nbsp;and run&nbsp;<code>make clean</code>.</p> <ul> <li>Run&nbsp;<code>make &amp;&amp; make check &amp;&amp; make install</code>.</li> <li>Go to the&nbsp;<code>.libs</code>&nbsp;folder and copy all c files there by doing&nbsp;<code>cp ../*.c .</code>.</li> <li>Copy&nbsp;<code>/tmp/libprobe/extra/cal_cov.py</code>&nbsp;into the&nbsp;<code>.libs</code>&nbsp;folder and run&nbsp;<code>python3 cal_cov.py .</code>. This will show the baseline coverage.</li> </ul> <p>Go to the SFML directory&nbsp;<code>/tmp/data/clients/SFML@@SFML</code>.</p> <ul> <li>Create build directory&nbsp;<code>mkdir build &amp;&amp; cd build</code>.</li> <li>Run&nbsp;<code>cmake -DSFML_BUILD_TEST_SUITE=TRUE -GNinja ..</code>.</li> <li>Run&nbsp;<code>ninja</code>.</li> <li>Run&nbsp;<code>ctest</code>. You will see some failing tests. Thats normal as we are only interested in the Audio tests for vorbis. All Audio tests should pass.</li> </ul> <p>Go back to the&nbsp;<code>.libs</code>&nbsp;folder in vorbis and re-run the&nbsp;<code>cal_cov.py</code>&nbsp;script.</p> <ul> <li>Now go /tmp/libprobe and run&nbsp;<code>python3 libprobe.py coverage vorbis</code></li> </ul> </li> </ul> <ul> <li>SDL: The client we will use in UFOAI.<br> <ul> <li> <p>Go to the SDL library dir <code>/tmp/data/libraries/libsdl-org\@\@SDL</code> and then the <code>build2</code> directory where the built library is. .</p> </li> <li>Run&nbsp;<code>make clean &amp;&amp; make &amp;&amp; make install &amp;&amp; make test</code>.</li> <li>Copy&nbsp;<code>/tmp/libprobe/extra/coverage_sdl.sh</code> into the <code>build2</code> folder and run <code>./coverage_sdl.sh baseline</code>. Run <code>genhtml baseline.info --output-directory out</code></li> </ul> </li> </ul> <p>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Go to the UFOAI directory <code>/tmp/data/clients/ufoaiorg\@\@ufoai</code></p> <ul> <li> <ul> <li>Run&nbsp;<code>./configure --target-os=linux --disable-uforadiant</code> &amp;&amp; <code>make</code>.</li> <li>Run .<code>/testall</code></li> </ul> </li> </ul> <p>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Now go /tmp/libprobe and run <code>python3 libprobe.py coverage sdl</code></p> <p>&nbsp;</p> <ul> <li>FFTW: The client we will use in CAVA.</li> </ul> <p>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Go to the FFTW3 library dir <code>/tmp/data/libraries/FFTW@@fftw3</code> and run <code>reset_cov.sh</code> then <code>make clean</code></p> <ul> <li> <ul> <li>Run&nbsp;<code>make &amp;&amp; make install &amp;&amp; make check</code>.</li> <li>Copy&nbsp;<code>/tmp/libprobe/extra/coverage.sh</code> into root directory of the library and run the&nbsp;<code>.libs</code>&nbsp;folder and run&nbsp;<code>./coverage.sh baseline</code>&nbsp;This will show the baseline coverage.</li> </ul> </li> </ul> <p>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Go to the CAVA directory <code>/tmp/data/clients/karlstav@@cava</code>.</p> <ul> <li> <ul> <li>Run <code>./autogen.sh</code> followed by <code>./configure</code> then <code>make</code></li> <li>Run the script <code>./run_all_tests.sh</code> .&nbsp;</li> </ul> </li> </ul> <p>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Go back to the FFTW3 library and run <code>./coverage.sh after_cava</code></p> <ul> <li> <ul> <li>Now go /tmp/libprobe and run&nbsp;<code>python3 libprobe.py coverage fftw3</code></li> </ul> </li> </ul>