LLM-SAT-Eval
<p>Exploring the theme “Do static analysis tools foresee actual bug fixes?," this poster investigates the practical utility of Static Analysis Tools (SATs) in anticipating genuine bug fixes within software code-bases. Leveraging a dataset comprising bug-fix pairs, our study scrutinizes the efficacy of SATs through quantitative analysis, revealing a significant disparity between SAT outputs and actual bug fixes, with nearly 93% of pairs rated at 0. Employing qualitative insights from Gemini-pro Large Language Model, we corroborate these findings, observing prevalent terms such as "buggy line" and "not able" in bug detection. Furthermore, correlation analysis showcases a moderate positive correlation (0.3) between SAT outputs and known bug fixes, emphasizing the importance of alignment for favorable ratings. Our findings underscore the pressing need for enhanced bug detection methodologies, prompting further research in software quality assurance.</p>
ShareScore
36/100
Overall dataset sharing score
Score breakdown
These five areas show where the dataset supports — or may limit — practical reuse.
- Stewardship
- 4
- Harmonization
- 4
- Access
- 20
- Reuse readiness
- 8
- Engagement
- 0