TrustMus benchmark: The Role of Large Language Models in Musicology: Are We Ready to Trust the Machines?

<p>TrustMus is an initial, rigorously validated benchmark designed to assess the accuracy and reliability of large language models (LLMs) in the domain of musicology. This dataset includes a collection of 400 human-validated multiple-choice questions, categorized into four thematic areas: People (Ppl), Instruments and Technology (I&amp;T), Genres, Forms, and Theory (Thr), and Culture and History (C&amp;H).</p> <p>The questions are derived from <em>The Grove Dictionary Online</em> using a semi-automated methodology. The process involves generating initial questions with a fine-tuned retrieval-augmented generation (RAG) model, filtering them through a series of automated checks, and finally validating them through expert human annotation. TrustMus is introduced in an initial paper, providing a critical resource for researchers and developers aiming to evaluate and improve LLM performance in this specialized field of musicology.</p> <p>This benchmark is discussed in the paper :&nbsp;</p> <p><strong>BibTeX Citation:</strong></p> <div> <div><code>@inproceedings{ramoneda2024trustmus,</code><br><code>&nbsp; title={The Role of Large Language Models in Musicology: Are We Ready to Trust the Machines?},</code><br><code>&nbsp; author={Ramoneda, Pedro and Parada-Cabaleiro, Emilia and Weck, Benno and Serra, Xavier},</code><br><code>&nbsp; booktitle={Proceedings of the 3rd Workshop on NLP for Music and Audio (NLP4MusA)},</code><br><code>&nbsp; year={2024},</code><br><code>&nbsp; month={November},</code><br><code>&nbsp; address={San Francisco, USA},</code><br><code>&nbsp; organization={Co-located with ISMIR'2024}</code><br><code>}</code><br><br> <div>&nbsp;</div> </div> </div>