Researchers have introduced VākQA, a novel benchmark designed to evaluate Spoken Factoid Question Answering (SQA) in Telugu, a language previously lacking dedicated spoken benchmarks. The dataset comprises 2,001 question-answer pairs spanning six distinct domains, supported by 2.53 hours of speech audio, bilingual transcriptions, and human-verified reference answers. The study first investigates the efficacy of automated evaluation metrics against human judgment. Findings indicate that while Gemini-as-a-judge most closely aligns with human ratings, it exhibits non-uniform strictness; conversely, open-weight models tend to disproportionately penalize correct Telugu responses that deviate from reference answers in surface form. Benchmarking across various proprietary and open-weight models reveals significant linguistic and technical challenges. Specifically, the researchers identified that Telugu phrasing contains cultural specificities often lost during translation and that speech-based inputs introduce phonetic confusions that can fundamentally alter question intent. Furthermore, the study highlights the compounding effect of error propagation in cascaded systems, where errors from Automatic Speech Recognition (ASR) and Machine Translation (MT) accumulate progressively. VākQA provides a foundational resource for advancing SQA capabilities in low-resource, morphologically rich languages.
Read original
huggingface/daily-papers