Evaluating Audio Understanding in Multimodal AI

Objectives

• Breadth of capability: How many distinct audio analysis tasks can a multimodal model meaningfully perform from a single voice recording?
• Acoustic vs. content inference: Can the model distinguish between what it hears in the audio signal and what it understands from the speech content?
• Safety boundaries: Where does the model draw ethical lines on sensitive inferences (health, demographics, deception)?
• Practical utility: Are the outputs actionable for real-world use cases like audio production, voice cloning, and speech coaching?
• Internal consistency: Does the model maintain a coherent characterisation of the speaker across dozens of independent prompts?

Methodology

A single freeform voice recording was made by Daniel Rosehill on a OnePlus Nord 3.5G phone in HQ mode (WAV 44.1kHz, converted to FLAC mono 24kHz). The recording is unscripted, covering topics from voice cloning and TTS technology to personal background and current events. The speaker was fatigued from disrupted sleep, providing a natural test of the model's ability to detect vocal state.

137 test prompts were designed across 22 categories. 49 prompts were implemented with full prompt text and executed against the audio using Gemini 3.1 Flash Lite via the Google Generative AI API. Each prompt was run independently with the full audio file as context. The remaining 88 prompts are catalogued as suggested extensions.

The Voice Sample

The recording features a male speaker in his late 30s with an Irish accent (Cork origin), living in Jerusalem for ~11 years. Voice type: bass/low baritone (median F0 ~110 Hz). Speaking rate: ~169 WPM. Recorded in an untreated room while pacing. A timestamped transcript (97.4% confidence, AssemblyAI) and detailed acoustic analysis (pitch, formants, signal levels) are included in the dataset.

Explore

Cite This Work

Dataset: danielrosehill/Audio-Understanding-Test-Set · Source: GitHub · DOI: 10.57967/hf/8154