ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence Menghe Ma*
Beijing University of Posts and Telecommunications Beijing, China [email protected]
arXiv:2604.20719v1 [cs.SD] 22 Apr 2026
Yaheng Wang
Beijing University of Posts and Telecommunications Beijing, China [email protected]
Siqing Wei*
Beijing University of Posts and Telecommunications Beijing, China [email protected]
Fanhong Meng
China Conservatory of Music Beijing, China [email protected]
Luu Anh Tuan
Nanyang Technological University Singapore, Singapore [email protected]
Yuecheng Xing*
Beijing University of Posts and Telecommunications Beijing, China [email protected]
Peijun Han
China Conservatory of Music Beijing, China [email protected]
Haoran Luo†
Nanyang Technological University Singapore, Singapore [email protected]
Abstract Omnimodal Notation Processing (ONP) represents a unique frontier for omnimodal AI due to the rigorous, multi-dimensional alignment required across auditory, visual, and symbolic domains. Current research remains fragmented, focusing on isolated transcription tasks that fail to bridge the gap between superficial pattern recognition and the underlying musical logic. This landscape is further complicated by severe notation biases toward Western staff and the inherent unreliability of "LLM-as-a-judge" metrics, which often mask structural reasoning failures with systemic hallucinations. To establish a more rigorous standard, we introduce ONOTE, a multiformat benchmark that utilizes a deterministic pipeline—grounded in canonical pitch projection—to eliminate subjective scoring biases across diverse notation systems. Our evaluation of leading omnimodal models exposes a fundamental disconnect between perceptual accuracy and music-theoretic comprehension, providing a necessary framework for diagnosing reasoning vulnerabilities in complex, rule-constrained domains.
CCS Concepts • Computing methodologies → Artificial intelligence.
Keywords Omnimodal Large Language Models, Omnimodal Notation Processing, Benchmark, Cross-Modal Reasoning, Audio-to-Symbolic Transcription, Music Generation
1
Introduction
While Omnimodal Large Language Models (OLLMs) excel in natural language and audio-visual tasks, omnimodal Notation Processing (ONP) remains a formidable challenge. Music requires precise ∗ Equal contribution.
† Corresponding author.
Figure 1: Task formulation of Omnimodal Notation Processing and evaluation framework spatial-temporal alignment across auditory, visual, and symbolic representations (see Figure 1). Unlike standard text, musical notation demands strict adherence to multi-dimensional physical and temporal constraints simultaneously. Recent advancements have shifted focus from audio synthesis [2, 6] to generating code-based Data :https://huggingface.co/datasets/Weisiqing123/ONOTE Code : https://github.com/T12knightally/ONOTE
Conference’17, July 2017, Washington, DC, USA
Menghe Ma* , Siqing Wei* , Yuecheng Xing* , Yaheng Wang, Fanhong Meng, Peijun Han, Luu Anh Tuan, and Haoran Luo†
Figure 2: This framework establishes a deterministic evaluation metric for ONP by benchmarking OLLMs across three notation formats and four distinct tasks. It eliminates subjective scoring biases by projecting all omnimodal outputs into a unified 1D pitch array to calculate exact accuracy via sequence alignment. symbolic representations [12, 19, 28, 53] and employing deep learning or vision-language architectures for Optical Music Recognition (OMR) [7, 8, 14]. However, these efforts remain heavily fragmented. Such disjointed, narrowly-focused systems fail to capture the holistic cognitive process required to seamlessly map visual scores to their underlying musical logic. Consequently, an AI that performs well on isolated transcription tasks often struggles to comprehend the generative syntax or structural rules of the same musical piece [5, 52]. Current models also exhibit severe notation biases, prioritizing Western staff over globally prevalent systems like Jianpu and Guitar Tablature, leading to catastrophic reasoning failures. Furthermore, subjective “LLM-as-a-judge” evaluations are fundamentally flawed for music due to algorithmic alignment deficiencies, obscuring vulnerabilities and causing systemic hallucinations. To address this, we introduce ONOTE, a comprehensive benchmark evaluating the entire lifecycle of symbolic music cognition across three notation systems and four task tracks. Its deterministic pipeline uses canonical pitch projection and sequence alignment to ensure objectivity, completely eliminating “LLM-as-a-judge” biases. Extensive experiments on baseline LLMs reveal a clear discrepancy between perceptual recognition and music-theoretic reasoning. Because researchers are unable to accurately quantify how culturally and structurally distinct formats impact overall model performance without a ground-truth metric, ONOTE provides a deterministic evaluation mechanism to diagnose these limitations and encourage deep, structurally sound musical comprehension.
2
Related Work
Omni-Modal Large Language Models. Recent advancements in foundational Large Language Models (LLMs) [15, 44] have rapidly catalyzed the evolution toward an “omnimodal” paradigm [48], where architectures natively process text, vision, and audio without cascaded modules. Pioneering commercial models like GPT-4o [35] and Gemini 1.5 [43], alongside influential open-source visionlanguage [3, 25, 26] and audio-language architectures [11, 39, 55], have achieved unprecedented cross-modal reasoning capabilities. While excelling in standard OCR and ASR, these models struggle with symbolic music’s strict rules, and ’LLM-as-a-judge’ evaluations introduce further bias by failing to reliably assess structural logic. AI in Music Processing. While early research primarily focused on generative acoustics and audio synthesis [2, 6], a recent paradigm shift has directed attention toward Omnimodal Notation Processing [12, 18, 19]. Systems like MuseCoco [28], ChatMusician [53], and recent autoregressive frameworks [20, 46, 52] demonstrate the immense potential of Large Language Models (LLMs) to understand and compose symbolic codes [27, 42, 54]. Concurrently, Optical Music Recognition (OMR) has transitioned from traditional heuristic pipelines toward end-to-end deep learning and vision-language architectures [4, 7, 8, 36, 41]. To address the severe notation bias in existing research that prioritizes Western staff notation [17, 32] while neglecting systems like Jianpu, Guitar Tablature [40, 47], and traditional folk codes [9, 50], ONOTE provides a systematic, multi-format benchmark across these heterogeneous musical representations.
ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence
Conference’17, July 2017, Washington, DC, USA
Figure 3: Overview of the ONOTE benchmark design workflow. The framework categorizes the evaluation into four core tasks (VSU, CNC, AST, and SMG), systematically illustrating the specific input modalities, analytical process flows, and objective output metrics for each task.
3
Tasks
4
In this section, we formally define the Omnimodal Notation Processing (ONP) task and our proposed evaluation framework. Unlike general text generation, ONP requires large language models (LLMs) to map highly heterogeneous inputs into precise, domain-specific symbolic representations. Let X = X𝑖𝑚𝑔 ∪ X𝑎𝑢𝑑𝑖𝑜 ∪ X𝑡𝑒𝑥𝑡 denote the omnimodal input space (comprising score images, audio waveforms, and textual prompts), and Y denote the discrete symbolic output space (e.g., standard staff, tablature, Jianpu). The baseline objective of the model, parameterized by 𝜃 , is to decode an optimal symbolic sequence 𝑦 ∗ ∈ Y that maximizes the conditional probability given the input 𝑥 and the music theory constraint set Ω: 𝑦 ∗ = arg max 𝑃𝜃 (𝑦|𝑥, Ω) 𝑦∈Y
(1)
To comprehensively quantify the model’s capabilities, our benchmark decomposes the evaluation into four distinct experimental tracks T = {𝑇𝑉 𝑆𝑈 ,𝑇𝐶𝑁𝐶 ,𝑇𝐴𝑆𝑇 ,𝑇𝑆𝑀𝐺 }. We formulate a unified capability scoring function 𝐶 (𝜃 ) to evaluate the model across these four orthogonal dimensions: ∑︁ ˆ 𝐶 (𝜃 ) = 𝜆𝑘 E (𝑥,𝑦) (2) ˆ ∈ D𝑘 [Φ𝑘 (𝑓𝜃 (𝑥), 𝑦)] 𝑘∈T
where D𝑘 represents the dataset specific to task 𝑘, 𝜆𝑘 are the tradeoff weights balancing the task dimensions, and Φ𝑘 denotes the specific objective evaluation metric for each track. The detailed definitions of these four tasks, their input modalities, representational logic, and the exact formulations of their respective evaluation metrics are thoroughly elaborated in Section 4.
Benchmark
This section introduces the modeling of our benchmark, ONOTE, a novel benchmark engineered for the evaluation of Omnimodal Notation Processing (ONP) and transcription tasks. Detailed explanations are provided regarding how the benchmark was constructed, the evaluation protocols, and the corresponding metrics used to quantify model performance.
4.1
Dataset Construction
The construction of the dataset follows a rigorous three-stage pipeline. First, we extracted initial diverse data distributions from the MusiXQA dataset [10] alongside the GuitarSet dataset [51], drawing inspiration from the representational diversity seen in massive corpora like MAESTRO [17], Slakh [32], and DadaGP [40]. This initial collection comprises PNG images, structured JSON annotations, and MIDI alignment files [38] of 100 staff notations, as well as JAMS annotation files of 80 guitar tabs and their corresponding WAV audio recordings. The raw data then underwent rigorous cleaning and cross-modal alignment, and was subsequently transformed into Numbered Musical Notation (Jianpu) PNG images, MP3 files, and ASCII Guitar Tab PNG images. Finally, we designed targeted single-choice questions for each image sample according to the unique representational logic of each notation system. Following this pipeline, a total of 1,120 highquality test samples were constructed for evaluation.
4.2
Benchmark Design
To comprehensively assess the capabilities of LLMs across different dimensions of ONP, ONOTE isolates four distinct evaluation tasks (see figure 3). Each task is tailored with specific input modalities,
Conference’17, July 2017, Washington, DC, USA
Menghe Ma* , Siqing Wei* , Yuecheng Xing* , Yaheng Wang, Fanhong Meng, Peijun Han, Luu Anh Tuan, and Haoran Luo†
representational logic, and objective metrics, directly corresponding to the theoretical tracks defined in Equation . Task 1: Visual Score Understanding (VSU) This task (Φ𝑣𝑖𝑠 ) assesses direct comprehension of musical images. While recent models improve polyphonic recognition [1, 49], our VQA format follows agentic reasoning paradigms [29, 31] to identify symbols within complex topologies [30] without textual hints. Performance is measured via standard classification accuracy. Task 2: Cross-Format Notation Conversion (CNC) This task (Φ𝑐𝑜𝑛𝑣 ) tests the model’s translation capabilities across notation formats. The input consists of a score image in one format (e.g., standard staff), and the model is required to accurately translate it into another designated format (e.g., textual Jianpu or ASCII Guitar Tablature). This evaluates whether the model has truly mastered the underlying musicological mappings (such as fixed-do to movable-do conversions) rather than merely memorizing visual shapes. Accuracy is validated by cross-checking the theoretical equivalence of the converted outputs against the ground-truth JSON. Task 3: Audio-to-Symbolic Transcription (AST). This task (Φ𝑡𝑟𝑎𝑛𝑠 ) diagnoses the model’s audio perception and transcription capability. Building upon recent advances in high-resolution onset/offset detection [16, 21, 22] and multi-track sequence modeling [13, 24], models receive segmented 10-second audio chunks (MP3/WAV) and must output the corresponding notation strings (e.g., ABC code, ASCII Tab). To strictly eliminate the “LLM-as-ajudge” bias, we extract absolute scientific pitches from the groundtruth JSON and logically translate the generated sequences into the same flattened format. A deterministic sequence-matching estimator (based on the Levenshtein Edit Distance algorithm [23]) is then employed to calculate the exact sequential alignment accuracy. Task 4: Symbolic Music Generation & Aesthetics (SMG) This task (Φ𝑐𝑜𝑚𝑝 ) assesses the model’s generative capacity through a dual-axis evaluation. First, we evaluate the Syntactic Renderability by passing the generated code into standard rendering engines (e.g., MuseScore [34] or ABC compilers [45]) to check for fatal syntax errors. Second, we evaluate the Musical Aesthetics, which focuses on the structural coherence and overall beauty of the generated composition[37] based on specific prompts. This ensures the output is not only formally correct but also aesthetically valid as music.
4.3
Deterministic and Anti-Bias Evaluation Paradigm
Given the propensity of LLMs to hallucinate during mathematical verification and long-context self-evaluation, relying on “LLM-asa-judge” introduces systemic grading bias. To fundamentally eliminate this and operationalize the metrics for the aforementioned tasks (particularly CNC and AST), we propose a programmatic evaluation pipeline that quantifies alignment accuracy via deterministic distance metrics. 4.3.1 Canonical Pitch Space Projection. We first introduce a deterministic projection function F (·) that maps both the ground-truth JSON annotation 𝑦 (𝑛) and the predicted raw string output 𝑦ˆ (𝑛) into flattened, chronologically ordered sequences of absolute scientific pitches: 𝑆𝑔𝑡 = F (𝑦 (𝑛) ), 𝑆𝑝𝑟𝑒𝑑 = F (𝑦ˆ (𝑛) ) (3)
To achieve cross-format unification, the decoding logic dynamically adapts to the source notation. For instrument-specific physical representations like guitar tablature, the conversion to a unified MIDI pitch[33] 𝑀 (𝑠, 𝑓 ) accounts for fretboard mechanics (string 𝑠 ∈ {1, ..., 6}, fret 𝑓 ∈ {0, ..., 24}) via standard tuning constraints: 𝑀𝑡𝑎𝑏 (𝑠, 𝑓 ) = 𝐵𝑚𝑖𝑑𝑖 (𝑠) + 𝑓
(4)
where 𝐵𝑚𝑖𝑑𝑖 is the base MIDI pitch of the open strings. Conversely, for relative pitch systems like Jianpu, the mapping incorporates key signature offsets 𝐾 and octave modifiers 𝑜: 𝑀 𝑗𝑖𝑎𝑛𝑝𝑢 (𝑑, 𝑜, 𝐾) = Base(𝐾) + Interval(𝑑) + 12 × 𝑜
(5)
where 𝑑 ∈ {1..7} represents the diatonic scale degree. These discrete MIDI values are subsequently mapped to the standard scientific pitch notation P (𝑀) [33]. Simultaneous notes (chords) occurring at the same temporal frame 𝑡 are sorted by pitch height to ensure permutation-invariant sequence flattening, effectively neutralizing the impact of arbitrary transcription orders: 𝑆 [𝑡] = Sort({P (𝑀𝑖 ) | ∀𝑖 ∈ Chord𝑡 })
(6)
4.3.2 Sequence Alignment and Task Metrics. To explicitly enforce logical and temporal consistency across these 1D pitch arrays, we compute the optimal alignment using the Levenshtein (Edit) Distance algorithm (ED) [23]. The alignment accuracy is designed to heavily penalize temporal drift and hallucinated notes: 𝐸𝐷 (𝑆𝑔𝑡 , 𝑆𝑝𝑟𝑒𝑑 ) 𝐴𝑐𝑐 (𝑆𝑔𝑡 , 𝑆𝑝𝑟𝑒𝑑 ) = max 0, 1 − (7) max(|𝑆𝑔𝑡 |, |𝑆𝑝𝑟𝑒𝑑 |) Crucially, utilizing max(|𝑆𝑔𝑡 |, |𝑆𝑝𝑟𝑒𝑑 |) as the denominator acts as a stringent penalty mechanism against systemic hallucinations. When an LLM suffers from catastrophic temporal drift and generates infinitely repeating or excessively long sequences (|𝑆𝑝𝑟𝑒𝑑 | ≫ |𝑆𝑔𝑡 |), the inflated denominator and the surge in required deletion operations aggressively compress the accuracy score toward zero. This sequence accuracy is aggregated into specific task metrics. ˆ is implemented as a For instance, the evaluation function Φ(𝑦, 𝑦) multi-granularity hybrid metric: ˆ = 𝑤 𝑝𝑖𝑡𝑐ℎ 𝐴𝑐𝑐 𝑝𝑖𝑡𝑐ℎ + 𝑤𝑑𝑢𝑟 𝐴𝑐𝑐𝑑𝑢𝑟 + 𝑤 𝑓 𝑚𝑡 I 𝑓 𝑚𝑡 (𝑦) ˆ Φ(𝑦, 𝑦)
(8)
ˆ ∈ {0, 1} is a binary indicator verifying strict adherwhere I 𝑓 𝑚𝑡 (𝑦) ence to requested syntactic formats via regular expressions.
5 Experiments and Analysis 5.1 Experimental Setup To evaluate OLLM performance on symbolic music analysis and generation, a set of representative state-of-the-art models was assessed using the ONOTE benchmark. This benchmark encompasses three primary notation systems (Standard Staff, Jianpu, and Guitar Tablature) and is designed to cover four diverse tasks: Symbolic Music Generation (SMG), Cross-Format Notation Conversion (CNC), Visual Score Understanding (VSU), and Audio-to-Symbolic Transcription (AST). All models were evaluated under a unified protocol, with specific metric decompositions (such as Pitch/Duration for AST, and Aesthetic/Technical/Fingering for SMG). The main results are summarized in Table 1.
Conference’17, July 2017, Washington, DC, USA
ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence
Table 1: Comprehensive evaluation results of various OLLMs on ONOTE. The tasks include Symbolic Music Generation (SMG), Cross-Format Notation Conversion (CNC), Visual Score Understanding (VSU), and Audio-to-Symbolic Transcription (AST). Bold values indicate the best performance, and underlined values indicate the second best. Standard Staff
Jianpu
Guitar Tablature
Models
SMG (Score)
CNC (Acc. %)
VSU (Acc. %)
AST (Acc. %)
SMG (Score)
CNC (Acc. %)
VSU (Acc. %)
AST (Acc. %)
SMG (Score)
CNC (Acc. %)
VSU (Acc. %)
AST (Acc. %)
Baichuan-Omni-1.5 Qwen2.5-Omni-7b Qwen-Omni-turbo Qwen3-Omni-flash Gemini-2.5-flash Gemini-2.5-pro Gemini-3.1-flash-lite-preview
1.24 4.51 2.07 3.84 1.31 3.03 4.47
18.54 14.27 14.72 17.31 12.98 17.04 17.29
4.00 44.00 48.00 88.00 45.00 97.00 99.00
3.96 3.79 8.55 9.32 4.11 7.50 7.61
1.39 1.07 1.39 1.86 1.52 4.33 4.72
5.51 8.62 8.86 5.49 9.44 23.04 13.06
19.8 65.30 62.38 82.10 46.07 90.38 80.20
14.75 20.63 14.78 17.96 19.85 15.67 24.32
1.67 2.67 2.79 3.19 1.17 3.71 3.68
6.42 7.05 7.45 4.07 46.08 43.58 22.47
18.5 80.2 60.49 94.37 36.00 82.72 93.83
1.53 3.30 4.32 2.55 2.17 2.57 1.64
Figure 4: This figure analyzes the structural and cognitive bottlenecks of OLLMs, specifically the performance collapse when transitioning from 1D to 2D notations and the technical-aesthetic imbalance in symbolic generation.
5.2
Cognitive Bottlenecks and Hallucinations in Notation Conversion
Although leading models achieve high accuracy in Visual Score Understanding (VSU), their performance on Cross-Format Notation Conversion (CNC) remains comparatively low. This stark contrast highlights a systemic mismatch between visual recognition and relational reasoning. For instance, while Gemini-3.1-flash-litepreview excels in Standard Staff VSU (99.00%), its CNC score drops significantly to 17.29%. This indicates that while the optical character recognition of sheet music is effective, the relational mapping of pitch and temporal values fails. Instead of true cross-modal deduction, models heavily rely on pre-training text distributions. Gemini-2.5-flash, for example, scores poorly on Guitar Tablature VSU (36.00%) yet achieves the highest CNC accuracy (46.08%), implying a probabilistic estimation based on ASCII corpora rather than visual-to-symbolic reasoning. Qualitative analysis of CNC tasks reveals persistent bidirectional errors, indicating a severe disconnect between visual parsing and musicological constraints. When converting complex 3/4 Standard Staff melodies to Jianpu, models often fail to capture meter, defaulting to uniform quarter-note durations and sequential scale
generation driven by textual priors. Conversely, in the reverse Jianpu-to-Staff task, models lose their cross-modal anchor and suffer from severe autoregressive degeneration, falling into infinite decoding loops of generic arpeggiated patterns. Together, these bidirectional pathologies—rhythmic flattening and prior-induced melodic hallucinations—demonstrate that current OLLMs approach notation processing primarily as weaklyconditioned text continuation rather than precise spatial-temporal alignment. This fundamental limitation underscores the necessity of ONOTE’s deterministic evaluation and knowledge-grounded enhancement mechanisms.
5.3
Dimensional Analysis in Audio-to-Symbolic Transcription
The breakdown of AST metrics reveals differences in acoustictemporal alignment capabilities, which vary depending on the spatial dimensionality of the target notation. Specifically, models generally exhibit higher accuracy when mapping audio to linear, one-dimensional formats, compared to navigating the complex, two-dimensional coordinate space required for Standard Staff.
Conference’17, July 2017, Washington, DC, USA
Menghe Ma* , Siqing Wei* , Yuecheng Xing* , Yaheng Wang, Fanhong Meng, Peijun Han, Luu Anh Tuan, and Haoran Luo†
Figure 5: Four typical failure cases and underlying causes in omnimodal notation processing. Current models struggle with cognitive bottlenecks and autoregressive hallucinations due to a lack of structural reasoning, while concurrently failing to align audio-visual dimensions and violating physical playability constraints. Linear acoustic alignment. For 1D linear notations like Jianpu, models show a moderate capacity for independent feature extraction. Gemini-3.1-flash achieves 61.74% in Pitch and 57.51% in Duration.However, the combined strict Full accuracy decreases to 24.32%, indicating difficulties in precisely aligning frequency variations with temporal sequences on a unified axis. Spatial coordinate mapping. In 2D notations such as Standard Staff, which involve complex polyphony, performance limitations become more apparent. While rhythmic perception (Duration) maintains moderate levels, Pitch accuracy is notably lower across all models (e.g., Gemini-2.5-flash at 11.05%, Qwen3-Omni-flash at 19.64%). Consequently, Full accuracy scores remain below 10%, suggesting that current architectures face challenges in disentangling overlapping acoustic spectra and mapping them accurately to a two-dimensional coordinate system.
5.4
Constraint Satisfaction in Symbolic Music Generation
Analyzing the generative sub-attributes indicates a divergence in how models handle structural compliance, physical constraints, and artistic output. Syntax and aesthetics trade-offs. In Standard Staff generation, models exhibit varying approaches to structural compliance. Qwen3-Omni-flash achieves a maximum Technical score (5.0) for adhering to measure constraints but yields a lower Aesthetic score (2.69). Conversely, Gemini-3.1-flash maintains a more balanced distribution between Technical (4.99) and Aesthetic (3.92) scores. In
Jianpu generation, the Qwen series shows limited syntactic alignment (Technical scores ≤ 2.21), whereas Gemini-3.1-flash demonstrates higher compliance with the specific localized notation rules. Physical and spatial constraints. Generating Guitar Tablature introduces additional constraints, including ergonomic feasibility (Fingering) and strict ASCII spatial layout (Technical). Gemini-2.5pro manages these multi-objective constraints relatively well, scoring 4.125 in both Fingering and Technical formatting. In contrast, while Qwen-Omni-turbo achieves high scores in fingering rationality (4.27) and musicality (3.08), its technical formatting score remains limited (1.04). These results indicate that maintaining specific text-matrix spatial configurations alongside aesthetic and physical parameters is an area requiring further development.
6
Conclusion
In this paper, we introduced ONOTE, a comprehensive benchmark for evaluating Omnimodal Notation Processing (ONP) across standard staff, Jianpu, and Guitar Tablature through four complementary tasks. To overcome the biases of subjective and model-based judging, we proposed a deterministic evaluation pipeline utilizing canonical pitch projection and sequence alignment for reproducible assessment. Our extensive experiments on state-of-the-art models reveal that while current systems excel in visual recognition, they exhibit substantial limitations in music-theoretic reasoning, temporal alignment, and structured symbolic generation. Ultimately, ONOTE provides a scalable testbed that highlights the intrinsic challenges of highly structured domains, aiming to drive the development of more robust, interpretable, and musically aware omnimodal systems.
ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence
References
[1] Rana L Abdulazeez and Fattah Alizadeh. 2024. Deep Learning-Based Optical Music Recognition for Semantic Representation of Non-overlap and Overlap Music Notes. ARO-THE SCIENTIFIC JOURNAL OF KOYA UNIVERSITY 12, 1 (2024), 79–87. [2] Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. 2023. MusicLM: Generating Music From Text. arXiv:2301.11325 [cs.SD] https://arxiv.org/abs/2301.11325 [3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and J Qwen-VL Zhou. 2023. A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 6 (2023), 3. [4] Arnau Baró, Pau Riba, Jorge Calvo-Zaragoza, and Alicia Fornés. 2017. Optical Music Recognition by Recurrent Neural Networks.. In GREC@ ICDAR. 25–26. [5] Keshav Bhandari and Simon Colton. 2024. Motifs, phrases, and beyond: The modelling of structure in symbolic music generation. In International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar). Springer, 33–51. [6] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. 2023. AudioLM: A Language Modeling Approach to Audio Generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023), 2523–2533. doi:10.1109/TASLP.2023.3288409 [7] Jorge Calvo-Zaragoza, Juan C. Martinez-Sevilla, Carlos Penarrubia, and Antonio Rios-Vila. 2023. Optical Music Recognition: Recent Advances, Current Challenges, and Future Directions. In Document Analysis and Recognition – ICDAR 2023 Workshops, Mickael Coustaty and Alicia Fornés (Eds.). Springer Nature Switzerland, Cham, 94–104. [8] Jorge Calvo-Zaragoza and David Rizo. 2018. End-to-end neural optical music recognition of monophonic scores. Applied Sciences 8, 4 (2018), 606. [9] Luca Casini and Bob Sturm. 2022. Tradformer: A transformer model of traditional music transcriptions. In International Joint Conference on Artificial Intelligence IJCAI 2022, Vienna, Austria, 23-29 July 2022. 4915–4920. [10] Jian Chen, Wenye Ma, Penghang Liu, Wei Wang, Tengwei Song, Ming Li, Chenguang Wang, Jiayu Qin, Ruiyi Zhang, and Changyou Chen. 2025. MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models. arXiv:2506.23009 [cs.CV] https://arxiv.org/abs/2506.23009 [11] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models. arXiv:2311.07919 [eess.AS] https://arxiv.org/abs/2311.07919 [12] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023. Simple and controllable music generation. Advances in neural information processing systems 36 (2023), 47704–47720. [13] Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel. 2021. MT3: Multi-task multitrack music transcription. arXiv preprint arXiv:2111.03017 (2021). [14] Carlos Garrido-Munoz, Antonio Rios-Vila, and Jorge Calvo-Zaragoza. 2022. A holistic approach for image-to-graph: application to optical music recognition: C. Garrido-Munoz et al. International Journal on Document Analysis and Recognition (IJDAR) 25, 4 (2022), 293–303. [15] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024). [16] Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. 2017. Onsets and frames: Dual-objective piano transcription. arXiv preprint arXiv:1710.11153 (2017). [17] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. 2018. Enabling factorized piano music modeling and generation with the MAESTRO dataset. arXiv preprint arXiv:1810.12247 (2018). [18] Wen-Yi Hsiao, Jen-Yu Liu, Yin-Cheng Yeh, and Yi-Hsuan Yang. 2021. Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 178–186. [19] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. 2018. Music transformer. arXiv preprint arXiv:1809.04281 (2018). [20] Yu-Siang Huang and Yi-Hsuan Yang. 2020. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In Proceedings of the 28th ACM international conference on multimedia. 1180–1188. [21] Jinjing Jiang, Nicole Teo, Haibo Pen, Seng-Beng Ho, and Zhaoxia Wang. 2024. Converting vocal performances into sheet music leveraging large language models. In 2024 IEEE International Conference on Data Mining Workshops (ICDMW). IEEE, 445–452.
Conference’17, July 2017, Washington, DC, USA
[22] Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang. 2021. High-resolution piano transcription with pedals by regressing onset and offset times. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021), 3707–3717. [23] Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, Vol. 10. Soviet Union, 707–710. [24] Dichucheng Li, Yongyi Zang, and Qiuqiang Kong. 2025. Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoders. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5. [25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. PMLR, 19730–19742. [26] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems 36 (2023), 34892–34916. [27] Jiafeng Liu, Yuanliang Dong, Zehua Cheng, Xinran Zhang, Xiaobing Li, Feng Yu, and Maosong Sun. 2022. Symphony generation with permutation invariant language model. arXiv preprint arXiv:2205.05448 (2022). [28] Peiling Lu, Xin Xu, Chenfei Kang, Botao Yu, Chengyi Xing, Xu Tan, and Jiang Bian. 2023. MuseCoco: Generating Symbolic Music from Text. arXiv:2306.00110 [cs.SD] https://arxiv.org/abs/2306.00110 [29] Haoran Luo, Haihong E, Guanting Chen, Qika Lin, Yikai Guo, Fangzhi Xu, Zemin Kuang, Meina Song, Xiaobao Wu, Yifan Zhu, and Luu Anh Tuan. 2025. Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning. arXiv:2507.21892 [cs.CL] https://arxiv.org/abs/2507.21892 [30] Haoran Luo, Haihong E, Guanting Chen, Yandan Zheng, Xiaobao Wu, Yikai Guo, Qika Lin, Yu Feng, Zemin Kuang, Meina Song, Yifan Zhu, and Luu Anh Tuan. 2025. HyperGraphRAG: Retrieval-Augmented Generation via Hypergraph-Structured Knowledge Representation. arXiv:2503.21322 [cs.AI] https://arxiv.org/abs/2503. 21322 [31] Haoran Luo, Haihong E, Yikai Guo, Qika Lin, Xiaobao Wu, Xinyu Mu, Wenhao Liu, Meina Song, Yifan Zhu, and Anh Tuan Luu. 2025. KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 41177–41199. https://proceedings.mlr.press/v267/luo25d.html [32] Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux. 2019. Cutting music source separation some Slakh: A dataset to study the impact of training data quality and quantity. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 45–49. [33] MIDI Manufacturers Association. 1996. MIDI 1.0 Detailed Specification. MIDI Manufacturers Association, Los Angeles, CA. [34] MuseScore BVBA. 2024. MuseScore: Create, play and print beautiful sheet music. https://musescore.org/ [35] OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/. [36] Alexander Pacha, Jan Hajič Jr, and Jorge Calvo-Zaragoza. 2018. A baseline for general music object detection with deep learning. Applied Sciences 8, 9 (2018), 1488. [37] Xingwei Qu, Yuelin Bai, et al. 2024. MuPT: A Generative Symbolic Music Pretrained Transformer. arXiv:2404.06393 [cs.SD] https://arxiv.org/abs/2404.06393 [38] Colin Raffel. 2016. Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching. Ph. D. Dissertation. Columbia University. [39] Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. 2023. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925 (2023). [40] Pedro Sarmento, Adarsh Kumar, CJ Carr, Zack Zukowski, Mathieu Barthet, and YiHsuan Yang. 2021. DadaGP: A dataset of tokenized GuitarPro songs for sequence models. arXiv preprint arXiv:2107.14653 (2021). [41] Elona Shatri and George Fazekas. 2024. Knowledge discovery in optical music recognition: Enhancing information retrieval with instance segmentation. arXiv preprint arXiv:2408.15002 (2024). [42] Bob L Sturm, Joao Felipe Santos, Oded Ben-Tal, and Iryna Korshunova. 2016. Music transcription modelling and composition using deep learning. arXiv preprint arXiv:1604.08723 (2016). [43] Gemini Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. (2024). arXiv:2403.05530 [cs.CL] https://arxiv.org/ abs/2403.05530 [44] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). [45] Chris Walshaw. 2011. The abc music standard 2.1. URL: http://abcnotation. com/wiki/abc: standard: v2 1 (2011).
Conference’17, July 2017, Washington, DC, USA
Menghe Ma* , Siqing Wei* , Yuecheng Xing* , Yaheng Wang, Fanhong Meng, Peijun Han, Luu Anh Tuan, and Haoran Luo†
[46] Yashan Wang, Shangda Wu, Jianhuai Hu, Xingjian Du, Yueqi Peng, Yongxin Huang, Shuai Fan, Xiaobing Li, Feng Yu, and Maosong Sun. 2025. Notagen: Advancing musicality in symbolic music generation with large language model training paradigms. arXiv preprint arXiv:2502.18008 (2025). [47] Andrew Wiggins and Youngmoo E Kim. 2019. Guitar Tablature Estimation with a Convolutional Neural Network.. In ISMIR. 284–291. [48] Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu. 2023. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData). IEEE, 2247–2256. [49] Jun Wu and Wanshan Guo. 2026. Autoregressive ConvNeXt-transformer fusion framework for polyphonic optical music recognition with focal loss optimization. Acoustical Science and Technology 47, 2 (2026), 86–96. [50] Shangda Wu, Xiaobing Li, Feng Yu, and Maosong Sun. 2023. TunesFormer: Forming Irish Tunes with Control Codes by Bar Patching. arXiv:2301.02884 [cs.SD] https://arxiv.org/abs/2301.02884 [51] Qingyang Xi, Rachel M Bittner, Johan Pauwels, Xuzhou Ye, and Juan Pablo Bello. 2018. GuitarSet: A Dataset for Guitar Transcription. Proceedings of the 19th International Society for Music Information . . . . [52] Yujia Yan and Zhiyao Duan. 2024. Measure by measure: Measure-based automatic music composition with modern staff notation. Transactions of the International Society for Music Information Retrieval (2024). [53] Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, et al. 2024. Chatmusician: Understanding and generating music intrinsically with llm. (2024), 6252–6271. [54] Shenghua Yuan, Xing Tang, Jiatao Chen, Tianming Xie, Jing Wang, and Bing Shi. 2025. Diffusion-based symbolic music generation with structured state space models. arXiv preprint arXiv:2507.20128 (2025). [55] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. 2024. Anygpt: Unified multimodal llm with discrete sequence modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 9637–9662.
ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence
Appendix A ONOTE Implementation Details The ONOTE benchmark is constructed over three music notation systems: standard staff notation, Jianpu, and guitar tablature, to support unified evaluation for omnimodal music score understanding. The dataset is built through a rigorous and controllable three-stage pipeline. After initial data collection, strict cross-modal alignment and format conversion are performed to ensure notation diversity and test validity. Based on the aligned omnimodal and cross-format data, task-specific test samples are constructed for the four core tasks of ONOTE. All samples are manually verified to guarantee musical correctness and structural validity. In total, the benchmark consists of 1,120 high-quality test samples.
A.1
Visual Score Understanding (VSU) Data
For the Visual Score Understanding task, we design one specialized visual question-answering (VQA) question for each music score image. All VQA questions are rigorously examined, revised, and validated by professional music annotators to ensure clarity, accuracy, and alignment with music-theoretic foundations. Each test sample in this task is composed of a single music score image in one of the three formats—standard staff notation, Jianpu, or guitar tablature—paired with its corresponding professionally designed VQA question. During evaluation, the model’s predicted answer is automatically compared with the authoritative ground-truth answer, and classification accuracy is computed to quantify performance. Through this process, we objectively assess the model’s ability of visual perception, symbol recognition, and high-level music-theoretic understanding directly from music score images without additional textual guidance.
A.2
Cross-Format Notation Conversion (CNC) Data
The Cross-Format Notation Conversion task focuses on assessing the model’s ability to translate music content across different representation systems while preserving musical semantics. For every input music score image in one notation format, we prepare a complete, semantically equivalent symbolic text in another target notation as the standardized ground truth. In particular, we provide standard staff notation texts as references for both guitar tablature and Jianpu inputs, and supply standard Jianpu texts for staff notation inputs to form comprehensive cross-format conversion pairs. When the model receives a source score image as input, it generates a symbolic sequence in the target notation, which is then automatically compared with the pre-prepared ground-truth text at the level of music logic, pitch consistency, rhythm accuracy, and structural completeness. This comparison allows us to quantitatively evaluate the model’s genuine music-theoretic reasoning and cross-format mapping capabilities.
A.3
Audio-to-Symbolic Transcription (AST) Data
In the Audio-to-Symbolic Transcription task, each music score image in the dataset is strictly aligned and synchronized with a corresponding high-quality music audio clip to ensure consistent
Conference’17, July 2017, Washington, DC, USA
musical content across modalities. In addition to aligned audio data, we also prepare complete, standardized symbolic groundtruth texts in the target notation format for each audio clip. During evaluation, the model takes a segmented music audio as input and predicts the corresponding symbolic music notation. The modelgenerated transcription is then quantitatively compared with the reference text from multiple dimensions, including pitch sequence, temporal position, note duration, and structural organization. By measuring the consistency between the predicted sequence and the ground truth, we reliably evaluate the model’s audio perception ability, temporal alignment precision, and the accuracy of end-toend audio-to-symbolic transcription.
A.4
Symbolic Music Generation & Aesthetics (SMG) Data
Different from the previous three discriminative and transductive tasks that rely on pre-collected score images or audio, the Symbolic Music Generation & Aesthetics task is a generative task that evaluates the model’s capacity for creative music expression. This task does not use existing score images as input; instead, it examines whether the model can independently produce coherent, standardized, and aesthetically reasonable symbolic music under given constraints. During evaluation, we focus on three core aspects: the overall quality of generated music, the rationality of music structure, and the continuity of musical aesthetics. By assessing formal correctness, structural logic, and artistic expression of the generated notation, we comprehensively evaluate the model’s higher-level music creation ability beyond understanding and transcription.
B Prompt used in ONOTE B.1 Task Execution Prompts In this section, we detail the exact execution prompts employed to evaluate the capabilities of Omni-models across the four primary tasks of the ONOTE benchmark. To ensure a fair evaluation, we adopted a standardized prompting paradigm. Each prompt is structured with a specific system Role (to activate the relevant domain expertise of the LLM) and a precise Task instruction. Furthermore, to systematically prevent model hallucination and ensure standardized output parsing during the automated evaluation, we applied a strict set of global formatting constraints (e.g., specific JSON schemas, sequence length limits, and anti-loop rules). To focus on the core semantic requirements of each task, these global formatting constraints are omitted from the individual task descriptions below. The core semantic prompts for the four primary evaluation tracks are illustrated in Figure 6. In evaluating the capabilities of Large Language Models (LLMs) across diverse music notation systems, unstandardized outputs often lead to the failure of automated parsing scripts. To ensure the fairness of the evaluation and the computability of the results, ONOTE introduces a strict set of Global Formatting Constraints. Independent of the task-specific prompts, these global constraints explicitly define the standard output specifications for three core music notation representations: ABC Notation, Numbered Musical
Conference’17, July 2017, Washington, DC, USA
Menghe Ma* , Siqing Wei* , Yuecheng Xing* , Yaheng Wang, Fanhong Meng, Peijun Han, Luu Anh Tuan, and Haoran Luo†
ONOTE Task Execution Prompts Task 1: Symbolic Music Generation Role: You are a top-tier composer. Task: Please compose an original classical-style melody for me. Key Signature: C Major. Time Signature: 4/4. Length: Strictly 8 measures. Task 2: Cross-Format Notation Conversion Role: You are taking a <notation>exam. Task: Please answer by outputting ONLY the single letter of the correct option (A, B, C, or D). Task 3: Visual Score Understanding Role: You are a highly precise Music OCR and Transcription AI. Task: Convert the provided sheet music image into Numbered Musical Notation (Jianpu). Task 4: Audio-to-Symbolic Transcription Role: You are an extremely precise AI music transcription expert. Task: Please transcribe the first 10 seconds of this audio. Transcribe the sequence exactly as you hear it. Do not repeat the same note indefinitely. If you are unsure, stop transcribing. Transcribe no more than 60 notes for these 10 seconds.
Figure 6: Core semantic execution prompts for the four primary tasks in the ONOTE benchmark Notation (Jianpu), and Structured Pitch-Duration Sequences. By enforcing these rules in the system instructions, we effectively mitigate formatting hallucinations and ensure that the models focus purely on the semantic music tasks. The detailed definitions of these formatting standards are presented in Figure 7.
The detailed critic prompts corresponding to the three primary notation formats (ABC Notation, Jianpu, and ASCII Guitar Tablature) are presented in Figure 8.
B.2
The ONOTE benchmark designs independent automated evaluation logics for four distinct tasks, all implemented through deterministic rules and sequence matching algorithms without subjective scoring or LLM-as-a-judge, ensuring reproducibility and impartiality. The evaluation process first performs structural parsing of model outputs, then executes task-specific verification logic, and finally provides quantitative results from dimensions including pitch accuracy, temporal consistency, music-theoretic compliance, and structural integrity.
Scoring Assessment Prompts
To enable a scalable, multi-dimensional, and nuanced evaluation of the generated music notations, ONOTE employs an LLM-as-a-Judge paradigm. While traditional string-matching metrics fall short in assessing the musicality and structural validity of complex scores, state-of-the-art LLMs can effectively serve as automated expert evaluators when guided by rigorous rubrics. In this subsection, we detail the Scoring Assessment Prompts designed for our evaluation pipeline. To ensure objectivity and consistency, each critic prompt is meticulously engineered with three core components: 1) Expert Persona: Assigning a strict and domain-specific role (e.g., a rigorous music theory professor or a fingerstyle guitar master) to calibrate the model’s evaluation standards. 2) Objective Verification Constraints: Mandating the mathematical verification of rhythmic values (e.g., exactly 4.0 beats per measure) and structural layouts before assessing subjective aesthetics. 3) Standardized Output Format: Forcing the model to output a parsable, continuous score (ranging from 1 to 5) without generating redundant conversational text.
C
ONOTE Evaluation Metrics Implementation Details C.1 Overall Evaluation Framework
C.2
Canonical Pitch Space Projection
Prior to task-specific evaluation, to achieve fair comparison across notation systems, all model-generated notation outputs are uniformly mapped to a one-dimensional temporally ordered canonical pitch sequence. The specific processing logic is as follows: for guitar tablature, the symbolic string-fret correspondences are accurately converted to standardized MIDI pitches by combining string position, fret information, and standard tuning rules; for Jianpu (numbered musical notation), the corresponding absolute scientific
ONOTE: Benchmarking Omnimodal Notation Processing for Expert-level Music Intelligence
Conference’17, July 2017, Washington, DC, USA
Global Constraints: Music Notation Formatting Standards System Instruction: When generating or transcribing music, you must strictly adhere to one of the following three designated notation formats as requested in your specific task. Do not mix formats. Format A: ABC Notation 1. Metadata: Must include standard headers (e.g., X:1, T:Title, M:4/4, K:C). 2. Syntax: Use standard ASCII characters for pitches, accidentals, and rhythms. 3. Structure: Group notes strictly according to the time signature. Use | to separate measures. Format B: Numbered Musical Notation (Jianpu) 1. Pitch: Use numbers 1-7 for scale degrees, and 0 for rests. 2. Octave Modifiers: Prefix with for one octave lower (e.g., 5) and ˆ for one octave higher (e.g., ˆ1). 3. Duration: Append duration fractions in parentheses immediately after the pitch. Example: 3(1/4). 4. Separation: Use a single space to separate individual notes and | for bar lines. Format C: Structured Pitch-Duration Sequence 1. Syntax Form: <Pitch><Octave>(<Duration>). Example: C4(1/4) or F#5(1/8). 2. Separation: Output must be a strictly space-separated sequence. 3. Constraint: Output ONLY the note sequence. Do not include measure lines, conversational text, or markdown wrappers.
Figure 7: The three standard music notation formatting constraints applied in ONOTE pitches are calculated based on key signature, scale degree markings, and octave indicators; for chord notes within the same time frame, they are sorted uniformly from lowest to highest pitch to eliminate evaluation biases caused by differences in note ordering across notation systems, laying a unified pitch representation foundation for subsequent accurate task-specific evaluation.
C.3
Task-Specific Evaluation Implementation Details
Visual Score Understanding (VSU) Evaluation: As a classification task, VSU adopts a ground-truth answer matching mechanism. The specific process involves parsing the model-generated answer, conducting dual-level matching (string-level and semantic-level) with manually annotated ground-truth answers, determining correctness based on exact consistency, and calculating the overall classification accuracy. This evaluation focuses solely on the correctness of visual recognition and music-theoretic understanding, excluding assessments of sequence generation quality or format standardization. Cross-Format Notation Conversion (CNC) Evaluation: As a sequence conversion task, CNC employs a combined evaluation approach of music-theoretic equivalence verification and sequence alignment. First, both the input notation and the modelgenerated output notation are projected to a unified canonical pitch sequence. Subsequently, the edit distance algorithm is used to calculate the matching accuracy of pitch sequences and rhythm sequences. Meanwhile, key signature, time signature, and overall structure are checked for consistency to ensure music-theoretic
equivalence. Finally, the grammatical legality of the target format is verified, and outputs with format errors or unparseable content are directly rejected. Audio-to-Symbolic Transcription (AST) Evaluation: As a temporal audio transcription task, AST adopts a dual-dimensional alignment evaluation logic (pitch and temporal alignment). First, the canonical pitch sequence, note onset times, and duration ground truths are extracted from the audio references. Then, the modelgenerated symbolic notation is converted into a sequence aligned with a uniform time axis, which is dynamically aligned with the ground-truth sequence. Pitch matching accuracy and temporal alignment accuracy are calculated separately. Outputs that are unparseable, contain illegal symbolic representations, or have severely mismatched lengths with the ground truth are directly deemed invalid. Symbolic Music Generation & Aesthetics (SMG) Evaluation: As a generative task, SMG adopts a multi-level rule-based verification evaluation scheme. First, renderability verification is performed to check the grammatical completeness, symbolic legality, and format standardization of the generated notation, ensuring it can be normally parsed by mainstream music score engines. Second, music-theoretic and structural verification is conducted to validate the consistency of key signatures, the rationality of measure division, and the legality of rest and note representations. Finally, structural coherence evaluation is implemented to measure the completeness of musical phrases, the self-consistency of rhythmic logic, and the rationality of musical structure based on preset sequence rules.
Conference’17, July 2017, Washington, DC, USA
Menghe Ma* , Siqing Wei* , Yuecheng Xing* , Yaheng Wang, Fanhong Meng, Peijun Han, Luu Anh Tuan, and Haoran Luo†
LLM-as-a-Judge: Evaluation Prompts for Automated Scoring Critic A: ABC Notation Evaluation Role: As an extremely strict music theory professor, review and score AI-generated text-based ABC notation. Evaluate the model’s proficiency by analyzing rhythmic values and musicality (Score 1-5). 1. Rhythmic Verification: A correct score contains exactly 4.0 beats per measure. The sum of rhythmic values between two | symbols must equal 4.0. (Score 5: All correct; Score 3: Half correct; Score 1: All incorrect). 2. Aesthetic Analysis: Motif development, melodic contour, and rhythmic groove should be rich and logical. (Score 5: Complete motif and beautiful melody; Score 3: Discernible motif; Score 1: Lacks motif, unappealing). Output Constraints: Output ONLY the final scores without analysis. Format: Technical Score: [ ]/5, Aesthetic Score: [ ]/5, Average Score: [ ]/5. Critic B: Numbered Musical Notation (Jianpu) Evaluation Role: As an extremely strict music theory professor, review and score AI-generated Jianpu based on rhythmic values and musicality (Score 1-5). Syntax Guide: | separates measures, spaces separate complete beats. A standalone number (e.g., 1, 0) or an extension dash (-) occupies 1 beat. Two numbers in parentheses (e.g., (3 4)) represent two eighth notes occupying 1 beat. 1. Rhythmic Verification: Exactly 4.0 beats per measure between | symbols. (Score 5: All correct; Score 3: Half correct; Score 1: All incorrect). 2. Aesthetic Analysis: Evaluate motif, melodic contour (rise/fall of notation numbers), and rhythmic groove. (Score 5: Rich and logical; Score 1: Does not constitute actual music). Output Constraints: Output ONLY the scores without analysis. Format: Technical Score: [ ]/5, Aesthetic Score: [ ]/5, Average Score: [ ]/5. Critic C: ASCII Guitar Tablature Evaluation Role: As an extremely strict fingerstyle guitar master, review and score AI-generated ASCII guitar tabs based on layout, fingering, and musicality (Score 1-5). 1. Layout and Timing: The 6 strings must align perfectly vertically. Assuming 4/4 time, characters on each string between two bar lines (|) must be identical, exactly 16 characters per measure. (Score 5: Layout correct, exactly 16 chars/measure; Score 3: Partial errors; Score 1: Mostly incorrect). 2. Musicality Analysis: Voicing should have distinct layers (bass root, middle harmony, high melody). (Score 5: Rich range/rhythm; Score 3: Monotonous; Score 1: Illogical). 3. Fingering Feasibility: Analyze simultaneous notes in the same column. A reasonable stretch is a maximum span of 7 frets. (Score 5: Logical, rarely ¿7 frets; Score 3: Frequent unreasonable stretches; Score 1: Physically impossible). Output Constraints: Output ONLY the scores without analysis. Format: Technical Layout Score: [ ]/5, Fingering Score: [ ]/5, Musicality Score: [ ]/5, Average Score: [ ]/5.
Figure 8: System prompts designed for the LLM-as-a-Judge automated evaluation, detailing strict scoring criteria for rhythmic verification, structural layout, and musical aesthetics across three notation formats