CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning

[2506.17818] CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning Skip to main content Search Submit Donate Log in Search arXiv Press Enter to search · Advanced search Computer Science > Sound arXiv:2506.17818 (cs) [Submitted on 21 Jun 2025] Title: CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning Authors: Angelos-Nikolaos Kanatas , Charilaos Papaioannou , Alexandros Potamianos View a PDF of the paper titled CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning, by Angelos-Nikolaos Kanatas and 2 other authors View PDF HTML (experimental) Abstract: Recent advances in music foundation models have improved audio representation learning, yet their effectiveness across diverse musical traditions remains limited. We introduce CultureMERT-95M, a multi-culturally adapted foundation model developed to enhance cross-cultural music representation learning and understanding. To achieve this, we propose a two-stage continual pre-training strategy that integrates learning rate re-warming and re-decaying, enabling stable adaptation even with limited computational resources. Training on a 650-hour multi-cultural data mix, comprising Greek, Turkish, and Indian music traditions, results in an average improvement of 4.9% in ROC-AUC and AP across diverse non-Western music auto-tagging tasks, surpassing prior state-of-the-art, with minimal forgetting on Western-centric benchmarks. We further investigate task arithmetic, an alternative approach to multi-cultural adaptation that merges single-culture adapted models in the weight space. Task arithmetic performs on par with our multi-culturally trained model on non-Western auto-tagging tasks and shows no regression on Western datasets. Cross-cultural evaluation reveals that single-culture models transfer with varying effectiveness across musical traditions, whereas the multi-culturally adapted model achieves the best overall performance. To support research on world music representation learning, we publicly release CultureMERT-95M and CultureMERT-TA-95M, fostering the development of more culturally aware music foundation models. Comments: 10 pages, 4 figures, accepted to the 26th International Society for Music Information Retrieval conference (ISMIR 2025), to be held in Daejeon, South Korea Subjects: Sound (cs.SD) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS) Cite as: arXiv:2506.17818 [cs.SD] (or arXiv:2506.17818v1 [cs.SD] for this version) https://doi.org/10.48550/arXiv.2506.17818 Focus to learn more arXiv-issued DOI via DataCite Journal reference: Proceedings of the 26th International Society for Music Information Retrieval Conference (ISMIR 2025), Daejeon, South Korea, pp. 569-578, 2025 Related DOI : https://doi.org/10.5281/zenodo.17706517 Focus to learn more DOI(s) linking to related resources Submission history From: Angelos-Nikolaos Kanatas [ view email ] [v1] Sat, 21 Jun 2025 21:16:39 UTC (4,269 KB) Full-text links: Access Paper: View a PDF of the paper titled CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning, by Angelos-Nikolaos Kanatas and 2 other authors View PDF HTML (experimental) TeX Source view license Current browse context: cs.SD < prev | next > new | recent | 2025-06 Change to browse by: cs cs.AI cs.LG eess eess.AS References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from

Related documents

• Composer2Vec: A Continuous Embedding Space of Composer Style Learned from Symbolic Melody Generation#1034867
• LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training#321861
• Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators#216832
• SongCraft: Unified Song Generation and Editing with Reconstructive Learning#1110680
• Probing Cultural Signals in Large Language Models through Author Profiling#924522