A P P L I C AT I O N
arXiv:2604.11560v1 [cs.LG] 13 Apr 2026
bacpipe: a Python package to make bioacoustic deep learning models accessible Vincent S. Kather1,2* Ghani1
|
|
Sylvain Haupert3
|
Burooj
Dan Stowell1,2
1 Naturalis Biodiversity Center, Leiden, The
Netherlands 2 Department of Intelligent Systems, Tilburg
University, Tilburg, The Netherlands 3 Muséum Nationale d’Histoire Naturelle,
Paris, France Correspondence Vincent S. Kather Email: [email protected] Funding information Supported by Marie Skłodowska-Curie Action Bioacoustic AI,grant agreement No 101116715
1. Natural sounds have been recorded for millions of hours over the previous decades using passive acoustic monitoring. Improvements in deep learning models have vastly accelerated the analysis of large portions of this data. While new models advance the state-of-the-art, accessing them using tools to harness their full potential is not always straightforward. 2. Here we present bacpipe, a collection of bioacoustic deep learning models and evaluation pipelines accessible through a graphical and programming interface, designed for both ecologists and computer scientists. 3. Bacpipe streamlines the usage of state-of-the-art models on custom audio datasets, generating acoustic feature vectors (embeddings) and classifier predictions. A modular design allows evaluation and benchmarking of models through interactive visualizations, clustering and probing. 4. We believe that access to new deep learning models is important. By designing bacpipe to target a wide audience, researchers will be enabled to answer new ecological and evolutionary questions in bioacoustics. KEYWORDS
bioacoustics, ecoacoustics, biodiversity monitoring, PAM, deep learning, embeddings, clustering
1
2
1
Kather et al.
|
INTRODUCTION
In bioacoustics, recordings of natural soundscapes span massive spatial and temporal scales [1] , the analysis of which has for a long time relied on automated processing for animal species detection [2] and soundscape characterization [3]1 . Bioacoustic recordings are evaluated to study evolutionary and ecological phenomena, i.e. animal behaviour [4], interaction with environments [5, 6], sound production [7] and more. Data is commonly gathered using passive acoustic monitoring (PAM) setups [8, 9, 10]. Studying changes of phenomena in large datasets was previously restricted to portions of datasets that were annotated manually by bioacousticians listening to recordings [10]. This precise study of recordings is a time-consuming practice, that therefore limits the amount of data that can be processed [8, 10]. In recent years, the rapid advance of deep learning in terrestrial, freshwater and marine bioacoustics has opened up the opportunity to develop tools to process and organize large acoustic datasets which has vastly improved automated species detection [11]. New bioacoustic deep learning models (see Glossary) are trained and released at increasing rates [11, 12, 13]. At the same time, means to compare and use models are often limited by how well their respective code is maintained. Computing acoustic indices and unsupervised classification of soundscapes have been made accessible through numerous Python software packages, e.g. for acoustic indices, scikit-maad [14] and for unsupervised classification, bambird [15], open soundscape [16], soundscape explorer [17] and pykanto [18]. This accessibility is lacking for bioacoustic deep learning models making it hard for users to switch and compare between models. Recent reviews [13, 12, 19] give some guidance about model capabilities. However, reproducing these results and comparing between models remains cumbersome. For example, BirdNET, a model trained on a large corpus of bird vocalizations, is available with a graphical user interface (GUI) tool, making it accessible which explains its widespread use [20, 21]. As the field of computational bioacoustics is an interdisciplinary field with researchers coming from varying degrees of ecological/computational backgrounds, the accessibility of newly developed methods to researchers with different skill sets, directly impacts if these methods get included in ecological workflows or not. Deep learning models compute rich, high-dimensional embeddings for any given input. Embeddings can then be used for classification tasks, e.g. providing predictions on species presence. Recent studies show that while models are trained to classify specific species, using these models to organize large acoustic datasets based on the extracted embeddings offers up a plethora of opportunities for other bioacoustic applications [22, 23, 24, 13]. Furthermore, using deep learning embeddings rather than model predictions provides a continuous space where all data is represented rather than relying on model predictions to filter datasets. Models are being trained on broader datasets, going from bird-focused training sets like BIRB [25] and the annotated bird recordings on the citizen science platform xenocanto [26] to incorporate more species groups like amphibians [27], mammals [28] and insects [29]. Models trained on general audio are also being trained for bioacoustic datasets [12, 13]. Yet, at this point, clear tendencies on what models to use for an acoustic environment are inconclusive and require researchers to have the tools to test and compare models in order to identify the best method for their research needs. Here, we present bacpipe (bioacoustic collection pipeline), a Python software package to facilitate the use of state-of-the-art bioacoustic deep learning models. By centering bacpipe on extracting acoustic representations using deep learning models (generating embeddings), large acoustic datasets can be structured and investigated without limiting the models to their classifier predictions. Thereby generating embeddings for a variety of models can be incorporated into existing workflows and models can be compared. As a stand-alone software, bacpipe provides interactive visualizations of acoustic representations (embeddings) from different models in a dashboard GUI enabling users to explore large datasets and benefit from the structuring capabilities of state-of-the-art deep learning models. 1 https://github.com/bioacoustic-ai/bioacoustics-datasets
Kather et al.
3
As an add-on, the package comes with a variety of evaluation tools enabling researchers to probe model performance on their own data. Bacpipe is built in a modular design ensuring that with further development, new deep learning models can be easily included. In the following we will describe how bacpipe enables researchers to
1.
process large PAM datasets with a variety of state-of-the-art deep learning models,
2.
interactively explore acoustic representations visually and aurally,
3.
extract classifier predictions from a variety of models and
4.
use bacpipe to evaluate models on alternative bioacoustic tasks.
2
|
BASIC WORKFLOW
Bacpipe can be used via its API or as a stand-alone software. The user can therefore decide which of bacpipe’s processing steps to run. Figure 1 shows a schematic overview of these processing steps. In the following section the package concept and workflow will be explained in more detail.
Glossary Key vocabulary used in this manuscript with their specific meaning in the field of computational bioacoustics. Deep learning model: Deep neural network machine learning model based on (for example) convolutional neural network (CNN) or transformer architecture and in this manuscript trained using acoustic recordings. Deep learning models usually consist of feature extractors and classifiers, the former of which is used to generate embeddings. In this manuscript deep learning model refers to the feature extraction part of the model. Embeddings: High-dimensional feature vectors created by deep learning models based on a section of audio. Embedding space: High-dimensional vector space containing embeddings of a model. Dimensionality reduction tools can be used to visualize these spaces. Classification: Class predictions are generated by the deep learning model by first computing an embedding and subsequently mapping that embedding onto a list of classes. Benchmarking: Evaluation of deep learning models including their pretrained classifiers. Clustering: Unsupervised algorithm that organizes the embeddings into clusters, used here to evaluate deep learning models. Probing: Supervised algorithm used to evaluate deep learning models by fitting a classifier on top of the model. Linear probing: Training a linear classifier consisting of a single fully connected layer. kNN probing: Fitting a (parameter-free) k nearest neighbours classifier. Using a fraction of the data to fit the classifier and using it to classify the remainder.
4
Kather et al.
F I G U R E 1 Processing workflow of bacpipe. From left to right, audio data (Data) is passed into deep neural networks (Models), which generate acoustic feature vectors (Embeddings). Embeddings are then used for a number of tasks which are applied to all selected models for comparability (Evaluation). Evaluation outputs are saved to a standardized folder structure (see Fig. 3). By default, evaluations include classification (outputs produce lists of predicted species), dimensionality reduction, (used for visualizations) and clustering (used to measure shared mutual information), but can be expanded for other tasks.
2.1
|
Design philosophy
Bacpipe is designed to target two audiences: 1. experienced listeners with expertise in ecology and bioacoustics and perhaps little experience with Python and 2. experienced deep learning researchers working in bioacoustics with Python experience. To address both audiences, bacpipe is available as a fully functional and well documented stand-alone GitHub repository2 and as a pip (PiPy) package. Once started, bacpipe processes files based on userdefined configurations and visualizes results in an interactive GUI dashboard (see Section 3). Alternatively, bacpipe’s embedding generation or evaluation modules can be used via its API and integrated into existing workflows (see Section 4). Loading and processing audio has been developed to work both on the CPUs of laptop computers and GPUs of computers or servers. In terms of processing, bacpipe is centred around the computation of embeddings with a large variety of bioacoustic models, as can be seen for sections the Data, Models and Embeddings in Figure 1. Add-on features to evaluate embedding spaces, like linear probing, clustering and visualization are included in the package (see Section 2.5). Furthermore, benchmarking of multi-label classifier predictions is possible. Accessibility and modularity are the core design principles of bacpipe to enable all researchers and practitioners to access the stateof-the-art deep learning models for their bioacoustic dataset. The models included in the package can be seen in Table 1.
2 https://github.com/bioacoustic-ai/bacpipe
Kather et al.
2.2
|
5
Simple starting-point
To run bacpipe, the user has to specify the source path to the audio files and which models to run. All further configurations are optional. These configurations are specified in the bacpipe/config.yaml file or, when using the API, using bacpipe.config . More fine-grained settings (like to run the computation on a GPU rather than the default CPU) can be modified in the bacpipe/settings.yaml file (or the attribute for API). Once bacpipe gets started, all data is processed and a dashboard visualization is started in a browser window with a GUI, allowing the user to interact and explore the results of their processing (see Section 3). For each execution, configurations are saved with a corresponding timestamp, ensuring that users can go back and reproduce previous experiments. This interaction and both visual and auditory exploration of the processed data makes the analysis accessible for all audiences. Figure 2 shows an example case study using a large unlabeled dataset where only the dataset path and model name were specified. Structurally, the package is made up of three subpackages: core, model_pipelines and embedding_evaluation. The subpackages handle audio loading and preparing inference (core), loading and executing the respective models (model_pipelines) and evaluating the generated embeddings through linear probing, classification and visualization (embedding_evaluation). Upon execution, core checks to see if the combination of input data and selected deep learning models already exist. If this is the case, embedding_evaluation is used to load and visualize the data within seconds rather than requiring to be computed again. This also allows continuously growing datasets or prematurely failed processing runs to be continued where left off, which is practical when processing very large PAM datasets. Throughout the processing, a (human-readable) metadata file (metadata.yml) is created with detailed information within the corresponding embeddings folder. All of these convenience features can be included or excluded in the API (see Section 4).
2.3
|
Reducing unnecessary recomputation
Bacpipe can be used on acoustic datasets of any folder structure or commonly supported audio file format. When processing, bacpipe creates a standardized folder structure where embeddings, reduced dimension embeddings, classifier predictions, evaluations and metadata are saved. The top level folder name corresponds to the name of the processed audio dataset. The output folder structure can be seen in Figure 3A. Inside it, the embeddings file structure mirrors the audio file structure, meaning that for every audio file there is a corresponding embedding file, this makes association easier for downstream analysis purposes. The same is true for classifier predictions. The evaluation folder contains classifier predictions, clustering results, generated labels and more, which are used during for the dashboard GUI and which can also be returned via the API (see Section 4). Saving these processed analyses vastly reduces the computational time for future processing. Once processing is complete, bacpipe no longer requires access to the audio files, as returning embeddings, visualizing or evaluation is done using the output files. By using a standardized output folder structure, results can be transferred between devices enabling one researcher to visualize the results that were processed by another (or to process on a GPU cluster and analyse on a laptop). High-dimensional embeddings are saved as Numpy array files (ending with .npy) to reduce storage requirements, while reduced low-dimensional embedding (by default, 2d-UMAP [30]) files are saved as human-readable dictionary files (ending with .json). By default, classification predictions are saved as annotation tables (one per audio file) in the Raven [31] format as well as a combined annotations file for all classifier predictions in the dataset.
6
Kather et al.
F I G U R E 2 A large unlabeled dataset from a noisy urban soundscape in the Netherlands is visualized in different ways. A: Bacpipe dashboard view of a 2d UMAP visualization of BirdNET embeddings from the entire dataset. The embeddings (≈ 105 ) were processed from 104 recordings (≈ 100 hrs of audio). Little boxes indicate what purpose different sections of the dashboard serve. The points are coloured by generated time-of-day timestamps. While the view of the entire dataset (A) shows some structure, the grey rectangle corresponds to the zoomed-in view (B). In C a spectrogram is shown visualizing the audio data corresponding to a selected embedding point in B. Using the inbuilt functions, a handful of unknown bird vocalizations could be found and isolated, listened to and exported as a .csv file to isolate them from the otherwise anthropogenic noise dominated soundscape. Without any ground truth, these processing and interactive capabilities make evaluating large noise-dominated datasets more feasible.
Kather et al.
7
TA B L E 1 List of models currently available in bacpipe. The column headings correspond to the following: clfier lists if a classifier is included for the model, training shows the training setup, i.e. ssl for self-supervised learning, supl for supervised learning, sup cl for supervised contrastive learning and ft for fine-tuning. architecture lists the model backbone and dim shows the dimension of the embeddings. trained on shows the model’s training data, sr lists the sample rate in kHz, size shows the model size in millions of parameters and ref. lists the respective publication. name
clfier
training
architecture
dim
trained on
sr [kHz]
size [106 ]
ref.
AudioMAE
no
ssl + ft
ViT
768
general
16.0
86.0
[32]
AudioProtoPNet
yes
supl
ConvNeXt
768
birds
32.0
98.0
[33]
AvesEcho
yes
supl
PaSST
1024
birds
32.0
86.3
[22]
AVES
no
ssl + ft
HuBERT
768
general, biotic
16.0
94.2
[34]
BEATs
no
ssl + ft
HuBERT
768
general
16.0
91.0
[35]
BioLingual
no
supl
CLAP
512
animals, birds
48.0
190.0
[36]
BirdAVES
no
ssl + ft
HuBERT
1024
general, birds
16.0
316.0
[34]
BirdMAE
no
ssl + ft
ViT
768
general
32.0
300.0
[37]
BirdNET
yes
supl
EffNetB0
1024
birds
48.0
12.8
[20]
ConvNeXt_bs
yes
supl
ConvNeXt
768
birds
32.0
88.0
[12]
Google_Whale
yes
supl
EffNetB0
1280
whales
24.0
5.0
-
HBdet
yes
supl
ResNet50
1024
hback. whales
2.0
23.0
[38]
Insect459NET
no
supl
EffNetv2s
1280
insects
44.1
20.6
-
Insect66NET
no
supl
EffNetv2s
1280
insects
44.1
20.5
-
Mix2
no
supl
MobileNetv3
960
amphibians
16.0
3.0
[39]
NatureBEATs
no
ssl + ft
HuBERT
768
all
16.0
90.7
[40]
Perch_Bird
yes
supl
EffNetB1
1280
birds
32.0
7.8
[41]
Perch_2
yes
supl
EffNetB3
1536
animals
32.0
12.0
[42]
ProtoCLR
no
sup cl
CvT-13
384
birds
16.0
19.6
[43]
RCL_FS_BSED
no
sup cl
ResNet9
2048
animals
22.0
7.2
[44]
SurfPerch
yes
supl
EffNetB0
1280
corals, birds
32.0
8.0
[45]
VGGish
no
supl
VGG
128
general
16.0
62.0
[46]
2.4
|
Labels are inferred
Bioacoustic datasets often feature a nested folder structure based on deployment sites and date. Using this information along with the file-specific timestamps can be helpful to analyse species activity and embedding structure in respect to spatial and temporal patterns. To make use of this, bacpipe creates default labels based on this information. At the point of writing the default labels that are automatically generated, are "time_of_day", "day_of_year", "continuous_timestamp", "parent_directory" and "audio_file_name". As different models use different input lengths of audio, the default labels get mapped onto the model-specific timestamps, thereby associating each embedding with a default label. If ground truth annotations are provided, the labels are read and also mapped onto the timestamps of models. This can be very useful for benchmarking, evaluating clustering, visualizing or linear probing.
2.5
|
Benchmarking, clustering and probing
Traditionally, deep learning models are evaluated using benchmarking, in which classifier predictions are generated for a benchmarking dataset. These benchmarking datasets must not be included in the training data. Ideally, bioacoustic
8
Kather et al.
F I G U R E 3 Output folder structure (A) and graphical user interface (B) of bacpipe. In A, the standardized folder structure is shown. This is how the processed files are saved, showing that the saved embeddings mirror the input structure from Figure 1. Furthermore, evaluations, dimensionality reduced embeddings and processing logs are saved. In B, the graphical interface that is produced by default is shown. The interface enables evaluating the results in an accessible way.
benchmarking datasets feature data from different environments, so that the generalization capabilities of the models can be tested. The resulting predictions are evaluated, for example using mean average precision (mAP) and compared against the state-of-the-art to showcase improvements. Common bioacoustic benchmarking datasets used for this, are BEANS [47] for single label or BIRB for [25] multi-label classifier performance. While bacpipe does not include any datasets, it includes benchmarking capabilities. Users have to specify the dataset name corresponding to the top level audio folder (see Section 2.3) and the model they wish to evaluate. The (multi-label) ground truth labels are mapped to the model-specific timestamps and compared to the generated predictions using mAP. To quantify a deep learning model’s ability to extract acoustic features, recent research has moved beyond the traditional benchmarking of classifier performance. Instead, to evaluate embedding spaces, new classification heads are trained and evaluated on top of pretrained models - this process is called probing (see glossary) [48]. In the following probing refers to the process of training kNN or linear probes on top of feature extractors, while classification refers to the use of the integrated classification head which is part of a published deep learning model. Recent studies have applied this practice to bioacoustics, enabling them to compare models that were trained on different species [12, 13, 42, 49]. Bacpipe includes both training linear and kNN probes on top of processed embeddings. A linear probe is a trainable single fully connected layer which can be useful to test the global structure of the embeddings by trying to separate classes linearly. A kNN probe on the other hand is a parameter-free classifier based on (euclidean) distance, which emphasizes local structure by testing if nearest neighbour associations are useful to separate classes. These evaluation tasks do not require access to the models, as they can be processed on the precomputed embeddings. Another evaluation strategy supported by bacpipe is clustering. Past studies have shown that KMeans clustering provides a fast and scalable way to evaluate bioacoustic embedding spaces [49]. If the clustering evaluation task is enabled, by default a KMeans clustering of the embeddings (in their original high dimension) is computed. The clus-
Kather et al.
9
tering is evaluated using adjusted mutual information (AMI) [50] and adjusted rand index (ARI) [51] by comparing it to the automatically generated default labels. Clustering can be computed without ground truth annotations. However, if ground truth annotations are provided, the clustering evaluations are also compared to it, and are computed twice: once for the entire dataset, and once for only the annotated portion of the dataset. This way embeddings corresponding to vocalizations of specific species can be evaluated disregarding periods of "noise". If the probing evaluation task is enabled, bacpipe searches for a ground truth annotations file (this file requires predefined column names: "audiofilename", "start", "end", "label:species" (species can be replaced) and its location needs to be specified in the settings). The user has to simply specify which evaluation tasks to run in the configuration file, and subsequently these evaluations can be immediately computed for each of the selected models. Parameters for these evaluations have default values and so can be computed immediately, but can also be customized by referring to the settings file. To train linear and kNN probes on the embeddings, the annotations are split into train, validation and test. Results are evaluated per label class and as a global average using mean average precision (see Fig 4).
3
GRAPHICAL USER INTERFACE
|
Bacpipe generates a browser-hosted graphical user interface based on the Python package panel 3 . A schematic of this is shown in Figure 3B. The interface shows different tabs, which visualize the results in two strategies, 1. display modelspecific embedding spaces and 2. display model-specific heatmaps of classifier predictions. For each visualization there are tabs showing the performance for 1. a single model and 2. a side-by-side comparison of two models. For the embedding visualizations there is an additional overview of all models. In the following the visualization strategies are explained in detail. Figures 2 and 4 show two case studies using the graphical user interface.
3.1
|
Explore visually and aurally
While deep learning models enable the processing of vast amounts of audio data, the interpretability of their output can be limited. Classification performance of models is steadily increasing, yet, many species of interest are not well represented in datasets (or not represented at all), leading to classifier predictions being irrelevant for a given research project. Beyond classifier predictions, it is also possible to visualize embeddings produced by deep learning models using dimensionality reduction techniques (see Section 4.1). The vast amounts of embeddings show emerging structures and clusters that, when overlaid with labels reveal patterns how data is organized by models (for more information on labels see Section 2.3). This way, visualizations of embeddings can be colour-coded with automatically extracted default labels like timestamps, as well as computed clusters or (if provided) ground truth labels. Side-byside comparisons of models make differences apparent and enable users to decide based on their study species or noise environment which model is best suited. For ecoacoustics this comparison can be helpful to investigate how differences between soundscapes relate to differences in space, time or habitat characteristics. Low dimensional (2 or 3-dimensional) UMAP (or t-SNE) embeddings have the benefit that they can be visualized. However, without the ability to link the embeddings back to the underlying audio source, evaluations are limited to qualitative differences in appearance [30]. We therefore provide interactive visualizations based on the Python package plotly [52], which displays a spectrogram of the audio when clicking on an embedding point. The spectrograms audio can also be played back. This connects the high level deep learning embeddings back to its source and enables an intuitive exploration of audio data whilst benefiting from the structuring and processing capabilities of deep learning. 3 https://panel.holoviz.org/
10
Kather et al.
As the spectrograms are generated in real-time, access to the original audio data is required. The visualized audio is resampled to the model-specific sample rate, which provides visual feedback on what portion of the spectrogram the embedding corresponds to. Figure 2 shows a case study in which embeddings and spectrogram visualizations were used to isolate bird vocalizations in a large unlabeled dataset. By using plotly, users can zoom and pan in the embedding space. For PAM datasets spanning years of data, this is essential, as the large number of embeddings contain complex structures and substructures that only become apparent when narrowing in on small sections. A selection tool can be used to choose a number of points in the embedding space, which can then be exported as annotations to a csv file. This can be useful for the task of annotating or investigating segments with similar acoustic characteristics.
3.2
|
Visualize clustering and probing metrics
Aside from embeddings, the clustering and probing performance metrics are displayed (if enabled in config file). The clustering results show AMI and ARI values, quantifying the overlap of the calculated clusters with the specified label. The overlap between the clusters and the automatically generated labels is an indication if the model is structuring the data according to diurnal, seasonal or habitat-based patterns. If ground truth data is provided, the clustering is also computed between the default labels and the ground truth. This can be insightful to quantify the overlap between the ground truth labels and automatically generated labels to show that the presence of classes coincides with diurnal, seasonal or habitat-based patterns. Figure 4 A shows a side-by-side comparison of two models, in which points are labeled by the parent directory. If the parent directory corresponds to the location site, the overlaps between the site and the ground truth can be quantified, which can indicate that species only vocalize in specific places and that the model’s embeddings are organized similarly. If probing is enabled in the configurations, kNN and linear probes are trained on the provided ground truth annotations (as described in Section 2.5). The results of the probing are shown as bar plots. Users can switch between kNN and linear probes to see the respective results. For each label class, the accuracy values are displayed. Using the probing evaluation, users can see how well a model’s embeddings can be classified into the annotated species using both kNN and linear probing. Figure 4 B shows the class-wise performance of the AvesEcho_Passt model using linear probing on the AnuraSet dataset. For both the clustering and the classification, overall performance is displayed in the all_models tab.
3.3
|
Classification and probing heatmaps
While generating embeddings, by default bacpipe also generates classifier predictions using the model provided classifier. These predictions are saved as Raven style annotations (see Section 2.3). The generated predictions are displayed in the dashboard of bacpipe as an activity heatmap. The heatmap is generated to show the dates on the y- and the hours on the x-axis and colour-coded by the number of occurrences. Users can choose between species from a drop-down menu and thereby visualize the activity for each species found within the dataset. As discussed in Section 2.5, probing can be applied to classify the embeddings. In this case the previously trained linear probe is loaded and used to classify the precomputed embeddings. This way a classifier for a new task (e.g. different species group, individual identification) is trained and can be used to generate predictions. For models that do not provide classification layers (see Table 1), this provides an opportunity to generate classifier predictions. The results of this classification can also be displayed in the same way as the pretrained classifiers. Figure 4 C shows a species presence heatmap which was generated using a linear probe trained on the model AvesEcho_Passt using frog
Kather et al.
11
F I G U R E 4 Case study of a large annotated dataset using AnuraSet which consists of recordings from the Amazonian rainforest [27]. In A a bacpipe dashboard is shown with a side-by-side comparison of 2d UMAP visualizations of embeddings generated using two different models: AvesEcho_Passt [23] and NatureBEATs [40]. Little boxes indicate what purpose different sections of the dashboard serve. The points are coloured by the parent directory of the audio file they were processed from, which corresponds to the recording site. The embedding plot for NatureBEATs shows fewer points, as only the annotated portion of the dataset was chosen to be displayed. For B the model was retrained using the probing procedure (see Section 2.5) on species which were annotated in the AnuraSet dataset. With the linear probe, the model is now able to classify frog species and the results of the class-wise performance are shown in a bar plot. In C an automatically generated presence heatmap is shown of the frog species Sphaenorhynchus surdus using the model AvesEcho_Passt with the trained linear probe. These heatmaps can be generated for all species present in the annotations. This workflow demonstrates that bacpipe can be used to retrain models on classes they were not originally trained on and visualize species presence immediately.
12
Kather et al.
vocalizations annotated in the AnuraSet dataset [27].
4
DEVELOPER SECTION
|
For users that look to integrate bacpipe into their existing bioacoustic pipelines, an API is provided. This API features the same modules and functions as described in Section 2. In the following section, the API will be described in more detail. For more information see the documentation 4 .
4.1
|
Extending bacpipe
The aim of bacpipe is to streamline the processing of audio recordings using interchangeable bioacoustic deep learning models. The models included in the package can be seen in Table 1. Once the brief installation of bacpipe is successful, the user will be able to process their audio data using a plethora of different deep learning models, among them the most used and state-of-the-art models. Comparing existing models against newly developed ones is crucial, which is why the modular design of the package allows users to easily add new models or add modified versions of existing ones. Any new model will go through the same evaluation pipeline, this way the results of new/modified models can be explored and compared. To visualize the model outputs, dimensionality reduction techniques (e.g. UMAP [30], t-SNE [53], PCA [54]) are included in bacpipe. Generated visualizations can be used to compare how different models structure the data and are interactive, allowing the user to connect the embeddings back to the audio segment they were processed on. Just like bioacoustic deep learning models, they too can be added or modified for comparisons. Finally, the evaluation tool-set are functions applied to all specified models and this tool-set can also be changed or expanded as the field advances. The section Evaluation in Figure 3 shows some of these downstream processing steps.
4.2
|
Low to high level pipelines
Bacpipe comes with a high level prepackaged fully operational pipeline, accessible through bacpipe.play(). Based on the attributes of bacpipe.config and bacpipe.settings all audio files in a given directory will be processed with all selected models. Subsequent evaluation will be performed and visualization of the results is facilitated in the dashboard GUI (explained in Section 3). However, to integrate smoothly into existing workflows, numerous pipelines are provided to work with varying levels of underlying processing. This way users can run a pipeline that will process audio folders for 1. all specified models, 2. a single specified model or 3. a single audio file for a given model. A selection of API functions along with descriptions are provided in the Appendix (see Table 2). The lower level workflow of bacpipe divides the main tasks of path-handling, audio-handling, model-handling and classifier-handling into the four classes, Loader, AudioHandler, Embedder and Classifier. To initialize the processing, bacpipe uses the Loader class, which organizes the loading of audio files and embedding files (if already processed). The loading of audio files itself, along with resampling to model-specific sampling rates, padding audio and segmenting it into batches is done by the AudioHandler class. The batched file segments then get passed on to the Embedder class, which handles loading the model and generating embeddings. Finally the Classifier class receives embeddings and generates class predictions. As the Loader class handles all paths, the object generated from it, can be used to load all embeddings, predictions and metadata after processing. 4 https://bacpipe.readthedocs.io/en/latest
Kather et al.
13
A large number of functions are provided to facilitate the integration into the different bacpipe pipelines. These functions can be used to extract date-time information from audio files, generate arrays of ground truth labels fitted to the model-specific timestamps, list all processable audio files in directories and more. Higher level functions allow users to generate clusterings, train and evaluate probes and visualize their results. Finally, a benchmarking function allows users to evaluate single or multi-label model predictions in respect to a provided annotation file.
5
|
PERSPECTIVES
The field of computational bioacoustics is experiencing an accelerated development, fuelled by advancements in deep learning and interest in questions of ecology and evolution. The previous decades have seen the creation of massive corpora of recordings being collected by researchers, conservationists and citizen scientists all over the world. However, evaluating datasets of very large proportions without computational tools is extremely time-consuming. In practice, for this reason, evaluations are often restricted to small portions of the datasets. Here, we presented bacpipe, a collection of bioacoustic pipelines consisting of state-of-the-art deep learning models and evaluation techniques, made accessible through a graphical and a programming interface. Bacpipe is inspired by and builds on the variety of tools, which have been developed and published for the processing of bioacoustic data (mentioned in Section 1). However, it is the first tool to include all the state-of-the-art models in bioacoustics and make them accessible to both computer scientists and ecologists. Its modular setup is designed to integrate with existing tools and workflows to ensure researchers can compare models for their specific hypotheses. With the accelerated pace of deep learning developments and models being released, accessibility of these methods will be increasingly important. At the same time archives of acoustic datasets all over the world, span millions of hours and contain valuable ecological and evolutionary information. By harnessing the full potential of deep learning models, this information can be organized. If combined with interactive tools, these datasets can be explored and isolated events and vocalizations can be discovered. While recent review papers show, that scores are improving for bird song classification, noisy and polyphonic soundscape recordings still lead to poor classifier performance [13, 12, 42]. We argue that the usage of deep learning models in bioacoustics should not be limited to classifier predictions. Instead, using deep learning models as acoustic feature extractors to organize and reveal structure in large PAM datasets, empowers users to explore and identify sounds of interest through interactivity. Figure 2 shows a case study of this workflow. Computational bioacoustics is an interdisciplinary field, that requires interdisciplinary workflows to best combine people’s expertise from their respective fields. Like many before it, the aim of bacpipe is to be a collaborative tool, shaped by and for the community. Contributions are very welcome, especially newly developed models by deep learning practitioners and evaluation workflows by ecologists. We refer to the github repository for more detailed descriptions 5 . We hope that this introduction and description will invite researchers to collaborate and design systems accessible to all audiences.
6
|
AUTHOR CONTRIBUTIONS
Conceptualization and Software: VK. Methodology: VK, SH, BG, DS. Supervision: BG, SH, DS. Writing—original draft: VK. Writing - Review & Editing: SH, BG, DS. All authors contributed critically to the drafts and gave final approval for publication. 5 https://github.com/bioacoustic-ai/bacpipe
14
7
Kather et al.
|
ACKNOWLEDGMENTS
The authors would like to thank all people that have contributed issues, pull requests, feedback and especially the creators of the deep learning models. The authors would like to also thank Nicole Allison for the design of the bacpipe logo.
8
|
CONFLICT OF INTEREST STATEMENT
The authors declare no conflict of interest.
9
|
DATA AVAILABILITY STATEMENT
Bacpipe is developed openly in GitHub at https://github.com/bioacoustic-ai/bacpipe, available under Apache 2.0 open-source license. Documentation is available at https://bacpipe.readthedocs.io. Bacpipe is installable through PyPi (https://pypi.org/project/bacpipe/) and has been tested on Linux, Mac and Windows.
references [1] Carrie C. Wall, Samara M. Haver, Leila T. Hatch, Jennifer Miksis-Olds, Rob Bochenek, Robert P. Dziak, and Jason Gedamke. The Next Wave of Passive Acoustic Data Management: How Centralized Access Can Enhance Science. Frontiers in Marine Science, 8, July 2021. ISSN 2296-7745. doi:10.3389/fmars.2021.703682. [2] Mark F. Baumgartner and Sarah E. Mussoline. A generalized baleen whale call detection and classification system. The Journal of the Acoustical Society of America, 129(5):2889–2902, May 2011. ISSN 0001-4966. doi:10.1121/1.3562166. [3] Jérôme Sueur, Almo Farina, Amandine Gasc, Nadia Pieretti, and Sandrine Pavoine. Acoustic indices for biodiversity assessment and landscape investigation. Acta Acustica united with Acustica, 100(4):772–781, 2014. [4] Daniella Teixeira, Martine Maron, and Berndt J. van Rensburg. Bioacoustic monitoring of animal vocal behavior for conservation. Conservation Science and Practice, 1(8):e72, 2019. ISSN 2578-4854. doi:10.1111/csp2.72. [5] Esther Sebastián-González and Cristian Pérez-Granados. Geographic Variation in Acoustic Signals in Wildlife: A Systematic Review. Journal of Biogeography, 52(6):e15116, 2025. ISSN 1365-2699. doi:10.1111/jbi.15116. [6] Camille Desjonquères, Simon Linke, Jack Greenhalgh, Fanny Rybak, and Jérôme Sueur. The potential of acoustic monitoring of aquatic insects for freshwater assessment. Philosophical Transactions of the Royal Society B: Biological Sciences, 379(1904):20230109, May 2024. doi:10.1098/rstb.2023.0109. [7] Robert M. Seyfarth and Dorothy L. Cheney. Production, usage, and comprehension in animal vocalizations. Brain and Language, 115(1):92–100, October 2010. ISSN 0093-934X. doi:10.1016/j.bandl.2009.10.003. [8] David K. Mellinger, Kathleen M. Stafford, Sue E. Moore, Robert P. Dziak, and Haru Matsumoto. An overview of fixed passive acoustic observation methods for cetaceans. Oceanography, 20(4):36–45, 2007. [9] Thomas A. Calupca, Kurt M. Fristrup, and Christopher W. Clark. A compact digital recording system for autonomous bioacoustic monitoring. The Journal of the Acoustical Society of America, 108(5_Supplement):2582, November 2000. ISSN 0001-4966. doi:10.1121/1.4743595. [10] Larissa Sayuri Moreira Sugai, Thiago Sanna Freire Silva, José Wagner Ribeiro, Jr, and Diego Llusia. Terrestrial Passive Acoustic Monitoring: Review and Perspectives. BioScience, 69(1):15–25, January 2019. ISSN 0006-3568. doi:10.1093/biosci/biy147.
Kather et al.
15
[11] Dan Stowell. Computational bioacoustics with deep learning: A review and roadmap. PeerJ, 10:e13152, March 2022. ISSN 2167-8359. doi:10.7717/peerj.13152. [12] Raphael Schwinger, Paria Vali Zadeh, Lukas Rauch, Mats Kurz, Tom Hauschild, Sam Lapp, and Sven Tomforde. Foundation Models for Bioacoustics – a Comparative Review, August 2025. [13] Marius Miron, David Robinson, Milad Alizadeh, Ellen Gilsenan-McMahon, Gagan Narula, Emmanuel Chemla, Maddie Cusimano, Felix Effenberger, Masato Hagiwara, Benjamin Hoffman, Sara Keen, Diane Kim, Jane Lawton, Jen-Yu Liu, Aza Raskin, Olivier Pietquin, and Matthieu Geist. What Matters for Bioacoustic Encoding, August 2025. [14] Juan Sebastián Ulloa, Sylvain Haupert, Juan Felipe Latorre, Thierry Aubin, and Jérôme Sueur. Scikit-maad: An opensource and modular toolbox for quantitative soundscape analysis in Python. Methods in Ecology and Evolution, 12(12): 2334–2340, 2021. ISSN 2041-210X. doi:10.1111/2041-210X.13711. [15] Félix Michaud, Jérôme Sueur, Maxime Le Cesne, and Sylvain Haupert. Unsupervised classification to improve the quality of a bird song recording dataset. Ecological Informatics, 74:101952, May 2023. ISSN 1574-9541. doi:10.1016/j.ecoinf.2022.101952. [16] Sam Lapp, Tessa Rhinehart, Louis Freeland-Haynes, Jatin Khilnani, Alexandra Syunkova, and Justin Kitzes. OpenSoundscape: An open-source bioacoustics analysis package for Python. Methods in Ecology and Evolution, 14(9):2321–2328, 2023. ISSN 2041-210X. doi:10.1111/2041-210X.14196. [17] Sound-scape-explorer/sound-scape-explorer. sound-scape-explorer, January 2026. [18] Nilo Merino Recalde. Pykanto: A python library to accelerate research on wild bird song. Methods in Ecology and Evolution, 14(8):1994–2002, August 2023. ISSN 2041-210X, 2041-210X. doi:10.1111/2041-210X.14155. [19] Bart van Merriënboer, Jenny Hamer, Vincent Dumoulin, Eleni Triantafillou, and Tom Denton. Birds, bats and beyond: Evaluating generalization in bioacoustics models. Frontiers in Bird Science, 3, July 2024. ISSN 2813-3870. doi:10.3389/fbirs.2024.1369756. [20] Stefan Kahl, Connor M. Wood, Maximilian Eibl, and Holger Klinck. BirdNET: A deep learning solution for avian diversity monitoring. Ecological Informatics, 61:101236, March 2021. ISSN 15749541. doi:10.1016/j.ecoinf.2021.101236. [21] Cristian Pérez-Granados. BirdNET: Applications, performance, pitfalls and future opportunities. Ibis, 165(3):1068–1075, 2023. ISSN 1474-919X. doi:10.1111/ibi.13193. [22] Burooj Ghani, Vincent J. Kalkman, Bob Planqué, Willem-Pier Vellinga, Lisa Gill, and Dan Stowell. Generalization in birdsong classification: Impact of transfer learning methods and dataset characteristics, September 2024. [23] Burooj Ghani, Vincent J. Kalkman, Bob Planqué, Willem-Pier Vellinga, Lisa Gill, and Dan Stowell. Impact of transfer learning methods and dataset characteristics on generalization in birdsong classification. Scientific Reports, 15(1):16273, May 2025. ISSN 2045-2322. doi:10.1038/s41598-025-00996-2. [24] Slade Allen-Ankins, Sebastian Hoefer, Jacopo Bartholomew, Sheryn Brodie, and Lin Schwarzkopf. The use of BirdNET embeddings as a fast solution to find novel sound classes in audio recordings. Frontiers in Ecology and Evolution, 12, January 2025. ISSN 2296-701X. doi:10.3389/fevo.2024.1409407. [25] Jenny Hamer, Eleni Triantafillou, Bart van Merriënboer, Stefan Kahl, Holger Klinck, Tom Denton, and Vincent Dumoulin. BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics, December 2023. [26] xeno-canto. Xeno-canto :: Sharing wildlife sounds from around the world. https://xeno-canto.org/, 2025. [27] Juan Sebastián Cañas, Maria Paula Toro-Gómez, Larissa Sayuri Moreira Sugai, Hernán Darío Benítez Restrepo, Jorge Rudas, Breyner Posso Bautista, Luís Felipe Toledo, Simone Dena, Adão Henrique Rosa Domingos, Franco Leandro de Souza, Selvino Neckel-Oliveira, Anderson da Rosa, Vítor Carvalho-Rocha, José Vinícius Bernardy, José Luiz Massao Moreira Sugai, Carolina Emília dos Santos, Rogério Pereira Bastos, Diego Llusia, and Juan Sebastián Ulloa. AnuraSet: A dataset for benchmarking Neotropical anuran calls identification in passive acoustic monitoring, July 2023.
16
Kather et al.
[28] Julian C. Schäfer-Zimmermann, Vlad Demartsev, Baptiste Averly, Kiran Dhanjal-Adams, Mathieu Duteil, Gabriella Gall, Marius Faiß, Lily Johnson-Ulrich, Dan Stowell, Marta B. Manser, Marie A. Roch, and Ariana Strandburg-Peshkin. Animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics, July 2024. [29] Mustafa Chasmai, Alexander Shepard, Subhransu Maji, and Grant Van Horn. The iNaturalist Sounds Dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, November 2024. [30] Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, September 2020. [31] Cornell Lab of Ornithology. Raven Pro: Interactive sound analysis software. Version 1.5, 2014. [32] Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked Autoencoders that Listen. Advances in Neural Information Processing Systems, 35:28708–28720, December 2022. [33] René Heinrich, Lukas Rauch, Bernhard Sick, and Christoph Scholz. AudioProtoPNet: An interpretable deep learning model for bird sound classification. Ecological Informatics, 87:103081, July 2025. ISSN 1574-9541. doi:10.1016/j.ecoinf.2025.103081. [34] Masato Hagiwara. AVES: Animal Vocalization Encoder based on Self-Supervision, October 2022. [35] Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. BEATs: Audio Pre-Training with Acoustic Tokenizers, December 2022. [36] David Robinson, Adelaide Robinson, and Lily Akrapongpisak. Transferable Models for Bioacoustics with Human Language Supervision, August 2023. [37] Lukas Rauch, René Heinrich, Ilyass Moummad, Alexis Joly, Bernhard Sick, and Christoph Scholz. Can Masked Autoencoders Also Listen to Birds?, August 2025. [38] Vincent Kather, Fabian Seipel, Benoit Berges, Genevieve Davis, Catherine Gibson, Matt Harvey, Lea-Anne Henry, Andrew Stevenson, and Denise Risch. Development of a machine learning detector for North Atlantic humpback whale song. The Journal of the Acoustical Society of America, 155(3):2050–2064, March 2024. ISSN 0001-4966. doi:10.1121/10.0025275. [39] Ilyass Moummad, Nicolas Farrugia, Romain Serizel, Jeremy Froidevaux, and Vincent Lostanlen. Mixture of Mixups for Multi-label Classification of Rare Anuran Sounds, March 2024. [40] David Robinson, Marius Miron, Masato Hagiwara, and Olivier Pietquin. NatureLM-audio: An Audio-Language Foundation Model for Bioacoustics, November 2024. [41] Burooj Ghani, Tom Denton, Stefan Kahl, and Holger Klinck. Global birdsong embeddings enable superior transfer learning for bioacoustic classification. Scientific Reports, 13(1):22876, December 2023. ISSN 2045-2322. doi:10.1038/s41598-023-49989-z. [42] Bart van Merriënboer, Vincent Dumoulin, Jenny Hamer, Lauren Harrell, Andrea Burns, and Tom Denton. Perch 2.0: The Bittern Lesson for Bioacoustics, August 2025. [43] Ilyass Moummad, Romain Serizel, Emmanouil Benetos, and Nicolas Farrugia. Domain-Invariant Representation Learning of Bird Sounds. September 2024. [44] Ilyass Moummad, Romain Serizel, and Nicolas Farrugia. Regularized Contrastive Pre-training for Few-shot Bioacoustic Sound Detection, January 2024.
Kather et al.
17
[45] Ben Williams, Bart van Merriënboer, Vincent Dumoulin, Jenny Hamer, Eleni Triantafillou, Abram B. Fleishman, Matthew McKown, Jill E. Munger, Aaron N. Rice, Ashlee Lillis, Clemency E. White, Catherine A. D. Hobbs, Tries B. Razak, Kate E. Jones, and Tom Denton. Leveraging tropical reef, bird and unrelated sounds for superior transfer learning in marine bioacoustics, May 2024. [46] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin Wilson. CNN Architectures for Large-Scale Audio Classification, January 2017. [47] Masato Hagiwara, Benjamin Hoffman, Jen-Yu Liu, Maddie Cusimano, Felix Effenberger, and Katie Zacarian. BEANS: The Benchmark of Animal Sounds. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, June 2023. doi:10.1109/ICASSP49357.2023.10096686. [48] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, November 2018. [49] Vincent S. Kather, Burooj Ghani, and Dan Stowell. Clustering and Novel Class Recognition: Evaluating Bioacoustic Deep Learning Feature Extractors. In Proceedings of the 11th Convention of the European Acoustics Association Forum Acusticum / EuroNoise 2025, 2025. doi:10.61782/fa.2025.0231. [50] Simone Romano, James Bailey, Vinh Nguyen, and Karin Verspoor. Standardized Mutual Information for Clustering Comparisons: One Step Further in Adjustment for Chance. In Proceedings of the 31st International Conference on Machine Learning, pages 1143–1151. PMLR, June 2014. [51] Douglas Steinley, Michael J. Brusco, and Lawrence Hubert. The variance of the adjusted Rand index. Psychological Methods, 21(2):261–272, 2016. ISSN 1939-1463. doi:10.1037/met0000049. [52] Carson Sievert, Chris Parmer, Toby Hocking, Scott Chamberlain, Karthik Ram, Marianne Corvellec, and Pedro Despouy. Plotly: Create Interactive Web Graphics via ’plotly.js’, November 2015. [53] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008. [54] Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2(1):37–52, August 1987. ISSN 0169-7439. doi:10.1016/0169-7439(87)80084-9.
18
10
Kather et al.
|
APPENDIX
In this section further materials are provided. TA B L E 2 Selection of API functions of bacpipe. All pipelines can be used with the supported models or models that are passed during runtime. name
brief description
config, settings
bacpipe config and bacpipe settings
supported_models
list all supported models
integrated pipelines play
runs the entire bacpipe pipeline, for all specified models and evaluation tasks
benchmark
calculates precision, recall and f1 score for a classifier based on provided ground truth annotations
Loader
class to handle file loading and saving
Embedder
class to handle model loading and processing
Embedder.get_embeddings_from_model
returns a single embedding array from an audio file
Embedder.generate_embeddings
generates embeddings and classifier predictions for all audio files in the specified directory
Embedder.run_pipeline_for_models
generates embeddings, classifier predictions and dimension reduced embeddings from all specified models
functions for data handling get_audio_files
list all audio files in a directory with supported formats
get_dt_filename
return array of datetimes corresponding to the time and date in each audio file name
ground_truth_by_model
return multi-label ground truth associated with the model specific input time length
create_default_labels
returns dictionary with automatically generated labels
Loader attributes Loader.embeddings()
returns a Numpy array (or dictionary) of all embeddings
Loader.predictions()
returns a Numpy array (or dictionary or dataframe) of all classifier predictions