Skip to main content An official website of the United States government Here's how you know Here's how you know Official websites use .gov A .gov website belongs to an official government organization in the United States. Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites. Search Log in Dashboard Publications Account settings Log out Search… Search NCBI Primary site navigation Search Logged in as: Dashboard Publications Account settings Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide PERMALINK Copy As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Arch Public Health . 2026 Mar 3;84:74. doi: 10.1186/s13690-026-01869-4 Search in PMC Search in PubMed View in NLM Catalog Add to search How to ensure effective and sustainable ILI surveillance in Belgian general practices? : protocol Mélanie Nahimana Mélanie Nahimana 1 Health Service Research Department, Sciensano, Rue Juliette Wytsman,14, Ixelles, 1050 Belgium Find articles by Mélanie Nahimana 1, ✉ , Sherihane Bensemmane Sherihane Bensemmane 1 Health Service Research Department, Sciensano, Rue Juliette Wytsman,14, Ixelles, 1050 Belgium Find articles by Sherihane Bensemmane 1 , Floriane Rouvez Floriane Rouvez 1 Health Service Research Department, Sciensano, Rue Juliette Wytsman,14, Ixelles, 1050 Belgium Find articles by Floriane Rouvez 1 , Laura Debouverie Laura Debouverie 1 Health Service Research Department, Sciensano, Rue Juliette Wytsman,14, Ixelles, 1050 Belgium Find articles by Laura Debouverie 1 , Sarah Moreels Sarah Moreels 1 Health Service Research Department, Sciensano, Rue Juliette Wytsman,14, Ixelles, 1050 Belgium Find articles by Sarah Moreels 1 , Robrecht De Schreye Robrecht De Schreye 1 Health Service Research Department, Sciensano, Rue Juliette Wytsman,14, Ixelles, 1050 Belgium Find articles by Robrecht De Schreye 1 , Nathalie Bossuyt Nathalie Bossuyt 1 Health Service Research Department, Sciensano, Rue Juliette Wytsman,14, Ixelles, 1050 Belgium 2 Epidemiology of Infectious Diseases Department, Sciensano, Rue Juliette Wytsman,14, Ixelles, 1050 Belgium Find articles by Nathalie Bossuyt 1, 2 Author information Article notes Copyright and License information 1 Health Service Research Department, Sciensano, Rue Juliette Wytsman,14, Ixelles, 1050 Belgium 2 Epidemiology of Infectious Diseases Department, Sciensano, Rue Juliette Wytsman,14, Ixelles, 1050 Belgium ✉ Corresponding author. Received 2025 Nov 6; Accepted 2026 Feb 14; Collection date 2026. © The Author(s) 2026 Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ . PMC Copyright notice PMCID: PMC13067514 PMID: 41772685 Abstract Background In Belgium, the COVID-19 Barometer in General Practices (cBGP), a semi-automated tool, was rapidly set up to support the surveillance of COVID-19 by extracting data from the Electronic Medical Records (EMR) of General Practitioners (GPs). Hence, the cBGP enabled a high GP participation rate and the rapid follow-up of COVID-19. Concurrently, the instrument collected data on Influenza-like Illness (ILI) as an early marker of COVID-19 activity. This extraction-based surveillance system could take over the registration of ILI cases currently done by the Belgian Sentinel General Practitioners network (SGP). Methods In order to identify the most suitable alternative for effective and long-term surveillance of ILI, the current study is investigating the contribution and limitations of using the code-based cBGP as a replacement for the questionnaire-based SGP system to monitor the syndrome. An observational retrospective study covering 3 influenza seasons from 2021 to 2024 is being carried out. Both qualitative and quantitative outcome measures are determined to gather evidence on the systems’ performance, based on nine attributes namely: Data Quality, ILI Incidence, Sensitivity, Representativeness, Timeliness, Acceptability, Simplicity, Stability and Flexibility. The methodological framework relies on the Centers for Disease Control and Prevention (CDC) guidelines for evaluating public health surveillance systems and the Simple Multi-Attribute Rating Technique (SMART) is used to facilitate the decision-making process of an appropriate alternative. Three alternatives are given a value according to the score and the weight-related importance of attributes by a group of experts. The alternative with the higher endorsement is then considered as the preferable one and recommendations for implementation are formulated. Discussion This combined approach, which blends performance analysis and multi-criteria decision analysis, allows for a sound and reproducible comparison of systems. Moreover, while the Covid-19 pandemic has accelerated the development of a national code-based system, this study aims to start redefining the role of the barometer in insuring robust disease surveillance. This protocol could serve as a basis for validating other syndromic surveillance data from extraction-based systems in primary care. Keywords: Influenza, Electronic medical records, Evaluation, Sentinel surveillance, Multi-criteria decision-making Text box 1: Contributions to the literature • This study protocol provides a thorough insight into barriers and possible improvements of code-based compared to questionnaire-based systems to ensure ILI surveillance. • Presents a replicable and transparent method for systems’ evaluation and adaptation. • Highlights the importance of systems’ validation to support accurate data-based decision-making. Open in a new tab Background Influenza remains a major concern for public health policies due to its associated burden of disease [ 1 , 2 ], the related socio-economic impact [ 2 ] and its pandemic potential [ 3 ]. Furthermore, the possibility of preventing influenza through vaccination [ 4 ] underlines the importance of adequate and rapid surveillance of the syndrome, which could, if necessary, help the implementation of targeted measures by health authorities. To guarantee good-quality surveillance, the WHO recommends the systematic and ongoing collection of data from a limited number of carefully selected surveillance sites [ 5 ]. These sentinel sites record suspected cases of influenza and take nasal or nasopharyngeal swabs from patients, which are then tested using RT-PCR (Reverse Transcription Polymerase Chain Reaction) to confirm the diagnosis. In Europe, sentinel primary care networks have been well established for several decades and were further strengthened by harmonisation initiatives [ 6 , 7 ]. However, difficulties can arise at the national level, particularly regarding the representativeness of the selected sentinel sites and the time lag between data collection and reporting. Seeking to push the boundaries of sentinel systems and move towards real-time data reporting, researchers, globally, evaluated the rapidity and correlation of data collected between sentinel and alternative systems. Among the data sources explored, surveillance systems based on Influenza-like Illness (ILI) data extracted from General Practitioners’ (GPs) Electronic Medical Records (EMR) have the potential to complement sentinel data [ 8 ]. In Belgium, an influenza surveillance pyramid [ 9 ] highlights the various data sources according to the severity of the disease, ranging from asymptomatic infection to death. It is part of a broader integrated ecosystem for the surveillance of acute respiratory infection comprising systems whose data comes from different sources: citizen volunteers, hospitals, general practitioners, nursing homes, sentinel laboratories and environmental sampling (wastewater). In addition, a management tool called “Respi-Radar” gathers epidemiological information from six of these systems to present a clear and accurate picture of the epidemic situation, facilitating a rapid and appropriate response by public health authorities. Among the integrated systems, the surveillance pyramid model encompasses notably the Sentinel General Practitioners Network (SGP) and the Sentinel Laboratories network (SL). The SGP [ 10 ] exists since 1979 and ensures ILI surveillance. GPs report cases on a weekly basis, using a questionnaire, and collect samples for pathogen identification. The data is then transmitted to the national authorities and the European Centre for Disease Prevention and Control (ECDC). The SL [ 11 ] consists of hospital and non-hospital laboratories. This network reports positive diagnostic tests for about 40 pathogens, including the influenza virus. More recently, following rapid advances in technology and healthcare needs, new data collection tools have been developed. The iCAREdata project [ 12 ] was launched in 2014 in the Flemish Region (one of Belgium’s three regions). This system is based on the automated daily extraction of ILI data from the software of GPs who work in Out Of Hours (OOH) services. Likewise, at the national level, the COVID-19 Barometer in General Practices (cBGP) [ 13 ], a code-based system, was established in September 2020. The cBGP demonstrated a high participation rate as well as rapid detection of COVID-19 outbreaks [ 14 ]. This tool, which relied on diagnostic codes extracted semi-automatically from GPs’ EMR, also enabled daily data collection on ILI as an early indicator of COVID-19 activity. Researchers have long emphasised the importance of evaluating surveillance systems to determine whether they are achieving their objectives and using resources effectively [ 15 ]. Comparative studies between data from sentinel systems and EMR-based systems exist, however, these generally take few characteristics into account, focusing mainly on ILI activity [ 16 , 17 ]. Other attributes, such as simplicity, flexibility, data quality or acceptability, are rarely studied. Similarly, for the cBGP, no comprehensive comparative study assessing all the system’s characteristics has been carried out in Belgium, and the tool is not yet validated for ILI surveillance. In this context, and considering the promising potential of systems based on EMR data extraction, exploring the possibilities offered by the cBGP, as well as its limitations, is warranted. This article presents the methodology used to identify the strengths and weaknesses of the cBGP system in relation to the established surveillance of the SGP, addressing the following question: ‘What are the gains and losses of replacing the questionnaire-based SGP method with the code-based cBGP method for ILI surveillance data collection in Belgian general practices, in order to support data-based decision-making?’ Methods Study design A retrospective observational study [ 18 ] is conducted based on aggregated data collected in Belgian general practices. The SGP and cBGP systems are compared regarding nine systems’ attributes: Data Quality, ILI Incidence, Sensitivity, Representativeness, Timeliness, Stability, Simplicity, Acceptability, and Flexibility. Additional data sources, iCAREdata and SL, are used for comparative evaluation of ILI Incidence. All studied surveillance systems rely on voluntary participation. The data covers the period from 20 June 2021 to 15 June 2024, thereby excluding the months during which GPs were granted a financial incentive for their participation in the cBGP (from 26/10/2020 to 31/03/2021) [ 14 ]. Three influenza seasons are then considered (defined as running from week 25 each year to week 24 of the following year). Figure 1 illustrates the conditions required to be part of the analyses concerning practices, GP OOH posts and laboratories. The related attributes are presented except for Simplicity, Flexibility, and Stability because their analysis is not based on practice-level information. The quantitative outcome measures of Timeliness only take into account the SGP practices involved during the study, therefore this attribute is not present in the figure. Fig. 1. Open in a new tab Eligibility criteria for attributes analysis The eligibility criteria for practices are described in Fig. 1 . They vary depending on the attributes. Practices must have participated in the cBGP and/or SGP surveillance systems during the study period for Representativeness and Acceptability assessment. In ILI Incidence analysis, the group of participants extends to GP OOH posts of the iCAREdata surveillance system and laboratories part of the SL. For the evaluation of Sensitivity, a minimum participation in the cBGP of three days for each week analysed is required, while for data provided weekly by the SGP, five working days are assumed. Additionally, the participants are restricted to practices that have reported data for the same week through both the SGP and cBGP systems. An extra criterion is applied to the cBGP system data, where only records with a percentage of overall daily coding of diagnoses above 70% are taken into account. This restriction does not apply to the Data Quality assessment, since this includes quantifying low coding percentages, but rather to the attributes previously mentioned (Representativeness, Acceptability, ILI Incidence and Sensitivity). Simplicity, Flexibility and Stability refer to general characteristics of the SGP and cBGP systems, therefore, not at the practice level. Finally for Timeliness, sentinel practices data is analysed for late reporting. The inclusive criterion used for this attribute is the participation of practices during the period under consideration. The study population consists of people of all ages who consulted for flu-like symptoms during the predefined period and whose physician participated in the examined systems, either by reporting ILI cases or by sending samples for pathogen characterisation (in the case of the sentinel laboratories). Data sources Every day in the cBGP system, GPs could perform an audit in their software, which generated the daily number of diagnostic codes recorded, including the R80 code (for ILI), from the ICPC-2 (International Classification of Primary Care, Second Edition [ 19 ]) classification. These figures were then entered manually by GPs into an electronic form (eForm) within their software. The data was transmitted to Healthdata.be [ 20 ], the national health registries platform that facilitates the secure collection, exchange and storage of health data, by 10 a.m. the following day at the latest. The cBGP semi-automatically captured the total daily number of patient contacts for ILI and enabled the participation of a large number of medical practices. From 26 October 2020 to 31 March 2021, the number of participants reached 2068 practices [ 14 ]. However, this number might be lower for the period under review. GPs involved in the sentinel network complete a LimeSurvey standardised form online to report ILI cases. They provide weekly aggregated data on all consultations for ILI diagnoses in 7 age categories (< 1, 1–4, 5–14, 15–19, 20–64, 65–84, 85+), using a clinical case definition. Data is stored on a secured Sciensano server. The size of the network in 2024 consisted of 68 practices (108 GPs) [ 21 ] that regularly participated (at least 26 weeks of reporting per year). In this study, nonetheless, practices that did not regularly take part in the network were also included. As for the microbiology laboratories of SL, diagnostic data is submitted on a weekly basis. The medical data is sent to Sciensano, either via email or through a secured sentinel laboratories platform. For this network, lab-confirmed influenza data is provided; however, it is not possible to determine a proper catchment population for this pathogen. In the iCAREdata system, GPs in Flanders used the ICPC-2 code R80 to record ILI cases in their EMR. The secure exchange and transfer of medical data automatically extracted from GPs’ software to iCAREdata is ensured by the encryption services of eHealth (a Belgian federal institution responsible for electronic services within the healthcare system). A total of 36 GP OOH posts shared their data and the results are available online via the iCAREdata project dashboard [ 22 ]. Performance analysis From the EU’s regulation on cross-border threats [ 23 ] and Belgium’s long-standing experience, the requirements for ILI surveillance are defined. These comprise the continuous surveillance and description of epidemic stages, the rapid detection of changes, the contribution to the global effort (on a national and international scale) to timely assess epidemics or pandemics based on sustainable surveillance systems, and hence support evidence-based decision-making. Following the CDC guidelines for evaluating public health systems [ 24 ], the systems’ performance to achieve these needs is appraised through nine attributes of the CDC framework but the concept “ILI Incidence” is used instead of the “Positive Predictive Value” attribute. This was established to better integrate the surveillance criteria in line with international recommendations for the syndrome. The overarching research question was split into one research question per attribute in order to accurately proceed with the evaluation. The research questions and outcome measures are presented in Table 1 . Table 1. Research questions and related outcome measures per attribute Attributes Research questions Qualitative outcome measures Quantitative outcome measures Data Quality How comparable are the questionnaire-based and the code-based methods to ensure the quality (completeness and validity) of ILI data collection? Comparison of data entry Comparison of error detection and handling in data cleaning Comparison of the frequencies of practice information errors Comparison of the frequencies of epidemiological information errors ILI Incidence How valid are the incidences of weekly GP contacts for ILI provided by the cBGP compared to ILI incidence provided by other Belgian surveillance systems? ILI/influenza intensity levels comparison using Moving Epidemic Method thresholds: - Between the cBGP and SGP systems - Between the cBGP and iCAREdata systems - Between the cBGP and sentinel laboratory systems Correlation of weekly ILI/Influenza activity: - Between the cBGP and SGP systems - Between the cBGP and iCAREdata systems - Between the cBGP and sentinel laboratory systems Sensitivity To what extent is the detection of ILI cases comparable between the cBGP and SGP systems? Comparison of case definitions Comparison of other factors potentially impacting sensitivity Regression analysis of the number of cases registered in the cBGP and SGP systems by the same practices Representative-ness Are the population coverage and geographical distribution of practices comparable between the cBGP and the SGP surveillance systems? Comparison of how to calculate the population at risk Comparison of population coverage Comparison of practices geographic distribution in urban and rural areas Timeliness Is the timeline for reporting ILI cases comparable between the cBGP and SGP systems? Comparison of systems’ timelines for reporting ILI cases Comparison of case reporting times Late reporting assessment Stability To what extent are the data collection problems encountered in the cBGP and SGP systems comparable in terms of type, frequency, and consequences for data reporting? Comparison of the types of problems encountered and their progression Comparison of the number of issues encountered Simplicity How simple is the operation of the cBGP system compared to the SGP system in terms of steps involved, data collected and user-friendliness for GPs? Comparison of variables collected Comparison of the systems’ data flows Comparison of GPs user Experiences Comparison of the number of variables collected Acceptability To what extent is the willingness of GPs to participate in the SGP and cBGP surveillance systems comparable? Description of cited reasons to participate in the system Description of the cited reasons to exit the system Comparison of the frequencies of cited reasons to participate Comparison of the frequencies of cited reasons to exit the system Comparison of participation ratio Comparison of participation duration Flexibility How comparable is the ease of adapting the SGP and cBGP systems to potential changes required for ILI surveillance in the future? Comparison of the type and role of the groups/ organisations involved in the operation of the systems Comparison of the number of organisations involved in the systems’ operation per type Comparison of the resources required to adapt the system Open in a new tab Table 1 shows the research questions by attribute and how they are addressed through the qualitative and quantitative outcome measures provided. Data quality refers to the validity and completeness of the data recorded. Structured data entry is essential to ensure consistency in the data collected and may vary between the SGP and cBGP systems, as well as between the software packages from which data is extracted in the cBGP system. The type of data (structured, semi-structured and unstructured) induced by the functionality of data entry fields, such as auto-complete search fields or free-text areas, is described and compared between systems and software. Error detection and handling within the data cleaning procedures for practice- or epidemiological-level information is also assessed qualitatively by reviewing the scripts. In addition, the frequency of errors is determined. Errors related to practices data or those likely to impact the calculation of the incidence numerator or denominator are counted. Missing values, duplicates, inappropriate values and low daily coding percentages (with a threshold set at 70% by experts consensus) in cBGP data are taken into account. Percentages below 20% are considered acceptable. The concept “ILI incidence” reflects, in this study, the ability to provide reliable data on ILI activity. To evaluate this attribute, weekly cBGP data is compared with data from three systems: SGP, iCAREdata and SL. Spearman’s correlation (with a p -value threshold set at 0.05) is used to estimate the strength of the correlation between the systems’ data, while Bland Altman analysis is applied to identify the extent of the disagreement, whether resulting from systematic bias or random error. Then, a qualitative assessment is carried out to define the start, duration, intensity and end of ILI epidemics. For this, a heat map whose colours correspond to ILI activity levels based on the cBGP data, is compared with heat maps derived from the other systems’ data. The period from week 40 of 2022 to week 24 of 2024 is chosen to limit the impact of COVID-19 activity [ 25 – 27 ]. The thresholds for going from one intensity level to another and to define the start of the epidemic are established with the Moving Epidemic Method (MEM) [ 28 ]. Sensitivity refers to the ability of capturing in a valid and complete way all consultations for ILI cases in the SGP and cBGP systems. Two aspects are reviewed in the qualitative evaluation. First, the different approaches to case definition are described. The clinical definition of cases is compared with the application of ICPC-2 code R80. Second, the implementation of measures to encourage participants to complete their forms within the specified time frame is presented. Finally, a conditional fixed-effects negative binomial regression analysis, accounting for the clustering effect of practices, is performed to assess the relationship between the number of cases reported to both systems by the same practices in a given week. Furthermore, the season and the type of week (epidemic or not) are included for time patterns to be modelled. Representativeness refers to the population covered by the SGP and cBGP systems and the distribution of participating practices across different geographical units. The population at risk is compared in order to highlight both the advantages and limitations of the systems’ representativeness. In addition, population coverage is calculated at the minimum level of the district (or NUTS (Nomenclature of Territorial Units for Statistics) 3 region), with a lower limit of 1% to be achieved. The geographical distribution of practices in rural and urban areas – using the classification in Eurostat’s methodological manual on territorial typologies – is analysed and compared between systems, but also with active GPs (having at least 500 contact patients/year) who are not part of the systems under study. The urban cluster, defined as ‘a set of contiguous cells with a surface area of 1 km² with a population density of at least 300 inhabitants per km² and a minimum total population of 5,000 inhabitants’ [ 29 ], is taken as the base unit. A district in which up to 50% of the population resides in urban clusters is classified as a rural area; between 50% and 80%, as an intermediate area; and more than 80%, as an urban area. Timeliness reflects the speed at which the steps composing the systems’ operation are carried out. It is determined by establishing a timeline for ILI case reporting in both systems. Each stage of the reporting process is described in detail, along with the maximum and minimum number of days required per stage. In the SGP, sentinel GPs can start completing the questionnaires without submitting them immediately, and the start and completion dates are logged. The proportion of ILI data late reporting is then calculated. Stability represents the system’s ability to provide data without failure and to operate without interruption. Tickets sent by Sciensano epidemiologists to Healthdata.be to report an incident via the Service Now ticketing system are counted. The types of problems and their evolution are described for the cBGP system. However, data from this ticketing system prior to 2023 is not available. The ‘Weekly bulletin respiratory infections’ [ 30 ] also serves as a source for identifying any gaps in the reporting of ILI data regarding the cBGP and SGP systems. The number of emails from the coordination team, HD tickets and ‘Weekly bulletin respiratory infections’ indicating a reporting issue is used as a quantitative indicator. Moreover, the period during which data was not reported due to an issue is provided in order to measure the impact of the incident. The averages, minimums and maximums number of days of interruption are compared. Simplicity refers to the ease of the systems’ operation and the user experience. The data flow, which encompasses all stages from the GP consultation to the researchers’ report, is examined for the SGP and cBGP systems. Ease of use from the perspective of GPs is appraised by studying the systems’ websites and procedure reports. Information on how GPs are invited to register or participate in the systems, how their data is submitted, and what documents are made available to support them, is included in the evaluation. As a quantitative indicator, a comparison is made between the number of variables required to define an ILI case in both systems. The system with the fewest variables and steps is considered simpler. Acceptability refers to the willingness of GPs to participate in the surveillance system. In the cBGP system, GPs could participate or stop their participation at any time, directly via their software. Comments about their participation in the cBGP, sometimes found in emails sent to the general cBGP mailbox requesting to unsubscribe from the newsletter, are reviewed. In the SGP system, participation requests and feedback emails, as well as the annually updated survey on GP profiles, the ‘Profile enquête’, are analysed to assess GPs’ willingness to be part of the network. In the Profile enquête of 2024, responses to the question ‘Why do you participate in the SGP as a registration partner?’ are examined for GPs who were active in registering ILI cases. The reasons why GPs in the network no longer wished to participate in the system could be specified in the emails sent to the network coordination team to indicate their decision to leave the network. The notified reasons for participation or cessation of participation are listed and categorised for both systems. The frequency of the resulting types is then presented. Subsequently, the participation ratio– the average number of participating GPs divided by the number of non-participating GPs in the same geographical area– is determined per season for each system by district, region and at the national level. The number of non-participating GPs is based on the number of active GPs per district provided by the National Institute of Health and Disability Insurance (NIHDI), minus the number of GPs in that district who participate in the system being analysed. In addition, the practice-specific number of weeks of participation for each season is calculated, and the box plot for each system is provided. The results are compared using the Mann-Whitney test to assess differences between systems, with a p -value threshold set at 0.05. Flexibility denotes the ability of systems to adapt to change. The role of organisations or groups within the operational process is described and classified according to their involvement in the data flow: data providers, software providers, data users and data collectors. Furthermore, the number of entities taking part in system modifications by level of involvement is shown. The assumption is that the more intermediaries a system has in key stages, the less flexible it is. A hypothetical scenario is developed to gather information on the resources needed to integrate new requirements for collected variables with the example of new recommendations on collected age categories. The human resources and time required to implement this change are compared between the SGP and cBGP systems. Although financial considerations are important when setting up changes, they are not part of the scope of this study. Decision making-process After evaluating the performance based on all attributes, a group of experts is selected to respond to a survey aimed at determining which of the following alternatives is chosen: (i) maintain the existing SGP system, (ii) replace the network with the cBGP data collection method for ILI surveillance, or (iii) monitor ILI simultaneously using both systems. The experts’ profession, organisation or workplace, their field of expertise and their involvement in a surveillance system are taken into account in order to limit selection bias. In the questionnaire, each attribute is weighted according to its importance, and a score is assigned per attribute for each alternative. The sum of all the products of the score and the weighting per attribute gives a value per alternative. The highest endorsed alternative is designated as the preferred option. The latter is discussed with the experts and researchers in order to formulate recommendations towards the funding health authorities. Discussion This validation protocol not only relies on a considerable number of characteristics and outcome measures but also builds further on the CDC framework by combining the assessment of attributes with a multicriteria decision-making technique. Therefore, the experts’ perspective accounts for both qualitative information and quantified results, facilitating a thorough, transparent and reproducible analysis. Given ILI surveillance needs, the protocol covers the key stages from evaluation to practical applications. This study reflects on the evolution of the barometer and incorporates the contributions from EMR research. Even though the tumultuous context of 2020 limited the time available to build up a sustainable monitoring system, the first version of the barometer [ 31 ] successfully anchored the tool at the national level, with the participation of a large number of GPs. In addition, this instrument integrated all software packages to collect data on consultations for respiratory complains, protective personal equipment and GPs’ workload using e-Forms (or less frequently, web forms). In contrast, subsequent versions benefit from the lessons learnt on the national and global scale. The updated version, focusing on epidemiological indicators, was highly accepted and provided rapidly available data. However, some areas for improvement have been pinpointed [ 14 ], such as limited harmonisation between software, variability in coding between GPs, and the lack of full automation. Similarly, Van der Bij et al. [ 32 ] found significant variations in the quality of recording in EHR (Electronic Health Records) between GPs and the software used. More specifically, in terms of coding, a Norwegian study [ 33 ] demonstrated the challenge of using the ICPC classification for clinical practice, notably due to missing codes, code entries that did not correspond to valid ICPC-2 codes, or diagnostic texts that did not match the correct ICPC-2 diagnosis. Considering the broader aspect of the transition from paper to EMR-based surveillance, in Massachusetts (United States), Yih et al. [ 34 ] highlighted the stability of the EMR-based system and its rapidity in reporting information, while reducing clinicians’ workload. Conversely, researchers also emphasised the different coding practices of family physicians. The chronology of the transition process, in the Netherlands, was described by Schweikardt et al. [ 35 ] and comprised the integration of an EHR-based system used for routine analyses (incidence calculations, etc.) with another network providing more in-depth data, enabling for example, the etiological study of a case. Therefore, the present work embeds a significant number of evaluation criteria in order to cover, but also go beyond, the mentioned aspects to durably support the cBGP usage. Performance The choice of attributes and indicators does not follow strict rules. The absence of a gold standard leads to a wide variety of approaches referring to different system attributes. Nevertheless, the CDC guidelines are commonly used, and the attributes’ definition provided are well accepted. This remains a solid framework to allow a comprehensive analysis for the examination of at least 5 characteristics, depending on the objective delineated beforehand [ 36 ]. An important part of the study is related to the determination of ILI activity levels. Multiple threshold-setting methods, for ILI transmissibility and severity assessment, exist (e.g. the average epidemic curvee [ 5 , 37 ], the MEM [ 28 ] or the percentile method), but they usually require historical data. Due to the recent launch of the cBGP system in September 2020 and the large proportion of disregarded data (because it was overly impacted by COVID-19 activity), an insufficient number of seasons are available for the classic methods’ application. However, the MEM method is widely used and adopted by the ECDC as a tool for comparison between countries and also within the same country. In Belgium, it has been applied for many years as a reference for SGP data analysis [ 30 ]. Furthermore, this model has another advantage: it can be adapted to different contexts, for example to virological data, which are also taken into account in this study. Alternatively, a new method presented in the Eurosurveillance rapid communications, the Mean Standard Deviation (MSD) [ 38 ], was designed to process limited historical data from COVID-19 surveillance systems but requires more detailed presentation and further publications for reinforcing the results. Although little historical data is available, the MEM method appeared more robust, while MSD could be explored once it has been investigated in other studies, until we have sufficient data to perform the MEM method as recommended. The difficulty in distinguishing COVID-19 from influenza activity is a limitation of this study. Symptoms were even more difficult to differentiate at the beginning of the outbreak, when no specific diagnostic code was available to separately record COVID-19 cases in the barometer. The exclusion of the critical period affected, the comparison with influenza-specific data from sentinel laboratories, and the consideration of virological surveillance among a subset of ILI patients could help to disentangle the data relating to both syndromes. However, this may not be sufficient to establish a clear distinction. Furthermore, concerning confirmed cases of COVID-19, which could bring a supporting insight, testing was not systematic during the period studied, particularly since early 2022, when it was no longer mandatory. Nevertheless, the sensitivity analysis, which shows how the two systems capture cases of ILI, should help to identify areas for improvement in that regard. The correspondence between ICPC-2 codes and ILI clinical case definitions remains a point of discussion within the scientific community. The study results could help to better understand this critical aspect, while barriers for drawing conclusions in other countries shouldn’t be neglected. Local coding practices and understanding of the coding system can influence coding decisions and may vary depending on the geographical region and the software packages [ 5 ]. Expert evaluation On the one hand, SMART offers fundamental advantages, such as adaptability, simplicity and the ability to evaluate dependent criteria. The technique has been utilised across various domains, including health-related topics and in multiple decision-making scenarios [ 39 ]. In addition, it is relatively easy to apply [ 40 ]. On the other hand, the use of the SMART method may lack specificity and sufficient references for evaluating surveillance systems in relation to infectious disease monitoring. Another disadvantage, inherent to the technique but common to multi-criteria decision-making analyses, is the subjective aspect of experts’ evaluation. Consequently, the transparency of the procedure is a critical point in the SMART approach to increase trust in the outcomes [ 39 ]. Further steps In mid-October 2024, the GP infection barometer [ 41 ], a fully automated tool, took over from the cBGP. Therefore, the recommendations arising from this study should be interpreted through the lens of the tool’s new version to ensure their proper application. In addition, some of the newly monitored diagnoses could be explored in greater depth using the methodological framework of this protocol. In light of a wider system of early warning setups, encompassing digital techniques such as Digital Contact Tracing (DCT) based on advanced machine learning analytics [ 42 , 43 ], could strengthen preparedness for a future pandemic while preserving privacy [ 44 ]. These models could help to identify patient zero or the source of a rumour as well as constrain the spread of transmission of a disease or misinformation. Alongside, the proactive contact tracing designed to anticipate the wide spread of an infection, showed encouraging results in generating early signals as well as maintaining individual privacy [ 45 ]. However, the reliability of Google Trends-derived data, which can be employed as a data source in DCT, is pending on certain factors, such as queries and web search indexes [ 46 ]. Acknowledgements We warmly thank all the teams and organisations that contributed to the collection and analysis of data for the studied surveillance systems as well as GPs who ensure the durability of these systems. Abbreviations AViQ Agence pour une Vie de Qualité cBGP COVID-19 Barometer in General Practices CDC Centers for Disease Control and Prevention DCT Digital Contact Tracing ECDC European Centre for Disease Prevention and Control eForm electronic form EHR Electronic Health Records EMR Electronic Medical Records GPs General Practitioners ICPC-2 International Classification of Primary Care, Second Edition ILI Influenza-like Illness INAMI/RIZIV National Institute for Health and Disability Insurance (in French/ in Dutch) MEM Moving Epidemic Method MSD Mean Standard Deviation NIHDI National Institute for Health and Disability Insurance NUTS Nomenclature of Territorial Units for Statistics OOH Out Of Hours RT-PCR Reverse Transcription Polymerase Chain Reaction SGP Sentinel General Practitioners network SL Sentinel Laboratories network SMART Simple Multi-Attribute Rating Technique Authors’ contributions M.N. and N.B. were responsible for the study protocol conception. M.N. prepared the initial draft of the manuscript. All authors participated in the critical appraisal of the protocol. All authors have read and approved the final manuscript. Funding The projects were commissioned by several Belgian health authorities representing the country’s federal and regional levels of governance: Agence pour une Vie de Qualité (AViQ) - Wallonia, Department Zorg - Flanders, Vivalis - Brussels-Capital and the National Institute for Health and Disability Insurance (INAMI/RIZIV). Data availability No datasets were generated or analysed during the current study. Declarations Ethics approval and consent to participate This work is based on the secondary use of data collected by routine surveillance activities, no personal data is used. The Sentinel General Practitioners network was authorised by the Social Security and Health Sectoral Committee by deliberation No 17/065 of 18 July 2017, amended on 20 March 2020 and 1 September 2020. It has been approved by the Commissie Medische Ethiek of the Vrije Universiteit Brussel (BUN: 1432021000555). Ethical committee and Social Security and Health Sectoral Committee approvals has been waived for cBGP. Therefore, no ethical authorisation was required. Consent for publication This manuscript contains no individual person’s data. Competing interests The authors declare no competing interests. Footnotes Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References 1. Troeger CE, Blacker BF, Khalil IA, et al. Mortality, morbidity, and hospitalisations due to influenza lower respiratory tract infections, 2017: an analysis for the Global Burden of Disease Study 2017. Lancet Respir Med. 2019;7(1):69–89. 10.1016/S2213-2600(18)30496-X. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 2. Molinari NAM, Ortega-Sanchez IR, Messonnier ML, et al. The annual impact of seasonal influenza in the US: Measuring disease burden and costs. Vaccine. 2007;25(27):5086–96. 10.1016/j.vaccine.2007.03.046. [ DOI ] [ PubMed ] [ Google Scholar ] 3. Tscherne DM, García-Sastre A. Virulence determinants of pandemic influenza viruses. J Clin Invest. 2011;121(1):6–13. 10.1172/JCI44947. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 4. Trombetta CM, Kistner O, Montomoli E, Viviani S, Marchi S. Influenza Viruses and Vaccines: The Role of Vaccine Effectiveness Studies for Evaluation of the Benefits of Influenza Vaccines. Vaccines. 2022;10(5):714. 10.3390/vaccines10050714. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 5. World Health Organization. Global Epidemiological Surveillance Standards for Influenza. World Health Organization; 2013. https://iris.who.int/handle/10665/311268 . 6. van Casteren V, Leurquin P. Eurosentinel: development of an international sentinel network of general practitioners. Methods Inf Med. 1992;31(2):147–52. [ PubMed ] [ Google Scholar ] 7. ECDC. European Influenza Surveillance Network (EISN). 2025. https://www.ecdc.europa.eu/en/about-us/partnerships-and-networks/disease-and-laboratory-networks/eisn . 8. Hammond A, Kim JJ, Sadler H, Vandemaele K. Influenza surveillance systems using traditional and alternative sources of data: A scoping review. Influenza Other Respir Viruses. 2022;16(6):965–74. 10.1111/irv.13037. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 9. Public Health Agency of Canada. Canadian Pandemic Influenza Preparedness: Planning Guidance for the Health Sector. 2015. https://www.phac-aspc.gc.ca/cpip-pclcpi/assets/pdf/report-rapport-2015-eng.pdf . 10. Van Casteren V. Thirty years Registration Network of Sentinel General Practitioners. Arch Public Health. 2009;67(Suppl 2):3–15. [ Google Scholar ] 11. Muyldermans G, Ducoffre G, Leroy M, Dupont Y, Quolin S. participating sentinel laboratories. Surveillance of Infectious Diseases by the Sentinel Laboratory Network in Belgium: 30 Years of Continuous Improvement. Chuang JH, ed. PLOS ONE. 2016;11(8):e0160429. 10.1371/journal.pone.0160429. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 12. Universiteit Antwerpen, iCAREdata. https://www.uantwerpen.be/nl/projecten/icaredata/wat-is-icaredata/ . Accessed 6 Nov 2025. 13. Sciensano. The COVID-19-Barometer in general practices. https://www.sciensano.be/en/covid-19-barometer-general-practices . Accessed 6 Nov 2025. 14. Vos B, Debouverie L, Doggen K, et al. Monitoring COVID-19 in Belgian general practice: A tool for syndromic surveillance based on electronic health records. Eur J Gen Pract. 2024;30(1):2293699. 10.1080/13814788.2023.2293699. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 15. Groseclose SL, Buckeridge DL, Public Health Surveillance Systems. Recent Advances in Their Use and Evaluation. Annu Rev Public Health. 2017;38(1):57–79. 10.1146/annurev-publhealth-031816-044348. [ DOI ] [ PubMed ] [ Google Scholar ] 16. Truyers C, Lesaffre E, Bartholomeeusen S, et al. Computerized general practice based networks yield comparable performance with sentinel data in monitoring epidemiological time-course of influenza-like illness and acute respiratory illness. BMC Fam Pract. 2010;11(1):24. 10.1186/1471-2296-11-24. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 17. Blozik E, Grandchamp C, Von Overbeck J. Influenza surveillance using data from a telemedicine centre. Int J Public Health. 2012;57(2):447–52. 10.1007/s00038-011-0240-1. [ DOI ] [ PubMed ] [ Google Scholar ] 18. Nahimana M, Bensemmane S, Rouvez F, et al. Validation of a Code-Based Data Collection Method in Primary Care for Influenza-like Illness Surveillance in Belgium: Protocol. Sciensano (Belgium); 2025. 10.25608/6VM2-2Q46. 19. World Organization of Family Doctors. International Classification of Primary Care. https://www.globalfamilydoctor.com/site/DefaultSite/filesystem/documents/Groups/WICC/International%20Classification%20of%20Primary%20Care%20Dec16.pdf . Accessed 6 Nov 2025. 20. Sciensano. Healthdata.be. https://healthdata.sciensano.be/fr/%C3%A1-propos-healthdatabe . Accessed 6 Nov 2025. 21. Moreels S, Rouvez F, Bossuyt N, De Schreye R, Bensemmane S. General Characteristics of the Sentinel General Practitioners (SGP) Network in Belgium: Report on 2024. Sciensano; 2025. p. 22. https://www.sciensano.be/nl/biblio/general-characteristics-sentinel-general-practitioners-sgp-network-belgium-report-2024 . 22. Universiteit Antwerpen. iCAREdata Public Dashboard. https://icare.uantwerpen.be/superset/dashboard/PublicDashboard/ . Accessed 6 Nov 2025. 23. Regulation ECDC. (EU) 2022/2371 of the European parliament and of the council of 23 November 2022 on serious cross-border threats to health and repealing Decision No 1082/2013/EU. 2022. https://eur-lex.europa.eu/eli/reg/2022/2371/oj/eng . 24. Centers for Disease Control and Prevention. Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. MMWR Recomm Rep. 2001;50(RR–13):1–35. [ PubMed ] [ Google Scholar ] 25. Adlhoch C, Sneiderman M, Martinuka O, et al. Spotlight influenza: The 2019/20 influenza season and the impact of COVID-19 on influenza surveillance in the WHO European Region. Eurosurveillance. 2021;26(40). 10.2807/1560-7917.ES.2021.26.40.2100077. [ DOI ] [ PMC free article ] [ PubMed ] 26. Adlhoch C, Mook P, Lamb F, et al. Very little influenza in the WHO European Region during the 2020/21 season, weeks 40 2020 to 8 2021. Eurosurveillance. 2021;26(11). 10.2807/1560-7917.ES.2021.26.11.2100221. [ DOI ] [ PMC free article ] [ PubMed ] 27. Sinnathamby MA, Meslé MMI, Mook P, Stolyarov K, The European Region Influenza Network Group Authorship, Pebody R. Impact of the COVID-19 Pandemic on Influenza Circulation During the 2020/21 and 2021/22 Seasons, in Europe. Influenza Other Respir Viruses. 2024;18(5):e13297. 10.1111/irv.13297. [ Google Scholar ] 28. Vega T, Lozano JE, Meerhoff T, et al. Influenza surveillance in Europe: establishing epidemic thresholds by the Moving Epidemic Method. Influenza Other Respir Viruses. 2013;7(4):546–58. 10.1111/j.1750-2659.2012.00422.x. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 29. Eurostat, Statistics Explained.https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Glossary:Urban_cluster#:~:text=Short%20definition%3A%20a%20cluster%20of,population%20of%205%20000%20inhabitants. Accessed 6 Nov 2025. 30. Sciensano. Bulletin hebdomadaire infections respiratoires aiguës. https://www.sciensano.be/fr/node/72870 . Accessed 6 Nov 2025. 31. Vaes B, Vos B, Foidart M, et al. Burden of COVID-19 on primary care in Belgium: a prospective nationwide observational study from March to August 2020. Arch Public Health. 2022;80(1):250. 10.1186/s13690-022-01003-0. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 32. Van Der Bij S, Khan N, Ten Veen P, De Bakker DH, Verheij RA. Improving the quality of EHR recording in primary care: a data quality feedback tool. J Am Med Inf Assoc. 2017;24(1):81–7. 10.1093/jamia/ocw054. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 33. Botsis T, Bassøe CF, Hartvigsen G. Sixteen years of ICPC use in Norwegian primary care: looking through the facts. BMC Med Inf Decis Mak. 2010;10(1):11. 10.1186/1472-6947-10-11. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 34. Yih WK, Cocoros NM, Crockett M, et al. Automated Influenza-like Illness Reporting—An Efficient Adjunct to Traditional Sentinel Surveillance. Public Health Rep. 2014;129(1):55–63. 10.1177/003335491412900109. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 35. Schweikardt C, Verheij RA, Donker GA, Coppieters Y. The historical development of the Dutch Sentinel General Practice Network from a paper-based into a digital primary care monitoring system. J Public Health. 2016;24(6):545–62. 10.1007/s10389-016-0753-4. [ Google Scholar ] 36. Drewe JA, Hoinville LJ, Cook AJC, Floyd T, Stärk KDC. Evaluation of animal and public health surveillance systems: a systematic review. Epidemiol Infect. 2012;140(4):575–90. 10.1017/S0950268811002160. [ DOI ] [ PubMed ] [ Google Scholar ] 37. World Health Organization. WHO Average Curves (Shinyapp). 2025. https://worldhealthorg.shinyapps.io/averagecurves/ . 38. Sinnathamby MA, Bourouphael T, Boateng J, et al. Setting thresholds to determine COVID-19 activity levels using the mean standard deviation (MSD) method, England, 2022–2024. Eurosurveillance. 2024;29(45). 10.2807/1560-7917.ES.2024.29.45.2400696. [ DOI ] [ PMC free article ] [ PubMed ] 39. Taherdoost H, Mohebi A. Using SMART Method for Multi-criteria Decision Making: Applications, Advantages, and Limitations. Arch Adv Eng Sci. 2024;2(4):190–7. 10.47852/bonviewAAES42022765. [ Google Scholar ] 40. Risawandi, Robbi Rahim. Study of the Simple Multi-Attribute Rating Technique For Decision Support. IJSRST. 2016;2(6):491–4. 41. Sciensano, GP infection Barometer. https://www.sciensano.be/en/projects/gp-infection-barometer . Accessed 6 Nov 2025. 42. Fei Z, Ryeznik Y, Sverdlov A, Tan CW, Wong WK. An overview of healthcare data analytics with applications to the COVID-19 pandemic. IEEE Trans Big Data Published online 2021:1–1. 10.1109/TBDATA.2021.3103458. 43. Hang CN, Yu PD, Chen S, Tan CW, Chen G. MEGA: Machine Learning-Enhanced Graph Analytics for Infodemic Risk Management. IEEE J Biomed Health Inf. 2023;27(12):6100–11. 10.1109/JBHI.2023.3314632. [ DOI ] [ PubMed ] [ Google Scholar ] 44. Hang CN, Tsai YZ, Yu PD, Chen J, Tan CW. Privacy-Enhancing Digital Contact Tracing with Machine Learning for Pandemic Response: A Comprehensive Review. Big Data Cogn Comput. 2023;7(2):108. 10.3390/bdcc7020108. [ Google Scholar ] 45. Gupta P, Maharaj T, Weiss M et al. Y Lai ed. Proactive Contact Tracing. PLOS Digit Health. 2023;2(3):e0000199 . 10.1371/journal.pdig.0000199. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 46. Rovetta A. Google trends in infodemiology: Methodological steps to avoid irreproducible results and invalid conclusions. Int J Med Inf. 2024;190:105563. 10.1016/j.ijmedinf.2024.105563. [ DOI ] [ PubMed ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Data Availability Statement No datasets were generated or analysed during the current study. Articles from Archives of Public Health are provided here courtesy of BMC ACTIONS View on publisher site PDF (1.1 MB) Cite Collections Permalink PERMALINK Copy RESOURCES Similar articles Cited by other articles Links to NCBI Databases Cite Copy Download .nbib .nbib Format: AMA APA MLA NLM Add to Collections Create a new collection Add to an existing collection Name your collection * Choose a collection Unable to load your collection due to an error Please try again Add Cancel Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top