AffectAI-Capture: A Reproducible Multimodal Protocol for Small-Group Meeting Research⋆
arXiv:2605.19794v1 [cs.HC] 19 May 2026
Meisam Jamshidi Seikavandi1,2 , Alice Modica1,3 , Anna Obara1,4 , Fabricio Batista Narcizo1,2 , Tanya Ignatenko1 , Ted Vucurevich1 , Jesper Bünsow Boldt1 , Paolo Burelli2 , and Andrew Burke Dittberner1 GN Advanced Science, GN Group, Ballerup, Denmark IT University of Copenhagen, brAIn lab, Copenhagen, Denmark 3 Copenhagen Business School, Copenhagen, Denmark 4 Aalborg University, Denmark [email protected] 1
2
Abstract. We present AffectAI-Capture, a protocol for collecting synchronized multimodal data in four-person meeting-like interactions, combining eye tracking, wearable physiology, close-talk and room audio, multi-view video, event logging, and structured self-report. Sessions use fixed task blocks grounded in established group-interaction paradigms, while acquisition and post-processing are organized around a single authoritative event timeline and standardized outputs. We describe the experimental rationale, synchronization philosophy, data organization, and practical trade-offs. Pilot-level validation of audio quality and video synchronization has been conducted using controlled bench tests; full protocol sessions with participants remain ongoing work. The contribution is a reproducible protocol architecture linking task design, instrumentation, timing provenance, and data packaging for affective, behavioral, and meeting-analytics research. Keywords: multimodal protocol · meeting corpus · affective computing · synchronization · eye tracking · wearable physiology · BIDS
1
Introduction
Understanding group interaction in realistic meetings requires more than recording speech and video. Researchers increasingly seek to study how attention, arousal, effort, coordination, and subjective experience unfold jointly across participants and over time, demanding multimodal datasets that combine rich sensing, interpretable task structure, auditable synchronization, and reusable data organization. Classic meeting corpora showed the importance of synchronized audio-visual capture and structured meeting scenarios for cross-session comparability [5,9]. ⋆
Accepted at the AI4Action Workshop, 19th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA 2026). Published by Springer in Communications in Computer and Information Science (CCIS).
2
M. J. Seikavandi et al.
More recent affective and physiological datasets demonstrated the value of eyerelated and wearable measurements for studying internal-state variation, but are rarely built around true small-group meeting interactions [13]. This leaves a gap between meeting corpora optimized for collaborative structure and affectivecomputing datasets optimized for participant-level sensing. AffectAI-Capture closes this gap by combining scenario-based small-group tasks with high-resolution multimodal sensing, explicit event-centered synchronization, and BIDS-oriented derived outputs. It builds on our prior work on gazebased emotion perception, multimodal affect modelling, and the interplay of eye tracking, personality, and temporal dynamics in face-to-face settings [20,19,21], which highlighted the need for richer interaction-centered datasets. The protocol serves as a reusable foundation for affective computing, meeting analytics, and human-centered AI research.
2
Design Goals
The protocol is guided by four design goals. 1. Cross-session comparability without abandoning meeting realism. Sessions are structured using fixed task blocks that reproduce distinct socialcognitive interaction regimes, while preserving a plausible overall meeting narrative. 2. Multimodal capture at both group and individual levels. The protocol combines room-level behavioral signals with participant-specific channels such as gaze, pupillometry, skin conductance, and heart rate. 3. Auditable temporal provenance. Synchronization is treated as a firstclass design problem. The protocol does not assume perfect device synchronization, but instead records sufficient timing anchors to estimate, validate, and document alignment. 4. Reproducible and reusable outputs. Vendor raw data are preserved, derived outputs are separated, and session outputs are structured in a BIDSoriented manner to support downstream analysis and long-term reuse [8,15,26].
3
Related Work
Classic meeting corpora demonstrated the value of instrumented multi-party data, including structured scenarios and close-talk audio for transcription, overlap handling, and speaker-specific analysis [5,9]. However, these corpora focused on speech, video, and interaction annotation with limited access to participantlevel measures such as gaze or physiology. Multimodal affect datasets have shown the value of combining physiology, video, and self-report, but their setups often fell outside genuine small-group collaborative interaction [13]. K-EmoCon [14] demonstrated the feasibility of combining wearable physiology and continuous annotation in naturalistic dyadic
AffectAI-Capture for Small-Group Meetings
3
conversation, and AFFEC [23] introduced a multimodal dataset for face-toface emotion communication integrating eye tracking, physiology, and video. Both bridge affective sensing and social interaction more closely than stimulusresponse paradigms, but neither targets multi-party meeting settings with structured task variation. AffectAI-Capture is designed at the intersection of these traditions.
4
Experimental Design
Session structure. The protocol targets four-participant sessions organized into five blocks: T0 (onboarding/warm-up), T1 (hidden-profile decision), T2 (negotiation), T3 (idea generation and selection), and T4 (public-goods microgame). The fixed order supports cross-session comparability while progressing through distinct demands: acclimatization, information pooling, strategic exchange, collaborative creativity, and cooperation under mixed incentives. Structured tasks are a scientific asset: they create repeated interaction contexts with known behavioral pressures, making multimodal measurements more interpretable [5]. Task-specific rationale. Each task block is included because it elicits a distinct and well-studied interaction regime. T1: Hidden-profile decision. This task targets information pooling under asymmetrically distributed knowledge. It is suitable for studying whether groups disclose and integrate unique information, how attention and speaking patterns shift when new evidence is introduced, and how collective decisions are shaped by discussion structure [24,11]. T2: Negotiation. Negotiation introduces strategic interaction, emotional signaling, concession dynamics, and interpersonal regulation. It is well suited for multimodal capture, as vocal, facial, physiological, and gaze-related signals may reflect tension, control, responsiveness, and the evolution of bargaining positions [25]. T3: Idea generation and selection. This task separates divergent and convergent collaboration. It allows examination of how ideas are generated, shared, compared, and evaluated, as well as how groups converge on a final choice while weighing priorities such as originality, feasibility, or overall appeal [17]. T4: Public-goods micro-game. This task introduces a controlled tension between private incentives and collective outcomes. It produces clear event boundaries for decision, reveal, and discussion, making it useful for event-locked multimodal analysis of cooperation, surprise, disappointment, and the negotiation of fairness and social norms [7,6]. Task timing. Timed and moderator-controlled phases are intentionally mixed. Timed phases ensure comparability; controlled briefing phases preserve flexibility.
4
M. J. Seikavandi et al. Table 1. Task and timing structure. Task Objective T0 T1
5
Timed phases
T2 T3
onboarding hidden-profile sus negotiation ideation
free talk: 300 s consen- reading: 75 s; discussion: 420 s; selection: 60 s
T4
public-goods game
negotiation: 480 s; settlement form: untimed generation: 180 s; board & discussion: 420 s; selection: 60 s contribution: 60 s; reveal: 60 s; discussion: 180 s
Multimodal Instrumentation
Primary devices. The stack includes 4 Tobii Pro Glasses 3 (eye tracking), 4 EmotiBit devices (physiology), 5–7 Jabra cameras, 5 DPA microphones (4 closetalk + 1 room/spare), optional Vicon motion capture, and 4 participant tablets with 1 shared display. Camera layout. The video subsystem balances participant coverage, tablelevel interaction visibility, and synchronization robustness. The setup uses seven room cameras: four desk-adjacent cameras providing participant-focused lateral views and three overview cameras capturing the room more globally. Most cameras are configured for 1920×1080 capture at 30 fps, with the wide-angle room camera subject to device-dependent effective resolution limits in practice. The layout is organized around a desk-centered spatial reference system. A permanent ChArUco board at the desk center defines the world origin, while desk-edge and participant-linked markers support calibration, participant localization, and later multimodal alignment. Four desk-adjacent cameras are physically mounted upside down to clear desk-edge obstructions and achieve downward-angled coverage of the participant’s face and upper body from a compact mounting position; the resulting image is corrected in software to preserve consistent downstream orientation. Capture settings prioritize temporal auditability. Cameras are recorded at a constant nominal frame rate, frame-level timing logs are preserved, and wallclockbased packet timestamps are preferred over device-local timestamps for crossdevice consistency. Audio is captured separately rather than multiplexed into camera streams to reduce USB bandwidth contention and simplify synchronization diagnostics. The protocol prioritizes complementary viewpoints over a single canonical camera, since occlusion, head turning, and shared attention can make single-view capture insufficient in small-group meetings. Modality complementarity. No single signal captures meeting dynamics adequately. Audio supports speech activity, overlap, diarization, and verbal content; video supports posture, gesture, and head orientation; eye tracking supports attentional allocation and interaction focus; physiological signals provide cautious proxies for arousal, effort, and autonomic activation; and
AffectAI-Capture for Small-Group Meetings
Fig. 1. Audio validation using headset microphones on Head and Torso Simulators (HATS).
5
Fig. 2. Pilot deployment of the AffectAICapture setup.
self-report provides subjective grounding. These are treated as complementary indicators rather than direct readouts of psychological states, especially for pupil and physiological signals confounded by illumination, motion, and task context [12,16]. Recent work on multitask affective modelling from physiological signals further underscores the value of combining wearable channels [22]. Eye tracking is also a theoretically motivated channel for studying social-emotional processing [20,19,21].
6
Self-Annotation Strategy
Participants provide subjective labels through brief in-task valence–arousal– dominance prompts and post-task questionnaires capturing overall affect and experience with the task and interaction. The rationale is twofold. First, self-report remains necessary when the goal includes subjective affect or perceived social experience, since behavioral and
6
M. J. Seikavandi et al.
physiological signals alone cannot fully determine internal states. Second, intask prompting allows labels to be linked to concrete phases and events rather than only retrospective summaries. At the same time, prompt burden must be controlled to avoid turning selfreport into the dominant task. Prompting is therefore event-contingent and phase-aware, with minimum spacing and condition-specific scheduling. The design follows the broader tradition of low-burden dimensional self-report, including valence–arousal–dominance style instruments and the Self-Assessment Manikin [3].
7
Synchronization
Synchronization as a protocol concern. In multimodal meeting studies, timing errors are not a minor implementation detail. They directly affect the interpretation of turn-taking, gaze-to-speaker relations, event-locked physiological responses, and self-report alignment. Accordingly, AffectAI-Capture treats synchronization as an explicit protocol concern rather than an invisible background service [1]. Authoritative event spine. The protocol is centered on one authoritative session-level events.tsv. This file serves as the main temporal and semantic backbone of the session, recording task, phase, participant, stream, and event details in a machine-readable form. The goal is not to claim that all modalities are perfectly synchronized at acquisition time, but to ensure that alignment assumptions can be reconstructed and inspected later. Redundant timing anchors. Synchronization relies on a hierarchy of anchors: LSL as the logical cross-device timebase, event logging at the protocol level, video frame logs and capture sidecars for camera timing, and modality-specific timing metadata when available. This redundancy is intentional: real acquisition systems fail in uneven ways, so the protocol records enough temporal evidence to validate and repair alignment post hoc rather than relying on a single fragile mechanism. This is consistent with LSL’s role as a unifying synchronization layer for heterogeneous recordings, while acknowledging that device-specific latencies require explicit validation rather than assumptions of perfect synchrony [10]. Empirical characterization of inter-device offsets and clock drift across full sessions is planned as part of ongoing validation work and will be reported alongside the first complete dataset release.
8
Data Organization
AffectAI-Capture adopts a BIDS-oriented organization to make multimodal recordings easier to inspect, reuse, and process across heterogeneous streams. The main rationale is that raw data, metadata, events, and derived outputs should be clearly separated so that later analyses can reconstruct provenance rather than depend on undocumented preprocessing decisions [8,26].
AffectAI-Capture for Small-Group Meetings
7
At the session level, outputs follow a BIDS-like hierarchy with study metadata, participant/session folders, a session-level events.tsv, modality-specific subfolders, and a separate sourcedata/ tree for vendor raw files. Raw data are preserved, while canonical and analysis-ready outputs are written separately. Derived products include modality-specific tables and media files, per-task run windows and event slices, normalized annotation outputs, and participant-signal mapping artifacts. This organization improves immediate usability while preserving the information needed for later re-analysis and audit. As stable BIDS extensions for eye tracking and wearable physiology are not yet finalized, the current organization follows BIDS naming and hierarchy conventions while defining local modality-specific sidecars informed by draft extensions BEP020 (eye tracking) and BEP029 (virtual and physical motion data). These local conventions are documented in the dataset-level metadata so that future alignment with ratified extensions can be traced.
9
Pilot Validation
The validation work reported here consists of controlled bench tests for audio and video subsystems. No full protocol sessions with participants have yet been conducted; end-to-end validation under the complete protocol is part of ongoing work. Audio validation. An audio quality test was conducted using four DPA d:fine CORE 4066 close-talk microphones connected to an RME Fireface 802 and recorded in REAPER. The goals were to verify gain staging, low noise floor, and minimal cross-talk behavior. The validation included playback of gray noise from a fixed source position, allowing measurement of signal-to-noise ratio across channels. To simulate realistic participant conditions, two Head and Torso Simulators (HATS) were used (Fig. 1). One HATS emitted either white noise or recorded speech, while the second wore a DPA microphone and was placed sequentially at different seating positions; the procedure was repeated for all microphones. Gain staging was set with conservative headroom, targeting approximately −12 dBFS peaks during normal speech. Quantitative post-hoc analysis of SNR, noise floor variation, and recommended gain settings remains part of ongoing QC work. Video synchronization. A multi-camera synchronization test used five Jabra cameras in the intended room configuration. The design principle was to record frame-level timing anchors and LSL-linked progress information during capture, then validate alignment offline using synchronized grid visualization across simultaneous views. This test was performed without participants present; the broader pilot deployment of the physical setup is illustrated in Fig. 2. QC philosophy. The broader QC strategy includes pre-flight checks for stream and device availability, fail-loud process supervision, session summaries and manifests, explicit reporting of missing streams and timing issues, and preservation of synchronization sidecars for later auditing. The aim is not to hide operational
8
M. J. Seikavandi et al.
imperfections, but to document them in a way that preserves the scientific usability of the dataset [1].
10
Applicability to Sign Language and AssistiveTechnology Research
Although designed for spoken meetings, several architectural choices transfer to sign language and assistive-technology research. The multi-view camera rig and ChArUco reference frame are suited to capturing the signing space, since sign languages exploit the space in front of the body for referential and syntactic purposes [18]. The synchronization infrastructure and BIDS-oriented organization address challenges equally pressing in sign language translation (SLT) datasets, where temporal alignment between video, annotation, and auxiliary signals is critical [4]. The gaze instrumentation is relevant because eye gaze plays grammatical and turn-taking roles in signed communication, marking topic, focus, and turn transitions [2]. These connections suggest AffectAI-Capture could serve as infrastructure for multimodal sign language and assistive interaction research.
11
Limitations
The protocol makes several deliberate trade-offs. Structured tasks improve comparability, but do not capture the full diversity of real-world meetings. Physiological and pupil-related signals are informative but indirect and must be interpreted with care. True temporal alignment across heterogeneous devices is never guaranteed by software alone, which is why redundant timing evidence is retained. Commodity camera stacks offer practical flexibility but can introduce platform- and bandwidth-specific instability, especially under USB-related hardware constraints. Some integrations also remain dependent on vendor tooling or device-specific constraints. As noted above, the validation reported here is limited to controlled bench tests of individual subsystems; full end-to-end sessions with participants have not yet been conducted. Empirical characterization of synchronization accuracy, including measured inter-device offsets and drift, will accompany the first complete dataset release. These are not incidental weaknesses; they are part of the real design space of multimodal meeting science, and the contribution of the protocol is to make such constraints explicit and manageable.
12
Conclusion
AffectAI-Capture is a reproducible protocol for multimodal small-group meeting research, integrating structured task design, participant- and room-level sensing, synchronization-aware event modeling, and BIDS-oriented data organization. It provides a practical foundation for studying group affect, coordination, attention, and collaborative behavior, extending earlier work on gaze dynamics
AffectAI-Capture for Small-Group Meetings
9
and emotion modelling [20,19,21] into richer meeting settings. The architectural choices also transfer to sign language and assistive-technology research, where synchronized multi-view capture, gaze instrumentation, and provenance-aware data organization address open infrastructure needs. Acknowledgments. This work was funded by GN Advanced Science, GN Group. Meisam Jamshidi Seikavandi received additional funding from Innovation Fund Denmark under the industrial postdoc programme. Disclosure of Interests. Meisam Jamshidi Seikavandi, Alice Modica, Anna Obara, Fabricio Batista Narcizo, Tanya Ignatenko, Ted Vucurevich, Jesper Bünsow Boldt, and Andrew Burke Dittberner are employed by or affiliated with GN Advanced Science, GN Group, which funded this research. Meisam Jamshidi Seikavandi received funding from Innovation Fund Denmark as an industrial postdoc. Paolo Burelli has no competing interests to declare.
References 1. Anguera, X., Bozonnet, S., Evans, N., Fredouille, C., Friedland, G., Vinyals, O.: Speaker diarization: A review of recent research. IEEE Transactions on Audio, Speech, and Language Processing 20(2), 356–370 (2012). https://doi.org/10.1109/ TASL.2011.2125954 2. Baker, C.: Regulators and turn-taking in American Sign Language discourse. In: Friedman, L.A. (ed.) On the Other Hand: New Perspectives on American Sign Language, pp. 215–236. Academic Press, New York (1977) 3. Bradley, M.M., Lang, P.J.: Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25(1), 49–59 (1994). https://doi.org/10.1016/0005-7916(94)90063-9 4. Camgöz, N.C., Hadfield, S., Koller, O., Ney, H., Bowden, R.: Neural sign language translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7784–7793 (2018) 5. Carletta, J., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Karaiskos, V., Kraaij, W., Kronenthal, M., Lathoud, G., Lincoln, M., Lisowska, A., McCowan, I., Post, W., Reidsma, D., Wellner, P.: The AMI meeting corpus: A preannouncement. In: Renals, S., Bengio, S. (eds.) Machine Learning for Multimodal Interaction. Lecture Notes in Computer Science, vol. 3869, pp. 28–39. Springer, Berlin, Heidelberg (2006). https://doi.org/10.1007/11677482_3 6. Chaudhuri, A.: Sustaining cooperation in laboratory public goods experiments: A selective survey of the literature. Experimental Economics 14(1), 47–83 (2011). https://doi.org/10.1007/s10683-010-9257-1 7. Fehr, E., Gächter, S.: Cooperation and punishment in public goods experiments. American Economic Review 90(4), 980–994 (2000). https://doi.org/10.1257/aer. 90.4.980 8. Gorgolewski, K.J., Auer, T., Calhoun, V.D., Craddock, R.C., Das, S., Duff, E.P., Flandin, G., Ghosh, S., Glatard, T., Halchenko, Y.O., Handwerker, D.A., Hanke, M., Keator, D., Li, X., Michael, Z., Maumet, C., Nichols, B.N., Nichols, T.E., Pellman, J., Poline, J.B., Rokem, A., Schaefer, G., Sochat, V., Triplett, W., Turner, J.A., Varoquaux, G., Poldrack, R.A.: The brain imaging data structure, a format
10
M. J. Seikavandi et al.
for organizing and describing outputs of neuroimaging experiments. Scientific Data 3, 160044 (2016). https://doi.org/10.1038/sdata.2016.44 9. Janin, A., Baron, D., Edwards, J., Ellis, D., Gelbart, D., Morgan, N., Peskin, B., Pfau, T., Shriberg, E., Stolcke, A., Wooters, C.: The ICSI meeting corpus. In: 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP ’03). vol. 1, pp. I–364–I–367. IEEE (2003). https://doi.org/ 10.1109/ICASSP.2003.1198793, available at https://www.ee.columbia.edu/~dpwe/ pubs/icassp03-janin.pdf 10. Kothe, C., Shirazi, S.Y., Stenner, T., Medine, D., Boulay, C., Grivich, M.I., Artoni, F., Mullen, T., Delorme, A., Makeig, S.: The lab streaming layer for synchronized multimodal recording. Imaging Neuroscience 3, IMAG.a.136 (2025). https://doi. org/10.1162/IMAG.a.136 11. Lu, L., Yuan, Y.C., McLeod, P.L.: Twenty-five years of hidden profiles in group decision making: A meta-analysis. Personality and Social Psychology Review 16(1), 54–75 (2012). https://doi.org/10.1177/1088868311417243 12. Mathôt, S.: Pupillometry: Psychology, physiology, and function. Journal of Cognition 1(1), 16 (2018). https://doi.org/10.5334/joc.18 13. Miranda-Correa, J.A., Abadi, M.K., Sebe, N., Patras, I.: AMIGOS: A dataset for affect, personality and mood research on individuals and groups. IEEE Transactions on Affective Computing 12(2), 479–493 (2021). https://doi.org/10.1109/ TAFFC.2018.2884461 14. Park, C.Y., Cha, N., Kang, S., Kim, A., Khandoker, A.H., Hadjileontiadis, L., Oh, A., Jeong, Y., Lee, U.: K-emocon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations. Scientific Data 7(1), 293 (2020) 15. Pernet, C.R., Appelhoff, S., Gorgolewski, K.J., Flandin, G., Phillips, C., Delorme, A., Oostenveld, R.: EEG-BIDS, an extension to the brain imaging data structure for electroencephalography. Scientific Data 6, 103 (2019). https://doi.org/10.1038/ s41597-019-0104-8 16. Posada-Quintero, H.F., Chon, K.H.: Innovations in electrodermal activity data collection and signal processing: A systematic review. Sensors 20(2), 479 (2020). https://doi.org/10.3390/s20020479 17. Rietzschel, E.F., Nijstad, B.A., Stroebe, W.: The selection of creative ideas after individual idea generation: Choosing between creativity and impact. British journal of psychology 101(1), 47–68 (2010) 18. Sandler, W., Lillo-Martin, D.: Sign Language and Linguistic Universals. Cambridge University Press (2006) 19. Seikavandi, M.J., Barrett, M., Burelli, P.: Modeling face emotion perception from naturalistic face viewing: Insights from fixational events and gaze strategies. In: Recent Advances in Deep Learning Applications: New Techniques and Practical Examples. Taylor & Francis (2024) 20. Seikavandi, M.J., Barrett, M.J.: Gaze reveals emotion perception: Insights from modelling naturalistic face viewing. In: 2023 International Conference on Machine Learning and Applications (ICMLA). pp. 2022–2025. IEEE (2023) 21. Seikavandi, M.J., Fimland, J., Barrett, M., Burelli, P.: Modelling emotions in faceto-face setting: The interplay of eye-tracking, personality, and temporal dynamics. arXiv preprint arXiv:2503.16532 (2025) 22. Seikavandi, M.J., Narcizo, F.B., Vucurevich, T., Dittberner, A.B., Burelli, P.: MuMTAffect: A multimodal multitask affective framework for personality and emotion recognition from physiological signals. In: Proceedings of the 3rd International Workshop on Multimodal and Responsible Affective Computing. pp. 100–108 (2025)
AffectAI-Capture for Small-Group Meetings
11
23. Sekiavandi, M.J., Dixen, L., Fimland, J., Desu, S.K., Zserai, A.B., Lee, Y.S., Barrett, M., Burelli, P.: Advancing face-to-face emotion communication: A multimodal dataset (affec). arXiv preprint arXiv:2504.18969 (2025) 24. Stasser, G., Titus, W.: Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology 48(6), 1467–1478 (1985). https://doi.org/10.1037/0022-3514.48.6.1467 25. Van Kleef, G.A., De Dreu, C.K.W., Manstead, A.S.R.: The interpersonal effects of emotions in negotiations: A motivated information processing approach. Journal of Personality and Social Psychology 87(4), 510–528 (2004). https://doi.org/10. 1037/0022-3514.87.4.510 26. Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J.J., et al.: The FAIR guiding principles for scientific data management and stewardship. Scientific Data 3, 160018 (2016). https://doi.org/10.1038/sdata.2016.18