arXiv:2604.09426v1 [cs.HC] 10 Apr 2026
Three Modalities, Two Design Probes, One Prototype, and No Vision: Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool Sanchita S. Kamath
Aziz Zeidieh
Venkatesh Potluri
School of Information Sciences University of Illinois Urbana-Champaign Champaign, Illinois, USA [email protected]
Informatics University of Illinois Urbana-Champaign Champaign, Illinois, USA [email protected]
School of Information University of Michigan Ann-Harbour, Michigan, USA [email protected]
Sile O’Modhrain
Kenneth Perry
JooYoung Seo
School of Information University of Michigan Ann-Harbour, Michigan, USA [email protected]
American Printing House for the Blind Louisville, Kentucky, USA [email protected]
School of Information Sciences University of Illinois Urbana-Champaign Champaign, Illinois, USA [email protected]
Figure 1: Iterative Prototyping based on the EBCD Framework
ABSTRACT Three-dimensional (3D) data visualizations, such as surface plots, are vital in STEM fields from biomedical imaging to meteorology and spectroscopy, yet remain largely inaccessible to blind and low-vision (BLV) people. To address this gap, we conducted an Experience-Based Co-Design (EBCD) with BLV co-designers with expertise in non-visual data representations to create an accessible, multi-modal, web-native visualization tool. Using a multi-phase
Please use nonacm option or ACM Engage class to enable CC licenses This work is licensed under a Creative Commons Attribution 4.0 International License. CHI ’26, April 13–17, 2026, Barcelona, Spain © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2278-3/2026/04 https://doi.org/10.1145/3772318.3791272
co-design methodology, our team of five BLV and one non-BLV researcher(s) participated in two iterative sessions, comparing a low-fidelity tactile probe with a high-fidelity digital prototype. This process produced a prototype with empirically grounded features, including reference sonification, stereo and volumetric audio, and configurable buffer aggregation, which our BLV co-designers validated as improving analytic accuracy and learnability. In this study, we explicitly target core analytic tasks essential for non-visual 3D data exploration: 3D orientation, landmark and peak finding, comparing local maxima versus global trends, gradient tracing, and identifying occluded or partially hidden features. Our work offers accessibility researchers and developers a co-design protocol for translating tactile knowledge to digital interfaces, concrete design
CHI ’26, April 13–17, 2026, Barcelona, Spain
guidance for future systems, and opportunities to extend accessible 3D visualization into embodied data environments. Three-dimensional data visualizations, such as surface plots, are vital in STEM fields from biomedical imaging to meteorology and spectroscopy, yet remain largely inaccessible to blind and lowvision people. To address this gap, we conducted an ExperienceBased Co-Design with BLV co-designers with expertise in nonvisual data representations to create an accessible, multi-modal, web-native visualization tool. Using a multi-phase co-design methodology, our team of five BLV and one non-BLV researcher(s) participated in two iterative sessions, comparing a low-fidelity tactile probe with a high-fidelity digital prototype. This process produced a prototype with empirically grounded features, including reference sonification, stereo and volumetric audio, and configurable buffer aggregation, which our co-designers validated as improving analytic accuracy and learnability. In this study, we explicitly target core analytic tasks essential for non-visual 3D data exploration: 3D orientation, landmark and peak finding, comparing local maxima versus global trends, gradient tracing, and identifying occluded or partially hidden features. Our work offers accessibility researchers and developers a co-design protocol for translating tactile knowledge to digital interfaces, concrete design guidance for future systems, and opportunities to extend accessible 3D visualization into embodied data environments.
CCS CONCEPTS • Human-centered computing → Accessibility design and evaluation methods; Accessibility technologies; Accessibility systems and tools; Empirical studies in accessibility.
KEYWORDS accessibility, three-dimensional data visualizations, AI for data visualization, multi-modality, embodied interaction ACM Reference Format: Sanchita S. Kamath, Aziz Zeidieh, Venkatesh Potluri, Sile O’Modhrain, Kenneth Perry, and JooYoung Seo. 2026. Three Modalities, Two Design Probes, One Prototype, and No Vision: Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13– 17, 2026, Barcelona, Spain. ACM, New York, NY, USA, 23 pages. https: //doi.org/10.1145/3772318.3791272
1
INTRODUCTION
While there have been notable advances in producing accessible data visualizations [2–4], blind and low-vision (BLV) people remain largely excluded from engaging with three-dimensional (3D) data visualizations that represent data using three axes (X, Y, and Z) to create spatial models, such as 3D scatter plots or volumetric rendering. Most accessibility efforts to date, have focused on audiotactile graphs [4, 42] and two-dimensional representations [13]. The lack of accessible 3D data visualization tools is particularly pressing given that 3D visualizations are central to fields such as meteorology [81], biomedical imaging [108], VUV spectroscopy [51], geoscience [78], and computational fluid dynamics [34]; when inaccessible, they not only limit analytic rigor but also alienate BLV individuals from meaningful participation in knowledge production across these
Kamath et al.
domains. Our target data type in this work is continuous heightfield surfaces—scalar fields defined over a two-dimensional domain that produce smooth, continuous 3D cloth-like surfaces commonly found in scientific surface plots such as in VUV spectroscopy viewable in Figure 4. Focusing on these continuous surfaces allows us to support analytic tasks that depend on stable topological continuity. While accessibility for 2D charts has progressed through tactile graphics [20, 53, 72] and multi-modal, web-based interactions [39, 89, 104], the spatial and complex nature of 3D data representations such as surface and point plots continue to pose persistent barriers to non-visual access. Yet without access to these visualizations, analysts risk missing essential insights in spatially complex datasets [11, 21]. The leap from two-dimensional (2D) charts to interactive three-dimensional (3D) data reveals critical gaps in current accessibility research. First, most studies still concentrate on 2D charts, leaving a gap in methods for interactively exploring 3D surfaces and point clouds [49, 50]. It is also important to clarify the scope of this claim within established visualization design practice, which generally cautions against projecting three-dimensional data onto two-dimensional displays due to perceptual distortions and the potential for misleading interpretations. In many cases, twodimensional encodings such as color gradients, glyph size, or textual summaries can successfully convey three-variable correlations, and users are often able to infer relationships from such encodings. However, when analytic tasks rely on understanding continuous surface structure or spatial topology, as is common with heightfieldstyle representations that remain prevalent in practice, collapsing a three-dimensional topology into a flat, single-channel representation can remove critical structural relationships. For BLV users in particular, TTS-only descriptions or reduced two-dimensional encodings may force the reconstruction of three-dimensional spatial logic mentally across modalities, increasing cognitive load and weakening the formation of coherent spatial mental models. Rather than asserting that three-dimensional representations are universally superior, this work focuses on accessibility challenges that arise in contexts where three-dimensional surface representations are already in use and where preserving topological structure is central to the analytic task. Second, to the best of our knowledge, no studies have applied evidence-based methodologies for transferring the spatial knowledge gained from tactile prototypes into web-based digital environments for the specific challenge of 3D plots [94, 96]. Third, prior systems often neglect the pedagogical scaffolding required for learners’ transition from initial orientation to independent analytic reasoning [68]. Fourth, few studies provide fine-grained evidence on how specific non-visual modalities affect performance in 3D data visualization contexts; existing work is often limited to object recognition applications [75, 84] or focuses on embodied interaction without exploring multi-modality in depth [8]. This paper introduces a methodology (inspired by the ExperienceBased Co-Design (EBCD) framework [82]) and two design probes which were used to prompt the non-visual exploration of 3D scientific data. Our work was motivated by a foundational goal: to ensure that accessibility tools are developed with and by the BLV community, not just for them. Instead of positioning BLV individuals as participants in user studies framed by sighted researchers, rather than as integral collaborators from a project’s inception [65]; this
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
study aims to flip that order. By adopting an EBCD approach [82], our work while proposed by a sighted researcher, is fundamentally designed with and inspired by BLV people. Unlike conventional co-design, which often focuses on collaborative ideation and prototyping, EBCD emphasizes grounding design in lived experiences, typically through in-depth narrative accounts, storytelling, and shared reflection. This distinction was crucial for our project: rather than relying solely on structured design workshops, we began by gathering experiential narratives from BLV co-designers who have expert knowledge in data visualization, which then directly informed the framing of our design challenges and priorities. In this way, EBCD shaped not just how we collaborated, but also what we considered meaningful design outcomes. The methodology ensured that our technical explorations were continually anchored in co-designers’ everyday practices and values, making the resulting tools not only functional but also genuinely usable and empowering. Within this process, our produced tactile probe (Section 4.1) served as the shared ground-truth reference against which all subsequent digital translation was calibrated, enabling consistent interpretation of spatial features across modalities. The significance of this work, therefore, is threefold: (1) it addresses the challenge of producing a transferable co-design protocol for translating tactile knowledge into effective digital interactions, (2) demonstrates a collaborative and inclusive research methodology - by adopting the EBCD Framework for HCI research on non-visual data access and representation, and (3) provides an evidence-based model for developing effective non-visual tools for 3D data exploration. By making surface visualizations accessible, we expand opportunities for BLV users to engage directly with spatial data, promoting fuller participation in STEM disciplines [46]. Our developed prototype, as evaluated by our ethnographic approach with expert data visualization researchers, helps demonstrate a successful translation of embodied, tactile knowledge into a well-known and well-adopted digital medium, providing non-visual features that were validated by our expert collaborators as improving their ability to orient, analyze, and accurately interpret 3D data. This gives our system a better position to help BLV people in producing spatial mental models of three-dimensional data, as compared to two-dimensional representations of the same data. The uniqueness of this work lies in demonstrating how one can transform a tactile learning experience into an effective digital counterpart. Such an approach equips educators with concrete workflows for teaching 3D spatial reasoning and provides developers with empirically grounded guidance for extending access to embodied or three-dimensional environments. While our system can handle the input of closed-volume or molecular surface data, those structures currently appear as disjoint surfaces due to our coordinate grid handling algorithm, and thus do not behave equivalently; producing non-intuitive interaction, making any transfer only partial and not assumed. Hence, we shall not focus on them in this paper. The research problem at hand, therefore, is to make 3D data visualizations accessible and learnable for BLV people. Following are the research questions that direct this study: RQ1. How can spatial knowledge gained through tactile, lowfidelity prototypes be effectively transferred into high-fidelity digital experiences that support core 3D analytic tasks?
CHI ’26, April 13–17, 2026, Barcelona, Spain
RQ2. How can we (re)design specific non-visual features to empower BLV users’ ability to orient, analyze and accurately interpret 3D surfaces with confidence? RQ3. What mediums and interaction tools can most effectively translate a web-based digital prototype of three-dimensional data visualizations into embodied forms of engagement that preserve users’ ability to execute the full set of analytic tasks? This scholarly work makes four key contributions - (1) it offers a co-design protocol that integrates tactile and digital probes, refined iteratively across two sessions with BLV collaborators; (2) it delivers a (re)designed high-fidelity prototype incorporating empirically grounded features such as reference sonification, stereo and volumetric audio cues, and configurable buffer aggregation; (3) it provides expert opinion on the impact of these features on analytic accuracy and user learnability in 3D data contexts; and (4) it contributes design ideas for the future extension to embodied 3D data visualization environments.
2
RELATED WORK
To contextualize our contribution, we review the EBCD theoretical framework and map how it can be employed in HCI practices and list three areas of related work: adaptations of two-dimensional visualizations for BLV audiences, immersive and embodied approaches to data interaction, and emerging efforts in three-dimensional visualization. Together, these streams highlight both the progress made and the critical gaps that remain, underscoring the need for methods that can center lived experience in tackling new accessibility challenges.
2.1
Theoretical Framework: Experience-Based Co-Design
EBCD, originally developed in health services, is built upon iterative cycles of understanding lived experience, prioritizing issues, and (re)designing services with users and stakeholders [28, 31, 70]. According to EBCD practitioners, key stages include gathering staff and patient experiences, producing a “trigger film” based on emotional touch-points, holding co-design events, forming working groups, and celebrating progress [30]. Although participatory and co-design approaches are common in HCI [87], EBCD is distinguished by its systematic elevation of lived experience as central design material; not merely an input but the motivating, framing resource [27, 69]. To the best of our knowledge, no prior HCI studies have employed EBCD for accessible 3D data visualization; we adopt and adapt it (Table 1) to structure cycles of eliciting BLV collaborators’ tactile and narrative experiences, prioritizing their analytical challenges, and collaboratively redesigning non-visual 3D representations in a meaningful, grounded way. Compared to design studies, which often emphasize artifact critique, visual aesthetics, and designer reflection [41], and more conventional co-design methods, which may solicit stakeholder input during ideation [48], EBCD offers a distinct advantage: it treats lived experience as the core design material [71]. In design studies, the focus may lie on how form and function evolve through prototyping and critique, but without grounding in deep personal narrative [32]. In typical co-design, participants might guide requirements or suggest features, but their stories are often abstracted [74]. EBCD, by
CHI ’26, April 13–17, 2026, Barcelona, Spain
Kamath et al.
Canonical EBCD Stage
Our Adaptation and Rationale
Setting up the project
We established our co-design team, clarified roles, and oriented BLV researchers to both the tactile and digital phases, ensuring a shared understanding and equal power in design. In our context, “staff” maps to non-BLV researcher; we collected their perspectives on constraints and design possibilities, balancing with BLV lived experience. We ran low-fidelity and high-fidelity design probe sessions and narrative interviews to collect first-person accounts of exploring 3D surfaces non-visually. Instead of a video film, we used the tactile probe and semi-structured questioning to surface emotional and perceptual “touch-points” in how BLV collaborators explore surface features. We brought BLV researchers together in our workshops, using prior experiential data to align on priorities and co-create interaction features through Session 1 and 2. We iteratively developed our high-fidelity prototype (drawing from insights gathered during Session 1) and tested the final version with requested features (e.g., sonification and buffer aggregation) during Session 2. We reserved the second half of Session 2 for reflection, where participants compared tactile and digital experiences, surfaced insights, and affirmed shared design ownership. We integrated the co-designed features into a web-native 3D visualization tool usable for 3D data analysis by BLV users. We performed iterative evaluation across both sessions where BLV collaborators used both the tactile and web-based versions, reflecting on accuracy, learnability, and fidelity.
Gather staff / practitioner experiences Gather patient / user experiences Trigger film (or tangible trigger) First co-design event Co-design working groups Celebration / review event Implementation Evaluation
Table 1: Mapping from canonical EBCD stages (per standard health-services literature) to our adaptations in this work.
contrast, privileges participants’ emotional “touch-points”: their real, embodied encounters, and uses these to prioritize problems, spark design ideas (e.g., via trigger films/probes), and evaluate solutions. For our work, this is especially powerful: the way BLV individuals navigate and make sense of tactile spatial data is not just a usability issue but a form of domain knowledge. By centering that experience, we ensure the resulting design is not only functional, but deeply aligned with how BLV users perceive, reason about, and interpret 3D data — leading to a more meaningful, effective, and empowering tool.
2.2
Accessible Visualization in Two Dimensions
Accessibility research has historically concentrated on two- dimensional charts, tactile graphics, and physical data representations. Research on haptic data visualization highlights how touch-based interfaces can support data access for BLV individuals by translating charts, maps, and diagrams into tactile forms [13, 20, 35, 53, 72]. Physical visualization and data sculptures have been proposed as educational tools to represent abstract data tangibly, enabling learners to grasp complex relationships through embodied interaction [99]. Data physicalizations have been shown to support effective analysis of elevation data when compared with digital or VR alternatives, emphasizing the promise of haptic perception in learning spatial structures [36]. Complementing tactile approaches, sonification-based systems have demonstrated effectiveness in conveying data through auditory mappings. Zhao and Vande Moere [106] pioneered interactive sonification techniques for georeferenced data, while more recent work has integrated natural sound representations [39] and spatial audio cues [93] to enhance data comprehension. Electrotactile feedback systems [46] and refreshable tactile displays combined with conversational agents [37, 38, 83] have expanded multimodal possibilities for chart comprehension. Haptic techniques such as Fan et al. [24] and broader sonification frameworks [6] have further enriched non-visual data access strategies. Screen-reader-based approaches, including tools like VoxLens [91], MAIDR [89], ChartA11y [104], AudioFunctions.Web [5], and Umwelt [110], have established frameworks for keyboard-driven navigation and semantic description of visualizations. Studies of screen-reader users’ experiences [90] and rich screen-reader interaction designs [109] have illuminated the
diverse needs and preferences of BLV individuals when accessing visual data. Broader surveys have documented the state of sonification visualization integration [23], established design frameworks for accessible visualization [22, 67], and articulated the need for sociotechnical perspectives that center disability justice [40, 62]. Work on semantic levels of natural language description [63] and identifying opportunities for accessible visualization [55] has further shaped our understanding of how to design for diverse sensory experiences. Additional research has explored sensory substitution [16], non-visual visualization design [15], audio-based infographics [38], and declarative sonification grammars [54]. Systematic reviews of sonification in assistive systems [64] and efforts to reach broader audiences through visualization [56] provide comprehensive overviews of the field’s trajectory. These works collectively demonstrate the potential of tactile, auditory, and multi-modal approaches but remain largely constrained to 2D or static contexts. Three lessons from this body of work directly informed our design approach. First, multi-modal redundancy: the principle that information should be encoded through multiple sensory channels simultaneously, guided our integration of sonification, spatial audio, and textual feedback to ensure robust comprehension regardless of individual perceptual preferences [23, 39, 46, 83, 89, 110]. Second, pedagogical scaffolding: as demonstrated in systems that progressively introduce complexity [89, 104, 109], shaped our inclusion of overview-then-detail (whole to part strategy) features such as multi-perspective autoplay and jump-to-peak navigation. Third, progressive disclosure: the idea that systems should allow users to control the granularity of information they encounter [55, 63, 91], directly motivated our implementation of a switch between point and surface modes which enables users to toggle between aggregated “squares” and point-by-point detail and the configurable buffer mechanism. However these 2D methods fail for 3D analytics due to a collapsed topology (squashing three variable correlations into two channel representations), and the lack of occlusion or volumetric traversal. By grounding our 3D visualization prototype in established principles from 2D accessibility research (our multi-modal infrastructure - is inspired by Seo et al. [89] and Seo et al. [88]), we extend their applicability to three-variable correlations and share their core commitment to user autonomy and multi-modal access.
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
Finally, A11yShape [105] is one of the first pieces of literature that allows BLV people to use AI-assist to create 3D models, however it focuses on generative shape modeling through code-based construction (OpenSCAD with GPT-4o) and cross-representation highlighting of discrete geometric components, rather than the systematic, real-time exploration of continuous three-dimensional data visualizations. While A11yShape enables users to create and verify geometric forms through AI-generated descriptions and hierarchical abstractions, our work addresses the distinct challenge of interpreting pre-existing scientific datasets through features like reference sonification, spatial audio, configurable buffer aggregation, and multi-perspective auto-play that support orientation, comparison, and statistical reasoning within complex data landscapes common to STEM disciplines.
2.3
Immersive and Embodied Approaches
Alongside tactile approaches, a growing body of research explores immersive and embodied interaction as alternative avenues for data access. Extended reality (XR) and embodied interaction research has highlighted the potential of immersive audio and haptic feedback for BLV users [60, 61], with prototypes demonstrating embodied navigation and object labelling in 3D environments. Embodied interaction has been used to support visualization literacy, allowing learners to construct and manipulate visualizations through tangible devices [47], embodied allegories for gesture design [14], and aesthetics-driven embodied installations [59]. These studies emphasize the importance of bodily engagement and spatial presence in data interaction. In parallel, immersive environments such as VirtualDesk integrate embodied gestures and virtual workspaces to enhance comfort and analytic performance in 3D settings [100], while systematic reviews point to immersive analytics as a rapidly expanding field that seeks to unify spatial, embodied, and multisensory techniques [43]. Although immersive approaches expand the possibilities of nonvisual access, they have primarily emphasized proof-of-concept VR or AR experiences for sighted users, rather than evidence-based accessibility for BLV audiences. This gap motivated our discussion of how embodied principles: particularly proprioception, physical movement, and spatial presence; might inform the design of more grounded, accessible interfaces. While our prototype is web-based and keyboard-driven, we deliberately engaged our co-designers in speculative discussions about translating the system into embodied forms (Section 6.4). These discussions, documented in Session 2, generated critical design insights about advanced haptics, mixed reality synchronization, and alternative physical input devices that both validate embodied approaches as a natural extension of our work and provide concrete pathways for future development beyond screen-based interaction.
2.4
Gaps in Accessible 3D Data Visualization
Despite progress in both tactile and immersive domains, the accessibility of analytic three-dimensional data visualizations remains underexplored. Existing research has developed methods for immersive exploration of 3D scatterplots, addressing challenges like occlusion and density perception through techniques such as Scaptics and Highlight-Planes [76], or by investigating movement types
CHI ’26, April 13–17, 2026, Barcelona, Spain
and spatial abilities in VR-based scatterplot analysis [92]. Collaborative environments such as Shared Surfaces and Spaces highlight how groups interact with 2D and 3D visualizations in immersive contexts [57], while studies of device-input combinations stress the impact of controllers and display modalities on analytic accuracy [103]. Other work has explored VR systems such as VR-Viz [86], Uncharted Territory’s design heuristics for VR data visualization [1], and web-based 3D visualization frameworks such as the Digital Library of Mathematical Functions [101] and dbslice’s infinite canvas approach [77]. These systems demonstrate the promise of 3D data interaction across scientific and educational domains but remain largely inaccessible to BLV users. Indeed, even when accessibility is considered, efforts are typically limited to object recognition tasks rather than analytic reasoning with continuous surfaces. Although physical tactile systems and AR offer powerful means of perceiving spatial information, they often do not scale well for analytic scenarios (Table 2). For instance, refreshable tactile displays remain expensive and low-resolution for continuous surfaces (as shown in dynamic tactile marker systems like Suzuki et al. [97]), while AR solutions for visual impairment frequently depend on co-located headsets or high-end devices, limiting deployment and web integration [17]. Shape-changing interfaces (e.g., pin-array displays) also struggle with cost, actuator density, and inability to dynamically render arbitrary, high-resolution scientific surfaces [58]. Our web-native, multimodal approach avoids these scalability bottlenecks by offering dynamically configurable, low-cost, highresolution access to 3D surfaces via standard web technologies, enabling widespread and flexible analytic use without requiring any specialized hardware. Taken together, prior work underscores three trends: (1) accessibility efforts have focused on audiotactile and 2D modalities, (2) immersive and embodied interaction research demonstrates promising but sighted-centered prototypes, and (3) existing 3D visualization systems emphasize technical innovation over inclusive design. To the best of our knowledge, no prior research has applied evidence-based co-design to make three-dimensional data visualizations accessible for BLV users by transferring knowledge from low-fidelity tactile models into high-fidelity, web-native multimodal experiences. This study addresses this gap by integrating tactile and digital probes, empirically evaluating non-visual features such as stereo sonification and volumetric cues, and offering concrete guidance for inclusive 3D visualization workflows.
3
CO-DESIGN PROCESS AND CONSIDERATIONS
Having positioned Experience-Based Co-Design (EBCD) as our methodological foundation in Section 2.2, we now detail how we operationalized this framework through concrete design activities. Our adaptation of EBCD’s canonical stages (Table 1) unfolded across two major phases. First, we gathered lived experiences through lowfidelity (Phase 2, Thrust 1) and high-fidelity probes (Phase 1 and Phase 2, Thrust 1). Then, in Phase 2 (Thrust 3), we held codesign sessions where BLV collaborators prioritized interaction gaps and validated proposed features. Finally, we iteratively implemented solutions grounded in these experiential “touch-points”. This section documents the structure, composition, and progression
CHI ’26, April 13–17, 2026, Barcelona, Spain
Kamath et al.
Criterion
2D Web-based Encodings
Physical Tactile Displays
Augmented Reality (AR)
Occlusion Handling
Cannot multi-modally represent depth or hidden structures and remain limited to surface projection for 3D surfaces [26]
Users must physically explore raised lines or pins, so occluded areas require manual navigation [107]
Persistence
Encoding via color, glyphs, or text is not appropriate for nonvisual access due to high cognitive load [98]
Physical relief or pins remain until manually reset, but reconfiguration may be limited to pre-existing states [45]
Resolution for Continuous Surfaces
Limited by the granularity of encoding (e.g., color bins, glyph size) [95] Users can access remotely via personal devices such as laptops and mobile phones [89, 104] Very low: software solutions are inexpensive
Limited by pin density or embossing precision; highly detailed surfaces are hard to render affordably [10] User must have the tactile display physically present but do not require co-location with others Often high: refreshable tactile displays or embossers are expensive to build or buy [73] Many tactile displays operate using specialized drivers and often require extensive troubleshooting
Can simulate 3D depth with audio or visual layering, but hidden or overlapping features may still be difficult to disambiguate for non-visual users [66] AR augmentations persist only while the device is held and tracked; tracking errors or device movement can break spatial consistency [33] High potential: AR can render very fine-grained spatial structure digitally, but non-visual sensing is currently limited [18] AR devices (e.g., headset or phone) must be co-located with the physical space or digital model [29] High: AR headsets or development of specialized AR for accessibility can be costly [7] Moderate to low: AR systems are often standalone applications and not deeply web-integrated [80]
Co-location Requirement Cost Web Integration
Excellent: software and web-native systems are naturally integrated [89]
Table 2: Descriptive comparison of modalities for non-visual or multimodal 3D data access.
of our collaborative inquiry, while Section 5 presents the empirical outcomes of these sessions, including the five design goals that emerged from co-designers’ narratives and the validated prototype features that these goals motivated. Our design methodology was rooted in collaborative autoethnography, an approach where researchers engage in a “relational journey” to systematically study their own collective experiences [52]. This methodology positioned the design team, particularly its BLV members, as co-participants whose data visualization and research expertise was the very locus of the research. Instead of acting as objective instruments, their role was to engage in dialogic reflexivity, where insights are co-constructed through shared narratives and mutual sense-making. We did not merely design for BLV users; rather, as a mixed-ability team, we systematically documented, analyzed, and responded to our own lived experiences interacting with 3D data visualizations. Our reflections on moments of confusion, discovery, and frustration served as the primary data driving the iterative development of the prototype.
3.1
Methodological Justification and Design Rationale
We deliberately chose this immersive, autoethnographic [19] EBCD approach [82] over conducting traditional user studies because our research goal was fundamentally generative, not evaluative. A traditional user study framework is optimized for assessing a predesigned artifact against defined metrics; a process that would have required us to first build a prototype based on our own, likely flawed, assumptions about non-visual interaction. Such a process inevitably centers the researcher’s perspective and positions participants as subjects who merely react to a proposed solution. Given that our primary challenge was to translate a deeply embodied, tactile form of 3D data visualization interpretation and discovery into an entirely different digital and auditory modality, a traditional study would have failed to capture the nuanced, iterative process of discovery required. It could tell us if a feature worked, but not why it felt intuitive or disorienting. In contrast, we adopted an Experience-Based Co-Design (EBCD) approach [82], which framed our collaborators as co-designers with lived expertise rather than research subjects. As detailed in Section 2.1, EBCD treats lived experience as core design material, not merely background context. This methodological orientation shaped our process in several ways. First, our collaborators were engaged from the outset in identifying which challenges mattered and why, with their stories and reflections becoming the starting points
for framing design problems grounded in lived realities rather than researcher assumptions. Second, their expertise guided evaluation of our tools not only in terms of usability, but also whether they contributed to autonomy, empowerment, and meaningful access, positioning them as co-judges of value rather than testers of functionality. As established in Section 2.1, EBCD distinguishes our work from conventional co-design [25, 79] through its epistemological commitment: while co-design often begins from researcher-defined problem spaces and treats participation as a means of generating better solutions, EBCD foregrounds experiential grounding where design work emerges only after lived experiences are surfaced, shared, and co-interpreted, thus reconfiguring whose knowledge counts in defining the problem space. In our project, this meant resisting the urge to “translate” existing visualization practices into accessible formats and instead reimagining what visualization could mean when defined through BLV collaborators’ perspectives. This shift was essential: it allowed our project to move beyond making visualizations technically accessible, toward exploring how visualization practices themselves might be transformed when coshaped by BLV experience. Section 5 provides concrete examples of how co-designer narratives about orientation confusion and spatial reasoning directly motivated implemented features such as stereo panning, volumetric audio, and reference sonification, demonstrating EBCD’s capacity to transform lived experience into grounded design decisions. The following subsections detail the composition of the team whose narratives drove these design trajectories, the specific analytic tasks they addressed, and the structured process through which their expertise was systematically elicited and translated into prototype features.
3.2
Design Team
Our research was conducted by an interdisciplinary, mixed-ability design team whose collective expertise and lived experiences were foundational to the co-design process. The team integrated seasoned academic scholarship, practical software and hardware engineering, and a rich spectrum of visual abilities, ensuring that our technical explorations were deeply grounded in the nuances of non-visual interaction. The core academic group was composed of graduate researchers and faculty who drove the project’s theoretical framing and iterative development. Sanchita, a sighted Graduate Teaching and Research Assistant with a minor in Data Analytics, led the technical implementation and interface development. She leveraged extensive prior experience creating 3D visualizations in MATLAB
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
and a strong understanding of statistical relationships to support the system’s analytical foundations. Aziz, a Graduate Teaching and Research Assistant who is blind, provided continuous, real-time feedback from the perspective of a user with a foundational understanding of data visualization. Aziz has experience reading and creating common plot types: including bar, line, box, scatter plots, histograms, and heatmaps; using visual, tactile, and sonified approaches, and developed his 3D visualization knowledge through participation in this project. They were joined by Venkatesh, an Assistant Professor of Information with seven years of research experience in data visualization and programming tool accessibility, who contributed intermediate-to-advanced expertise shaped by his doctoral training and ongoing research in the field. Sile, a Professor of Information with 25 years of teaching experience, helped shape the cognitive aspects of this project. She regularly analyzes data in her own research and has contributed to preparing instructional materials through initiatives such as the Summer Institute. JooYoung, an Assistant Professor of Information Sciences, brought over eight years of expert knowledge in creating accessible and multimodal data visualization tools. His work includes the development of R packages for accessible visualization and deep expertise in statistical concepts and research methods. This faculty trio guided two graduate researchers who were central to the prototype’s creation. To bridge academic theory with real-world application, the team was expanded to include Ken, a Senior Software Engineer and R&D specialist in assistive technology. Ken contributed extensive expertise spanning both software and hardware, including the design of tactile graphics displays, paper tactile graphics, braille-to-digital pipelines, audio–tactile multimodal visualization systems, and data exploration workflows using tools such as pandas, Seaborn, and Matplotlib. He has also developed 3D tactile displays and collaborated with 3D printing workflows to represent STEM data, including exploratory work with haptic and tactile pin-array systems. He provided a critical perspective on the practical needs of blind and low-vision learners beyond the laboratory. Crucially, the team’s strength was rooted in its diversity of lived experience. Five of the six members (JooYoung Seo, Sile O’Modhrain, Venkatesh Potluri, Aziz Zeidieh, and Ken Perry) are blind or have low vision, with visual acuities represented in Table 3. This rich tapestry of perspectives was the engine of our collaborative autoethnography, positioning BLV researchers as the primary research actors and ensuring that the design process was driven not by assumptions, but by a nuanced, collective understanding of non-visual data interaction.
3.3
Design Process
Our inquiry was methodologically structured through a sequence of interconnected stages, inspired by the EBCD Framework [82]. We adapted its healthcare-centric model by translating the “patient pathway” into the user’s journey of exploring a 3D visualization. Phase 1, (May to June 2025), initiated the project with a foundational ideation design cycle involving BLV and non-BLV collaborators (JooYoung, Aziz, and Sanchita), which produced an initial high-fidelity digital probe. The research then progressed to Phase 2 (July to August 2025), organized into three distinct thrusts to systematically refine this probe. Thrust 1 (first week of July 2025)
CHI ’26, April 13–17, 2026, Barcelona, Spain
Figure 2: Experience-Based Co-Design: Research Timeline involved the fabrication of a physical, low-fidelity probe to serve as an embodied reference. In Thrust 2 (July 2025), insights from the end of Phase 1 were integrated into the high-fidelity digital probe. Finally, Thrust 3 which expanded the collaborative team to include BLV co-designers (Sile, Venkatesh, Ken) and comprised two iterative co-design sessions lasting 90 minutes each: Session 1 (12 Aug 2025) centered on a comparative analysis of the experience provided by two probes to identify interaction gaps (mostly situated around knowledge transfer from physical to digital prototype), while Session 2 (22 Aug 2025) was dedicated to testing and validating new features that emerged from this analysis, culminating in future directions that could provide the platform to observe embodied learning and interactions of users. 3.3.1 Phase 1: Foundational Ideation and High-Fidelity Probe Development. Phase 1 was grounded in a mixed-ability co-design methodology, drawing upon the interdependence framework [9] to structure a sustained, two month long collaboration between two blind researchers (JooYoung, Aziz) and one non-BLV researcher (Sanchita) for creating an initial version of our high-fidelity design probe. In this initial phase, JooYoung and Aziz provided crucial design insight rooted in their lived experience, while Sanchita led the implementation and interface development. The process was highly iterative, characterized by daily collaboration with Aziz and weekly sessions with JooYoung, where each meeting involved direct evaluation of the evolving probe, the proposal of new features, and the identification of interaction pain points. This continuous feedback loop allowed Sanchita to implement refinements in real-time. A central activity of this phase was the conceptual reframing of conventional visual affordances, such as grid overlays and highlighting, into non-visual strategies like wireframe traversal, axis-specific navigation, and the synchronized alignment of auditory, visual,
CHI ’26, April 13–17, 2026, Barcelona, Spain
Kamath et al.
Table 3: Team members’ visual acuity, onset of impairment, age, and roles. Name Sanchita Aziz Venkatesh Sile Ken JooYoung
Visual Acuity Not BLV; corrected to 20/20 with lenses No vision in left eye; 20/2000 in right eye Totally Blind Some light perception Totally Blind No light or shape perception in right eye, light and shape perception in left eye
Age of Onset Not applicable Congenital Congenital Congenital 20 11
and textual feedback. This foundational work culminated in the production of the initial high-fidelity digital design probe, which served as the empirically grounded starting point for the expanded research activities in Phase 2 - Thrust 2 and Thrust 3.
3.3.2 Phase 2: Evidence-Based (Re)Design and Knowledge Transfer. The second phase of the research was designed to rigorously test, critique, and reconstruct the initial probe. The overarching goal of Phase 2 was to convert the high-fidelity probe into a functional prototype capable of facilitating effective knowledge transfer from a physical, embodied medium to a digital, multi-modal environment. This work was organized into three distinct but interconnected thrusts, with Sanchita, Aziz, and JooYoung leading the first two. Thrust 1 of Phase 2 was the creation of a physical, low-fidelity probe. This probe functioned not merely as a demonstrative tool but as a crucial epistemological anchor; it established a “ground truth” of embodied understanding, providing a shared sensory baseline against which the digital experience could be systematically evaluated. Thrust 2 of Phase 2 focused on implementing initial insights from our foundational work. Prior to engaging the broader group, Sanchita integrated the key design recommendations from JooYoung, which had been identified in Phase 1, into the digital probe. This act of translation ensured methodological continuity and established a starting point for the expanded sessions that was already grounded in expert user knowledge. Thrust 3, the most extensive component of Phase 2, was the execution of two iterative co-design sessions. (1) Session 1: This session was structured as a direct comparative analysis. JooYoung, Sile, Venkatesh, and Ken were invited to first explore the physical probe from Thrust 1, using manual and tactile inspection to construct a direct understanding of the data’s form by feeling its gradients, slopes, and peaks. Subsequently, they transitioned to the high-fidelity digital probe from Thrust 2, using keyboard-only navigation and sonification to explore the same data structure. Session 1 concluded with a group discussion and a silent brainwriting exercise whose prompts were designed by Sanchita (readable at Appendix Section A.1). This exercise produced a clear consensus on four major features required to bridge this experiential gap which we will talk about in Section 5. (2) Interim Development: During the interim period between Session 1 and 2, Aziz and Sanchita (posing themselves as co-developers - with Sanchita leading implementation and Aziz providing iterative feedback) implemented three of the four prioritized features.
Age 24 28 32 59 55 35
Occupation Graduate Teaching and Research Assistant Graduate Teaching and Research Assistant Assistant Professor of Information Professor of Information Senior Software Engineer and R&D Specialist Assistant Professor of Information Sciences
(3) Session 2: This session was bifurcated into two distinct parts. The first part was dedicated to the systematic testing and validation of three of the four prioritized features implemented after Session 1 (Section 5.3.1). The second part of the session shifted to future-oriented ideation, where collaborators engaged in a brainwriting exercise to brainstorm how the embodied knowledge from the physical prototype could be transferred into more immersive digital environments (Section 5.3.2). Session 2 concluded with a collaborative brainwriting and discussion exercise to generate speculative ideas that could generate an embodied 3D visualization learning experience. The prompts were designed by Sanchita (readable at Appendix Section A.3). 3.3.3 Analytic Tasks and Co-Design Prompts. In these Sessions, we explicitly targeted core analytic tasks essential for nonvisual 3D data exploration, tasks that are foundational to scientific reasoning across STEM disciplines. Our co-designers were prompted to engage with the following challenges during both probe exploration sessions and the brainwriting exercises: • 3D Orientation: Establishing and maintaining awareness of position and direction within the three-dimensional coordinate space; prompts included “How do you know where you are in the plot?” and “What cues help you understand the axes?” • Landmark and Peak Finding: Identifying local maxima, minima, and other salient features; prompts included “How would you locate the highest point?” and “Can you find where the data changes most dramatically?” • Comparing Local Maxima versus Global Trends: Distinguishing between isolated peaks and broader patterns; prompts included “How do you differentiate a single peak from an overall rising trend?” and “What would help you compare multiple high points?” • Gradient Tracing: Following the direction and steepness of slopes across the surface; prompts included “Can you follow the path of steepest ascent?” and “How would you trace a ridge or valley?” • Identifying Occluded or Partially Hidden Features: Detecting regions obscured by overlapping data or structural complexity; prompts included “How do you explore areas that might be hidden from one perspective?” and “What strategies help you uncover features that aren’t immediately apparent such as those with similar Y values?” These tasks were not presented as formal test scenarios but rather as conversational prompts woven into our co-design sessions, encouraging co-designers to articulate their exploration strategies, confusions, and moments of insight. The brainwriting exercises at
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
the end of Session 1 and Session 2 (Appendix Sections A.1 and A.3) further formalized these prompts, asking co-designers to reflect on which features supported or hindered each task. The complete set of task-based questions used throughout both sessions is documented in Appendix Sections A.2 and A.4, connecting the five analytic tasks directly to the specific prompts and evaluation criteria employed during probe exploration and feature validation. The brainwriting exercise in Session 1 produced a clear consensus on four major features, but the path from co-designer narratives to these implemented features merits closer examination. Two such examples illustrate this translation process. During probe exploration, JooYoung articulated a fundamental analytic limitation: the difficulty of comparing different regions within a single plot. Where the physical probe allowed simultaneous tactile comparison using multiple fingers, the digital interface forced sequential, memory-dependent exploration. Their narrative directly motivated the configurable buffer feature, implemented during the interim development period, which enables users to save a region’s sonification and compare it against their current focus point. Similarly, when co-designers reported spatial disorientation, Ken and Sile proposed adding reference sonifications. These narratives coalesced into the fixed origin sonification with replay mechanism, validated in Session 2. These trajectories exemplify how EBCD centered co-designers’ experiential knowledge to produce features addressing real, lived challenges rather than theoretically anticipated needs.
4 DESIGN PROBES 4.1 Low-Fidelity Design Probe The low-fidelity probe (created in Phase 2, Thrust 1 and employed in Phase 2, Thrust 3, Session 1) was constructed on an 11.7 x 16.5 inch, 3/16-inch thick polystyrene foam sheet, which formed a sturdy base and provided the co-designers with a clear orientation for the plot. We affixed thicker plastic tubes (from 19.69-inch balloon sticks - marked with tape to represent the ticks) along two perpendicular edges to represent the Z-axis (longer edge) and X-axis (shorter edge). A small peg holder was attached at the origin for the co-designers to insert a separate tube for the Y-axis, perpendicular to the base. To represent the raw data (5 points), an array of peg holders was attached across the foam board; into these, we inserted thinner plastic tubes cut to varying lengths corresponding to the height of each data point. Small balls of heavy-duty aluminum foil were placed on the tip of each tube to create a distinct tactile representation of the points. All elements, including the axes, were labeled using a 6-dot braille label maker for full accessibility. Finally, a 38 x 40 inch muslin cotton cloth was draped over the points to simulate the continuous mesh of the surface plot. Developed by Sanchita, this tactile artifact translated the abstract, visual topography of a 3D surface plot into a tangible landscape of peaks, valleys, and slopes (viewable in Figure 3).
CHI ’26, April 13–17, 2026, Barcelona, Spain
Figure 3: Low-Fidelity Prototype
4.2
High-Fidelity Design Probe
Our high-fidelity design probe (created in Phase 2, Thrust 2 and employed in Phase 2 Thrust 3, Session 1 and 2) is a browserbased, multi-modal platform that evolved directly from our autoethnographic inquiry. To facilitate this, we made a deliberate methodological choice to build a modular system architecture. This architectural separation was the technical manifestation of our EBCD methodology; it was chosen specifically because it allowed us to be immediately responsive to collaborator feedback. By isolating concerns, we could rapidly prototype and integrate new, complex features—often within a single layer—without destabilizing the entire application. This enabled us to translate the insights from our co-design sessions directly into functional system logic. The system is composed of five strictly hierarchical layers that communicate through an event-driven architecture using centralized event constants (defined in EventConstants.js): • Data Layer (PlotData.js): Manages data ingestion, validation, normalization, and statistical analysis. • Engine Layer (VisualizationEngine.js): Handles WebGL rendering for points, surfaces, and contours using custom GLSL shaders. • Accessibility Layer (NavigationController.js): Contains all non-visual interaction logic coordinating eleven controllers for sonification, text feedback, navigation, autoplay, jump-to-peak, drag-select, and review modes. • UI Layer (UIController.js): Manages visual interface components including menus, color schemes, axes rendering, and AI chat. • Application Layer (app.js): Serves as global coordinator, initializing dependencies and orchestrating cross-layer communication. Layer communication follows strict architectural constraints: components access only their own layer and lower layers through dependency injection, while cross-layer events propagate through a custom event bus. Representative event payloads include displaymode-changed (notifying mode switches), drag-select-select ion-confirmed (communicating buffer boundaries), and autopl ay-state-changed (coordinating playback status). This architecture enabled us to implement complex features like buffer aggregation and jump-to-peak functionality within isolated controllers while maintaining system stability across iterative (re)designs.
CHI ’26, April 13–17, 2026, Barcelona, Spain
Sonification Engine and Web Audio Implementation. The sonification engine (SonificationController.js) is implemented as a custom Web Audio API synthesis pipeline operating at the browser’s native sample rate (typically 48kHz). Audio synthesis occurs entirely client-side without external libraries, using the AudioContext interface to create real-time parameter-mapped oscillators. The core signal path begins with oscillator generation (createOscillator()), routes through envelope shaping (createGain()), applies spatial positioning via binaural panning (createStereoPanner()), and adds depth perception through convolution reverb (createConvolver()). Timbre encoding employs three oscillator waveforms: sine for smooth low-frequency regions, triangle for mid-range transitions, and square for high-frequency accents; selected dynamically based on normalized X-axis position. The binaural setup maps X-coordinates to stereo pan values (-1.0 for leftmost to +1.0 for rightmost), creating horizontal spatial positioning that requires headphone playback for accurate localization. Depth perception (Z-axis) is rendered through algorithmic reverb: we generate a 2.2-second impulse response buffer with exponential decay (power 3.5) and apply variable wet/dry mixing (20-95% wet) scaled by normalized depth values. Pre-delay ranges from 10ms (foreground) to 90ms (background), with lowpass filtering (6500Hz to 2000Hz cutoff) simulating distance-dependent damping. Round-trip latency from keystroke to audio output measures approximately 15-30ms on modern browsers, below the threshold for perceived interaction delay. Visual Highlighting System and WebGL Implementation. The visual highlighting system (HighlightController.js) provides high-contrast visual feedback for navigation and selection through direct manipulation of WebGL rendering buffers. The controller operates by maintaining state for three distinct highlighting contexts: single-point navigation (magenta highlight with 4x size multiplier), multi-selection regions (white highlighting for drag-selected areas), and cursor positioning (dynamic yellow/white coloring based on selection state). The core highlighting mechanism integrates with the Engine Layer’s VisualizationEngine.js through the createBuffers() pipeline: when a point or wireframe rectangle is marked for highlighting via setHighlightedPoint(index) or setHighlightedWireframeRectangle(index), the controller triggers complete buffer regeneration, during which the rendering engine queries isPointHighlighted(index) for each data point to determine if enhanced visual properties should be applied. For point-mode visualization, highlighted vertices receive modified RGBA color arrays (retrieved via getHighlightColor()) and scaled point sizes (computed through getHighlightSize Multiplier()) before being uploaded to GPU vertex buffers using WebGL’s gl.bufferData() with STATIC_DRAW usage pattern. In surface mode, the highlighting system employs separate WebGL buffer objects (highlightPosition, highlightColor, highlightIndices) to isolate highlighted wireframe rectangles from the base mesh geometry. During rendering, these buffers are drawn with aggressive depth handling; disabling depth testing (gl.disable(gl.DEPTH_TEST)), applying polygon offset (gl.polygonOffset(-1.0,-1.0)), and forcing opaque rendering through the simpler line shader pipeline to ensure highlighted elements render atop the base visualization regardless of y-ordering.
Kamath et al.
This architecture maintains strict separation between highlighting logic (Accessibility Layer) and rendering implementation (Engine Layer), enabling the system to support simultaneous multi-selection highlighting, cursor feedback, and navigation focus without buffer conflicts. 4.2.1 Initial System: Before Session 1. The initial version of the probe was developed during and after the foundational work in Phase 1 (read more in Section 5.1), with features aimed at enhancing user autonomy. To address the tedium of manual traversal, we implemented two key exploration strategies within the Accessibility Layer: • Multi-Perspective Auto-Play (AutoPlayController.js): This controller implements our “whole-to-part” navigation strategy by sequencing navigation commands and triggering corresponding sonification events, allowing users to receive auditory “screenshots” of the data without exhaustive manual traversal. • Jump-to-Peak Function (JumpController.js): This feature statistically analyzes the data (view paragraph below for a detailed explanation) to identify the most significant peaks and troughs, enabling rapid navigation between a surface plot’s most salient features with a simple J key press. In addition to the core visualization features, Sanchita implemented a multi-threaded AI chat assistant that leverages Gemini 2.5 Pro to support interactive data exploration. This feature integrates directly with the Visualization Engine, enabling users to query the underlying dataset or trends visible in the 3D plot. However, this feature was not extensively tested during Session 1 or Session 2, as our primary research focus remained on independent, non-visual data exploration and the transfer of knowledge between low-fidelity and high-fidelity design probes. A detailed description of the AI chat assistant implementation is provided in Appendix B. Jump-to-Peak Statistical Algorithm. The Jump-to-Peak feature (JumpController.js) identifies salient features through statistical outlier detection operating on wireframe rectangles. The algorithm: (1) extracts avgY values from all rectangles; (2) sorts by avgY to select top 20 positive and bottom 20 negative candidates; (3) removes spatial duplicates by (x,z,avgY) key; (4) calculates dataset statistics (mean, min, max, range); (5) computes 20% threshold = (max − min) × 0.2; (6) tests proximity: if negatives are within threshold but positives are not, selects top 10 positive peaks only; if positives are within threshold but negatives are not, selects bottom 10 negative peaks only; otherwise selects top 10 of each; (7) sorts final peaks with positives descending and negatives ascending; (8) returns array of peak objects with sign and coordinates. When activated by pressing J in surface mode, the controller saves the current navigation position, detects peaks using this algorithm, and sequentially jumps through the identified features. Each jump updates the grid position to align with the peak rectangle’s center coordinates, highlights the rectangle visually, and plays a distinctive frequency-swept tone (400Hz base for negative peaks, 800Hz for positive peaks). Pressing J again cycles to the next peak, while Escape exits the mode and restores the saved navigation position.
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
4.2.2 System (Re)Design: After Session 1. Based on Sile, Venkatesh, Ken and JooYoung’s interaction with both low-fidelity and highfidelity design probes, we postulated some design features to be implemented (read more in-depth in Section 5.2). To address the critical need for stable orientation, we enhanced the SonificationController.js within the Accessibility Layer to provide a fixed reference sound for the origin (0,0,0) by pressing the 0 key and a replay sonification mechanism of current point by pressing . key. The reference sound acts as a constant, while the replay function, bound to the . key, allowing a user to re-listen to their current location’s sonification without moving. This same controller is also responsible for encoding position and depth cues through stereo panning and volumetric spatial audio. As a user navigates along the X-axis, the sound pans between the left and right stereo channels; movement along the Z-axis is conveyed through changes in volume, creating an intuitive auditory analog for proximity and distance. Finally, to balance analytic detail with cognitive load; another key finding from Session 1 (read in depth in Section 5.2), we developed DragSelectController.js to implement configurable buffer playback as detailed below. The implementation spanned multiple layers: the Accessibility Layer managed selection logic and queried the Data Layer to retrieve buffered points or rectangles, calculated arithmetic means for aggregated mode, and organized sequential mode playback. The HighlightController.js (Accessibility Layer) coordinated with UIController.js (UI Layer) to render visual feedback through VisualizationEngine.js (Engine Layer). This empowers users to fluidly shift between high-level summaries and detailed inspections of specific areas, streamlining the discovery process. Users could upload their own data as well, and all described multi-modal features are extended to the rendered plot.
Buffer Aggregation and Playback Modes. Buffer selection (DragSe lectController.js) implements a three-step rectangular region selection: users press D to enter selection mode, then press Enter to begin anchoring the selection start point at their current navigation position. They navigate using arrow keys to define the opposite corner (visual highlighting also follows the expanding selection area), then press Enter again to confirm and store the region. The controller queries the Data Layer for all points or wireframe rectangles with coordinates falling within the defined bounds, storing them in currentSelection as a buffer array that persists until a new selection is made. Aggregation applies arithmetic mean calculation: for each selected item, we normalize coordinates (handling both point objects with x/y/z properties and rectangle objects with centerX/avgY/centerZ properties), sum all valid Y-values, and divide by count to produce averageYValue. The aggregated playback mode (playable though the G-key) sonifies this single mean value for 1.0 second duration. In sequential (non-aggregated) mode, the buffer is organized by Z-index rows (front-to-back) with left-toright X-coordinate sorting within each row. Each item plays for 0.3 seconds with 125ms inter-item silence, creating a systematic auditory sweep. Users toggle between modes by pressing G: the first press activates sequential playback, the second switches to aggregated, and the third returns to sequential.
CHI ’26, April 13–17, 2026, Barcelona, Spain
5
FINDINGS
Our findings are drawn directly from the auto-ethnographic data collected during the end of Phase 1 and the two co-design sessions in Phase 2, Thrust 3.
5.1
End of Phase 1: Allowing User Autonomy over Data Exploration
Conversations and iterative testing with JooYoung revealed that empowering users with autonomy was the most critical factor for successful data exploration, consistent with prior findings in accessible visualization that emphasize user-controlled interaction and progressive disclosure [55, 63]. They expressed that a purely manual, point-by-point traversal of the data was not only tedious but also made it difficult to build a cohesive mental model of the 3D plot, echoing challenges reported in sequential non-visual exploration of complex data [90]. Their feedback consistently highlighted the need for tools that would allow them to control the scope and focus of their exploration, balancing high-level overviews with the ability to investigate specific features efficiently. This led to the development of our first two design goals aimed at granting users greater control over their interaction with the data. DG1. The prototype should provide multi-perspective autoplay to deliver auditory “snapshots” of the dataset: Codesigners emphasized the need for rapid overviews of complex plots; auto-play from multiple perspectives was valued as a way to capture structural patterns across all three axes without exhaustive manual traversal, aligning with overviewfirst strategies in accessible visualization [89, 109]. DG2. The prototype should include a jump-to-peak function to quickly locate salient features: JooYoung highlighted the value of a dedicated command for automatically cycling between local maximum peak and trough values, enabling efficient interrogation of surface plot trends without exhaustive searching, similar to landmark-based navigation approaches [91, 110].
5.2
Phase 2, Session 1: Surfacing the Experiential Gap
Session 1 directly addressed RQ1 by revealing a fundamental disconnect between the tactile prototype’s physical understanding and the digital version’s abstract experience, a gap previously observed when translating tactile or embodied representations into screenbased systems [36, 99]. Co-designers immediately surfaced multiple orientation and comparison challenges that mapped directly onto five essential analytic tasks: 3D orientation, gradient tracing, comparing local maxima versus global trends, landmark and peak finding, and identifying occluded features. Table 4 systematizes the prompts we used to elicit these experiences, the analytic tasks they addressed, the observed issues that emerged from co-designer narratives, and the design changes we subsequently implemented to support each task. A primary point of friction was the orientation of the coordinate system, which we anticipated from JooYoung’s experience in Phase 1. However, we wanted to gather insight from Sile, Venkatesh and Ken before we implemented features, so that we could learn from
CHI ’26, April 13–17, 2026, Barcelona, Spain
Kamath et al.
Prompt
Analytic Task
Observed Issue
Design Change Implemented
“How do you know where you are in the plot? What cues help you understand the axes?”
3D Orientation
Axis convention misalignment: co-designers expected Y-axis on board, Z-axis pointing out; digital used Y-vertical, Z-depth. Ken: “I would expect the y-axis to be on the board and the zaxis pointing out.” Venkatesh: “X-axis would be along the horizontal edge of the board, Y-axis would be along the vertical edge... and Z-axis would be the one popping.”
Fixed reference sonification at origin (0,0,0) via 0-key; replay mechanism via .-key to re-hear current position (DG3).
“Can you follow the path of steepest ascent? How would you trace a ridge or valley?”
Gradient Tracing
Lack of spatial encoding for horizontal and depth dimensions. Venkatesh: “it would be nice to have it pan from left to right, and decrease the volume as it’s going forward.”
Stereo panning for X-axis (left-right), volumetric audio (volume + reverb) for Z-axis depth (DG4).
“What would help you compare multiple high points?”
Comparing Local Maxima vs. Global Trends
Difficulty maintaining mental representation of previous positions during sequential exploration; physical probe allowed simultaneous tactile comparison.
Configurable buffer: drag-select to save region, toggle between aggregated (mean) and sequential (point-by-point) playback via G-key (DG5).
“How would you explore areas that might be hidden from one perspective?”
Identifying Occluded Features
Inter-plot comparison not supported; need to differentiate datasets by categorical variable. Ken: “What if I wanted to compare two sets of data... Is there any way to overlay them and switch between the two surfaces?” Sile: “if you gave the sound for Georgia a square wave instead of a sine wave.”
Identified for future implementation: dualdataset overlay with distinct timbres and keybinding toggle.
Table 4: Session 1 evaluation: prompts, analytic tasks, observed issues, and implemented design changes.
their expertise and follow the EBCD framework guidelines with all our collaborators [70]. Notably, this issue surfaced even before the co-designers interacted with our digital prototype; participants expressed expectations about axis orientation while experiencing the physical tactile probe, consistent with prior work on embodied spatial expectations [47]. For example, Ken remarked, “Yeah, I find that kind of odd, too. I would expect the y-axis to be on the board and the z-axis pointing out.” Venkatesh shared a similar mental model: “...what I was expecting is X-axis would be along the horizontal edge of the board, Y-axis would be along the vertical edge... and Z-axis would be the one popping.” These comments referred specifically to their expectations for the physical device, but the same mental model carried over when they later interacted with the digital prototype, whose coordinate system followed standard visualization conventions (vertical Y, horizontal X, depth Z). This mismatch helped us identify the need to explicitly align coordinate orientations across both physical and digital components. The (re)design process was central to our methodology, representing a direct, evidence-based response to the core challenges identified during our co-design sessions. The impetus for the (re)design was the experiential gap that collaborators surfaced when comparing the low-fidelity tactile probe with the initial high-fidelity digital version, reflecting similar tensions observed in prior tactileto-digital translation efforts [98]. While the tactile model afforded a rich, tactile understanding, the digital probe felt “abstract” and “disoriented”, necessitating a fundamental reconstruction. This redesign involved implementing empirically grounded features to bridge this gap, including a fixed reference sonification for orientation, stereo and volumetric audio to encode position [39, 93], and a configurable buffer playback to balance analytic detail with cognitive load. Conceptually, this (re)design process was our primary method for beginning to translate tactile knowledge into a web-based digital form. It signifies a shift from creating a static artifact to engaging in a dynamic, collaborative refinement aimed at enhancing users’ analytical agency and empowering them to independently interpret complex 3D data.
Venkatesh proposed using spatial audio, suggesting, “it would be nice to have it pan from left to right, and decrease the volume as it’s going forward.” This idea of mapping spatial dimensions to audio properties was a recurring theme and formed the basis for our efforts to incorporate stereo and volumetric spatial audio to encode position and depth cues. The concept of a stable reference point also emerged as a critical need. Ken suggested, “You could even mark a reference and then walk away from it on the surface”. Sile expanded on this, proposing a dedicated key binding: “You could have a reference just available on a key, where you would just ping it”. This exchange was the genesis of our goal to provide a reference point and replay mechanism to restore orientation during exploration. A significant limitation of the initial digital probe was the difficulty of comparing different points or regions within a single plot. This led to the idea of a “custom buffer”. As JooYoung proposed, “...it might be instrumental, if we have a, like, custom buffer, so that, say, for example, we can save one specific segment in specific buffers... And then you can hear that buffer... and you can also replay your currently focused segment”. This concept of selecting and saving a region for later comparison formed the basis for our goal to implement configurable buffer playback to balance analytic detail and cognitive load. Beyond intra-plot comparison, the discussion surfaced the need for inter-plot comparison. Ken raised a critical use case: “What if I wanted to. . . Compare two sets of data, same surface types... Is there any way to overlay them and. . . Switch between the two surfaces?” This question identified a fourth key feature: the ability to compare two datasets differentiated by a categorical variable (e.g., precipitation in Texas vs. Georgia). The group brainstormed solutions, with Sile suggesting the use of distinct timbres to differentiate the two plots: “if you gave the sound for Georgia a square wave instead of a sine wave”. This idea of layering two datasets in the same coordinate space and allowing a user to switch between them using a keybinding, with distinct auditory feedback for each, was identified as a crucial direction for future development, even
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
Figure 4: Prototype interface though it was not implemented before Session 2. Thus, our final three design goals were formulated: DG3. The prototype should provide a reference point and replay mechanism to restore orientation during exploration: Co-designers identified the fixed origin sonification and replay (.) as essential for regaining bearings after becoming disoriented in complex data spaces. DG4. The prototype should incorporate stereo and volumetric spatial audio to encode position and depth cues: Stereo panning and distance-based loudness was collectively decided upon as the most effective method to convey horizontal and vertical positioning within surface plots. DG5. The prototype should implement configurable buffer playback to balance analytic detail and cognitive load: The ability to toggle between aggregated and non-aggregated modes allowed co-designers to select their customized set of values to be replayed and compared; giving them the ability to adjust buffer density depending on task complexity and preference.
5.3
Phase 2, Session 2: Testing and Validating New Features
Session 2 served two functions: validating the three features implemented after Session 1 and brainstorming future directions for embodied interfaces. The validation phase systematically tested each feature through task-specific prompts, while the brainstorming phase generated speculative designs for translating the prototype beyond web-based interaction, in line with embodied and immersive accessibility research [43, 61]. 5.3.1 Part 1: Testing of features. This validation phase addressed RQ2 by systematically testing whether each implemented feature successfully supported its target analytic tasks. Each feature was evaluated through specific task-based prompts, following taskcentered evaluation practices in accessible visualization research [90]: orientation restoration for DG3, gradient tracing for DG4, and region comparison for DG5. Table 5 summarizes the test prompts, task
CHI ’26, April 13–17, 2026, Barcelona, Spain
performance results, and refinements for each validated feature. The reference sonification and replay mechanism (DG3) received immediate validation. Co-designers found it essential for restoring orientation after extended navigation. Sile’s observation that the feature allowed both coincidental and sequential playback highlighted its flexibility; users could hear the reference and current position simultaneously or separately depending on their analytic need. Venkatesh’s positive assessment confirmed that the feature successfully addressed the spatial disorientation identified in Session 1. Spatial audio (DG4) yielded mixed results. Stereo panning effectively encoded horizontal position (X-axis), with Sile noting its utility for tracing gradients. However, volume alone proved insufficient for depth perception (Z-axis). Sile could not perceive the volumetric changes during navigation. JooYoung’s suggestion to augment volume with reverb emerged directly from this limitation. His proposal to combine amplitude decay with echo creates a multidimensional depth cue that reinforces distance perception through both loudness and spatial reflection. This refinement exemplifies the iterative validation process: co-designers not only identified failures but proposed empirically grounded solutions rooted in their perceptual expertise. The configurable buffer (DG5) validated as a powerful analytic concept but surfaced critical usability issues, consistent with prior findings that aggregation mechanisms require explicit feedback to support interpretation [63]. Venkatesh questioned the two-step selection initiation process, suggesting that the system should anchor the selection at the user’s current position when D is pressed, rather than requiring a separate Enter confirmation. Ken’s question about the aggregated sound’s utility revealed that the feature’s purpose was not immediately apparent. Venkatesh clarified the intended use case: comparing aggregated values to individual points within the selected region. This narrative exposed a gap in the interaction design; the feature lacked sufficient feedback to communicate its analytic function. Venkatesh further suggested that the system announce when users enter or exit the selected region during subsequent navigation, providing continuous spatial awareness of the buffer’s boundaries. These findings further refined our understanding of RQ2, demonstrating that empowering accurate task performance and confident interpretation requires not only implementing features but ensuring their interaction design clearly communicates how they support specific analytic workflows. Co-designers discovered they could select a region using the buffer, then invoke the AI assistant to ask targeted questions about only that subset of the data. This scoped query pattern aligns with recent work on coupling spatial selection with conversational analysis [83]. This scoped query pattern emerged organically from Venkatesh’s observation that “if I’m looking at just this peak, I want to ask the AI about just this peak, not the whole dataset.” The buffer mechanism, originally designed for audio-based region comparison, thus revealed a second analytic function: constraining the context window for natural language queries. This finding demonstrates how multimodal features can support complementary interaction modalities—spatial selection via keyboard navigation directly informs textual query scope, creating a tighter coupling between embodied exploration and AI-mediated analysis. Table 6
CHI ’26, April 13–17, 2026, Barcelona, Spain
Kamath et al.
Feature
Test Prompt
Result & Co-designer Quote
Next Steps / Refinements
Reference Sonification & Replay (DG3)
“Navigate away from the origin and use the reference key to reorient yourself.”
Success. Sile: “it does help, actually... having it separate like that, but allowing you to play either coincidentally or in sequence, that’s really nice.” Venkatesh: “that was the main observation I had, but I think it’s pretty cool that I get to hear the reference point.”
None required; feature successfully validated for orientation restoration.
Stereo & Volumetric Audio (DG4)
“Trace a horizontal path and describe the spatial cues you perceive.”
Partial success. Stereo panning validated. Sile: “(stereo panning) it actually has been very interesting.” Volume for depth too subtle. Sile: “(volume) I couldn’t notice it that much.”
Add reverb to depth encoding. JooYoung: “Using volume only may not be sufficient, so I suggest using echo sound (reverb)...if you combine volume plus, echo, then, it will be more pronounced.”
Configurable Buffer (DG5)
“Select a region, compare its aggregated sound to individual points, then compare the region to a different area.”
Partial success. Concept validated, interaction issues surfaced. Venkatesh: “why...I mean, just flip the order of actions, right? Like, first go to the point that you want to start selection from, and then when you press D, the backend knows.” Ken: “I’m wondering...How this is useful... what am I getting from that?”
Redesign interaction: anchor selection at current position on first D-press. Add boundary entry/exit announcements. Venkatesh: “my use case for suggesting this is comparing whatever you might hear in aggregate to individual points.” See Table 6 for prioritized fixes.
Table 5: Session 2 feature validation: test prompts, results, and refinements.
documents the prioritized issues surfaced during buffer interaction testing and the corresponding design refinements necessary to support both audio comparison and AI scoping workflows. 5.3.2 Part 2: Discussed Future Directions. The brainwriting portion of Session 2 was primarily focussed on answering RQ3, generating a rich set of ideas for moving beyond the current webbased implementation. These ideas coalesced around three interconnected themes that align with prior embodied and immersive accessibility research [47, 60]: advanced haptics to restore persistent tactile feedback, mixed and augmented reality to merge physical and digital exploration spaces, and alternative physical interfaces to replace keyboard-centric navigation. Collectively, these proposals represent co-designers’ vision for translating the intuitive, exploratory nature of the tactile probe into digitally mediated forms that preserve, and potentially augment, its embodied affordances. (1) Advanced Haptics and Tactile Displays: Haptic feedback emerged as the most direct pathway for restoring the persistent tactile cues that grounded the physical prototype. Sile articulated haptics’ strength: “really good at dynamic feedback,” proposing that force resistance could encode data density—an aggregated data point “could be heavier, or harder to press against, the more data it is aggregating.” This maps physical effort directly onto information density, creating an embodied metric for data complexity. Ken identified nearterm implementation targets, advocating for existing devices like the Inverse3 Haptic Controller1 or the Graphiti tactile display2 as a “quickstep to trying a different method of ’feeling’ the graph.” Sile extended this vision to multi-line braille displays, imagining parallel multi-finger sensing that would allow simultaneous selection and comparison of multiple regions—a shift from sequential keyboard navigation to parallel tactile interaction. These proposals share a common 1 https://www.haply.co/inverse3 2 https://www.orbitresearch.com/product/graphiti-plus
thread: leveraging haptic modalities to restore the simultaneity and persistence lost in the transition from physical to digital. (2) Mixed and Augmented Reality: MR aims to merge the physical exploration space with digital data overlays. JooYoung positioned MR as “critical,” envisioning “bidirectional synchronization between the digital environments and physical artifacts.” His three-step operationalization concretizes this vision: (a) construct a physical play mat demarcating the X-Z grid, (b) align the AR boundary with the mat’s physical edges, and (c) sonify Y-values as pitch while users physically traverse the mat. This design recasts the user’s body as an “embodied cursor,” directly mapping physical movement through space onto data navigation. The approach dissolves the screen boundary, allowing co-designers to “walk through” the dataset using proprioceptive and vestibular cues—modalities entirely absent from keyboard-based interaction. This proposal directly extends the tactile prototype’s affordances by preserving spatial exploration while augmenting it with dynamic, real-time sonification. (3) Alternative Physical Interfaces: Beyond haptics and MR, co-designers proposed novel input devices to escape keyboard constraints while remaining grounded in accessible hardware. Venkatesh suggested repurposing familiar controls: two-way trackballs, surface dials, laptop touchpads, or even the keyboard surface itself as a 2D positional sensor. These ideas prioritize learnability and availability, adapting existing assistive technology interaction patterns to 3D data navigation. Ken proposed a more elaborate system: a glovemounted hand tracker controlled by CNC-style XYZ pulleys, providing kinesthetic feedback as users “sculpt” through the data space. This design borrows from haptic sculpture interfaces, mapping fine motor control onto precise data interrogation. Collectively, these speculative interfaces share a
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
CHI ’26, April 13–17, 2026, Barcelona, Spain
Priority
Usability Issue
Co-designer Quote / Observation
Proposed Fix
P1
Two-step selection initiation breaks flow; users must navigate away, press D, navigate to start point, press Enter.
Venkatesh: “why...I mean, just flip the order of actions, right? Like, first go to the point that you want to start selection from, and then when you press D, the backend knows.”
Anchor selection at current position on first D-press; eliminate redundant Enter step.
P2
Lack of boundary awareness; users cannot tell when they re-enter or exit the selected region during subsequent navigation.
Venkatesh: “it would be nice if, as you’re navigating around...it announces when you enter the region or when you leave the region.”
Add audio announcement when cursor crosses buffer boundary; consider persistent background cue (e.g., subtle ambient tone) while inside region.
P3
Aggregated playback purpose unclear; codesigners unsure when mean value is useful versus sequential point-by-point replay.
Ken: “I’m wondering...How this is useful... what am I getting from that?” Venkatesh clarified: “my use case for suggesting this is comparing whatever you might hear in aggregate to individual points.”
Provide contextual help (e.g., on-demand description of aggregation modes); add example prompts in tutorial demonstrating comparison workflow.
P4
No visual or audio confirmation when buffer selection is saved; uncertain whether system registered the region.
Observed during testing: co-designers pressed G multiple times, unsure if buffer was stored.
Add distinct confirmation sound when buffer is saved; optionally announce region dimensions (e.g., “Buffer saved: 6 by 7 region”).
Table 6: Prioritized buffer usability issues and proposed fixes.
rejection of traditional keyboard navigation in favor of controls that more closely mirror the physical prototype’s directness—whether through rotational input, positional sensing, or kinesthetic manipulation. Each proposal answers RQ3 by translating a specific dimension of the tactile probe’s embodied affordances into digitally augmented form.
6
DISCUSSION
Moving beyond a simple summary of outcomes, this section reflects on the insights from our iterative process. We interpret our findings by connecting them to existing academic literature and systems, focusing on the conceptual challenges of knowledge transfer from physical to digital forms, the nuanced role of non-visual features in empowering users, and the emerging pathways toward more physically-grounded data interaction [22, 44, 67].
6.1
Limitations of Our Study
Our study’s methodological scope introduces five primary limitations that constrain the generalizability of our findings. First, expert sample bias: our six BLV co-designers all possess advanced technical literacy, extensive experience with assistive technologies, and professional backgrounds in data analysis or accessibility research. While this auto-ethnographic approach provided invaluable, highfidelity insights grounded in lived expertise, features that were intuitive for this group may not be learnable for novice users unfamiliar with data visualization or advanced assistive technologies [90, 109]. Our prototype reflects design priorities and interaction patterns that emerged from expert users who bring sophisticated mental models and analytic strategies to 3D data exploration, which may not align with the needs, preferences, or learning trajectories of BLV individuals with less prior exposure to data visualization, scientific computing, or spatial reasoning tasks. Second, limited dataset types: our evaluation was conducted using relatively clean, dense datasets (benzene spectroscopy, Gaussian surfaces, weather data). The effectiveness of our sonification and navigation strategies has not been tested on sparse, disjointed, or noisy datasets, where
features like peak-jumping or aggregated buffering might behave differently [64]. Third, cross-sectional study design: we captured co-designers’ immediate reactions during two sessions but have not assessed longitudinal effects of using the prototype over time, where issues like auditory fatigue, habituation, or the development of new usage patterns might emerge [23]. Fourth, volumetric audio subtlety: as identified in Session 2, volume alone proved insufficient for depth perception (Z-axis), requiring augmentation with reverb—a refinement not yet implemented in the validated prototype [93]. Fifth, buffer UX issues: the configurable buffer surfaced critical usability barriers documented in Table 6, including two-step selection initiation, lack of boundary awareness, unclear aggregation purpose, and absent confirmation feedback. While the buffer concept validated successfully, its interaction design requires substantial refinement before deployment with non-expert users.
6.2
Conceptual Contributions Derived from the Co-Design Study
Our Experience-Based Co-Design process with expert BLV collaborators yielded three interrelated conceptual contributions that extend beyond the specific implementation of our system. These contributions address longstanding gaps in accessible 3D data visualization by centering lived experience as design material [62, 70, 82]. • We developed a transferable co-design protocol for translating tactile expertise into multimodal digital interaction. By pairing a low-fidelity tactile probe with a highfidelity digital prototype, we established a methodological framework that systematically elicits tactile knowledge [99] and translates it into non-visual digital affordances. This protocol operationalizes EBCD principles [82] for HCI contexts where physical and digital modalities must interoperate. The tactile probe served as an epistemological anchor, providing a shared sensory baseline against which digital experiences could be evaluated. Co-designers’ narratives about orientation confusion, spatial comparison challenges, and analytic workflows directly motivated implemented features
CHI ’26, April 13–17, 2026, Barcelona, Spain
(reference sonification, configurable buffer, spatial audio), demonstrating how experiential “touch-points” [30] can be systematically converted into design requirements. This protocol is transferable: future projects targeting other 3D data types (e.g., molecular structures, terrain models) can adapt our phased approach; tactile grounding, comparative evaluation, iterative refinement, to surface domain-specific analytic needs. • We produced a validated prototype enabling independent non-visual 3D data exploration. Our system demonstrates that web-native, multimodal interaction can support core analytic tasks (3D orientation, gradient tracing, landmark identification, region comparison) without requiring specialized hardware. Session 2 validation confirmed that features like reference sonification, stereo panning, and buffer aggregation improved co-designers’ ability to orient, analyze, and interpret 3D surfaces with confidence. Unlike prior systems that rely on expensive tactile displays [58, 97] or sighted-centered VR [44], our web-based approach offers scalable, low-cost access to continuous height-field surfaces across scientific domains. The prototype’s modular architecture enabled rapid iteration during co-design sessions, allowing us to implement complex features (e.g., buffer aggregation with dual playback modes) within days of co-designer feedback. This responsiveness exemplifies how technical infrastructure can support rather than constrain participatory design [87]. • We established empirically grounded design principles for accessible visualization. Our findings converge on four principles that extend existing accessible visualization guidance [22, 67] into three-dimensional contexts: (1) Multiparameter redundancy; (2) Progressive disclosure; (3) Bufferaggregation as analytic scoping; and (4) Persistent spatial reference. These principles, validated through co-designer testimony and artifact evaluation, offer concrete guidance for designers extending accessible visualization beyond 2D charts into volumetric, spatially continuous datasets [23].
6.3
System-based Design Insights
Drawing from both our prototype implementation and co-designer validation, we distill two complementary sets of insights: boundary conditions specific to our system’s current instantiation, and design principles applicable to future accessible 3D visualization systems. This dual framing, requested by co-designers during Session 2 debriefing, clarifies which design decisions are implementationdependent versus conceptually transferable [41]. 6.3.1 Boundary Conditions and Prototype Specifics. Our system’s validated configuration represents a constellation of design decisions that emerged iteratively through co-designer feedback and artifact evaluation. We document these boundary conditions to distinguish implementation-specific choices from transferable principles, clarifying the scope within which our findings remain empirically grounded. • Binaural audio engine: Our browser-native implementation is built entirely on Web Audio API primitives operating
Kamath et al.
at 48kHz sample rate, deliberately avoiding external synthesis libraries to ensure zero-latency compatibility across devices. The audio signal chain progresses sequentially from oscillator generation through envelope shaping, stereo panning, and convolution reverb, creating a modular architecture that our co-designers found responsive during real-time exploration. This architectural choice prioritized immediate auditory feedback over sophisticated synthesis capabilities, which was validated when co-designers consistently praised the system’s responsiveness during navigation. • Sonification mappings: Spatial coordinates translated into perceptible audio through four concurrent encodings that emerged directly from Session 1 deliberations. Following Aziz’s suggestion, horizontal position (X-axis) maps to stereo panning ranging from fully left (-1.0) to fully right (+1.0), leveraging users’ innate spatial hearing capabilities. Vertical position (Y-axis, representing surface height) maps to pitch via a logarithmic scale spanning 200Hz to 800Hz, ensuring that equal perceptual differences correspond to proportional height changes [39]. Depth perception (Z-axis) proved most complex, ultimately requiring JooYoung’s proposed multiparameter encoding that combines dynamic volume with frequency-dependent reverb: closer points play at higher amplitude (20% wet mix) with minimal pre-delay (10ms) and preserved high frequencies (6500Hz lowpass), while distant points grow quieter (95% wet mix), gain substantial echo (90ms pre-delay), and lose treble clarity (2000Hz lowpass). Additionally, waveform timbre varies—sine waves for low-frequency regions, triangle waves for mid-range, square waves for high-frequency areas—to provide redundant height information through timbral brightness. • Configurable buffer mechanism: JooYoung’s Session 1 brainwriting proposal established rectangular region selection through a three-stage keyboard interaction: D-key initiates selection mode, Enter anchors the starting corner at the current cursor position, arrow keys navigate to define the opposite corner, and a second Enter press confirms the bounded region. The buffer stores all grid rectangles or individual data points falling within these bounds, enabling subsequent analytic operations. When users activate aggregation mode via G-key, the system computes the arithmetic mean of normalized Y-values within the buffered region and plays a sustained 1.0-second tone representing this summary statistic. Alternatively, sequential playback mode traverses each buffered item individually at 0.3-second intervals with 125ms inter-item silence, allowing detailed inspection. These timing parameters emerged through Aziz’s iterative testing during artifact evaluation, balancing playback duration against cognitive processing demands. • Dataset density constraints: Our validation focused exclusively on dense, continuous surface datasets where neighboring points maintain close spatial proximity: benzene VUV spectroscopy data (3,116 measurement points), synthetic Gaussian surfaces (2,500 points), and meteorological height fields (1,200 points). These dataset constraints have direct implications: peak-jumping functionality reliably identifies local maxima when surfaces exhibit smooth gradients, but
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
CHI ’26, April 13–17, 2026, Barcelona, Spain
its behavior remains undefined for sparse point clouds or disjoint data distributions with large spatial voids. Similarly, buffer aggregation produces meaningful summary statistics when regions contain sufficient data density to yield representative means, a condition not guaranteed for irregular sampling patterns. We explicitly acknowledge this limitation in Section 6.1, recognizing that alternative dataset types may require modified algorithms or additional safeguards. • Two-tier navigation structure: Navigation defaults to a 20×20 wireframe grid overlay where each rectangular cell aggregates the underlying surface region it bounds, reducing the initial exploration space from potentially thousands of individual points to 400 manageable zones. Aziz specifically requested this “surface mode” during Phase 1 co-design to mitigate cognitive load when first encountering unfamiliar datasets, allowing users to grasp overall topography before diving into granular detail. Users can toggle to point mode via 2-key when finer-grained analysis becomes necessary, shifting from regional overviews to vertex-by-vertex traversal. This structure mirrors cartographic conventions of zooming from overview maps to detailed street views [85]. • Spatial orientation support: Two complementary mechanisms that Ken and Sile advocated during Session 1 was about reference sonification. It provides an absolute coordinate anchor: pressing 0-key triggers a fixed 300Hz sine tone representing the dataset origin (0,0,0) regardless of current cursor position, allowing users to recalibrate their mental spatial model at any moment. Complementing this, the replay mechanism (. key) re-sonifies the user’s current position without causing navigation, enabling repeated listening when interpreting ambiguous or complex local features. These orientation aids address the fundamental challenge of maintaining spatial awareness in non-visual environments where users cannot glance at coordinate axes as sighted analysts routinely do. These specifications collectively define the empirical scope of our co-design validation. We present them not as prescriptive requirements but as documented implementation choices that proved effective with our expert co-designer cohort exploring dense surface datasets via keyboard interaction. Different user populations, alternative dataset types, or novel deployment contexts may necessitate adjusted parameter ranges, modified interaction patterns, or architectural alternatives. 6.3.2 Design Principles. Beyond our system’s specific configuration, our co-design process surfaced four transferable principles applicable to accessible 3D visualization systems regardless of implementation platform. • Multi-parameter redundancy for spatial encoding: Encoding each spatial dimension through multiple reinforcing modalities (e.g., X-axis via stereo panning and verbal coordinate announcements; Z-axis via volume and reverb and textual depth cues) ensures robust perception across diverse sensory preferences and abilities. Co-designers with varying visual acuities reported differential reliance on auditory vs.
textual feedback, validating the necessity of parallel channels [39, 46]. Source: Co-designer testimony (Session 2); Sile noted reliance on text, Venkatesh prioritized audio. • Progressive disclosure through whole-to-part navigation: Scaffolding users from high-level overviews (auto-play across multiple perspectives) to granular detail (point-bypoint traversal) supports pedagogical transitions from initial orientation to hypothesis-driven analysis. This mirrors established accessible visualization patterns [89, 104] but extends them into volumetric contexts. The ability to quickly “snapshot” a dataset’s topography before diving into specific features reduces cognitive load and accelerates mental model formation. Source: Co-designer testimony (Phase 1); JooYoung identified tedium of point-by-point exploration. • Buffer-aggregation as analytic scoping tool: Enabling users to define spatial regions of interest and toggle between summary statistics (mean) and sequential detail empowers two critical workflows: (1) comparing local maxima to surrounding context, and (2) scoping natural language queries to specific subsets of data. This principle extends beyond sonification; any multimodal system benefits from user-controlled granularity adjustments. The emergent use case of bufferscoped AI queries (Section 5.3) demonstrates how spatial selection mechanisms can integrate with conversational interfaces. Source: Co-designer testimony (Session 2); Venkatesh articulated comparison use case, emergent AI scoping observed during testing. • Pedagogical scaffolding for tactile-to-digital transition: Designing digital features to replicate the cognitive functions (not literal affordances) of physical probes facilitates knowledge transfer. Our tactile probe’s persistent frame of reference became digital reference sonification; its multifinger comparison became configurable buffer playback. This principle aligns with mixed-reality prototyping best practices [94, 96] but specifies a pathway for non-visual contexts: deconstruct physical affordances into core cognitive operations, then implement digital analogs serving those operations. Source: Co-designer testimony (Session 1 comparative analysis); conceptual mapping documented in Section 5.2. The principles aforementioned, grounded in co-designer narratives and validated through artifact evaluation, provide actionable guidance for future accessible 3D visualization systems. Unlike the boundary conditions in Section 6.3.1, these insights transcend specific implementation choices and apply across platforms, modalities, and dataset types.
6.4
Exploratory Insights from Co-Designer Speculations
Session 2’s brainwriting exercise (Section 5.3.2) produced speculative extensions of the validated web-based prototype toward more embodied forms of 3D data exploration. These concepts were not implemented or evaluated; instead, they reflect co-designers’ expert intuitions about how embodied cognition [60] and proprioception [8] could support non-visual analysis beyond screen-based interaction. We present these ideas as exploratory insights that
CHI ’26, April 13–17, 2026, Barcelona, Spain
outline possible future directions rather than validated design outcomes. 6.4.1 Advanced Haptics as Persistent Tactile Grounding. Co-designers consistently identified haptic feedback as the most promising mechanism for restoring a sense of physical persistence to data exploration. Mechanisms to encode data density through force resistance, allowing users to “feel” aggregated regions as heavier or harder to press against; maps analytic properties (e.g., density, gradients) onto physical effort, providing continuous, user-controlled cues rather than transient auditory signals. Co-designers also proposed large, multi-line braille displays to enable parallel, multi-finger comparison across regions of a dataset. These ideas build on prior work in touch-based accessible graphics [12] and shape-changing interfaces [58], but emphasize haptics as an analytic encoding rather than a visual surrogate. Future work could prototype taskspecific haptic mappings (e.g., force gradients for slope, vibration for density) and compare their learnability to sonification. Source: Co-designer speculation (Session 2 brainwriting). 6.4.2 Embodied Spatial Exploration Beyond the Screen. Mixed and augmented reality were discussed only as background inspiration for extending interaction beyond the screen, rather than as implementation targets. Co-designers emphasized the value of leveraging whole-body proprioception, such as walking or reaching, to support spatial reasoning in 3D data. Related suggestions focused on spatialized audio, with data points rendered as a 3D soundscape that users could explore by moving through space. These ideas align with prior work on immersive and embodied analytics [44, 61], but were not central to the prototype’s design or evaluation. Instead, they highlight longer-term opportunities for coupling proprioception and spatial audio to support non-visual mental models of complex data. Source: Co-designer speculation (Session 2 brainwriting). 6.4.3 Alternative Physical Interfaces Beyond Keyboard Navigation. Co-designers also critiqued the mismatch between discrete keyboard input and continuous 3D data spaces [102]. Arrow-key navigation enforces sequential, axis-aligned movement, increasing cognitive load when exploring continuous surfaces. Across proposals (Section 6.4.3), a shared principle emerged: input modalities should match the dimensionality and continuity of the data. Future work could compare navigation efficiency, learnability, and analytic accuracy across keyboards, analog controllers, and haptic devices, particularly for users with diverse motor abilities and assistive technology experience. Source: Co-designer speculation (Session 2 brainwriting).
6.5
Future Directions
Our future work will move from foundational design to rigorous validation and conceptual expansion. The immediate priority is to conduct formal user studies to validate the usability and learnability of our current feature set with a broader, non-expert BLV population. Following this, we will pursue a research agenda focused on enhancing the user’s control and perception. This includes investigating alternative physical input devices, such as joysticks and trackballs, to afford more fluid, multi-dimensional navigation that better aligns with the exploration of 3D space. We will also refine the sonification engine, exploring more perceptually salient
Kamath et al.
audio cues, such as reverb, to create a richer and more intuitive representation of depth. Finally, we will expand the prototype’s analytical scope by implementing features for more complex data scenarios, such as the comparative plot view, to study how nonvisual users perform analysis across layered datasets. We will also attempt to prototype a spatially adaptive interface to examine how proprioception can transform data exploration from a navigational task into an immersive experience.
7
CONCLUSION
Through an experience-based co-design process with BLV collaborators with expertise in non-visual data representations, our aim was to create a digital experience that preserves spatial understanding and enables independent multiperspectival exploration that is scalable into multiple fields irrespective of data types. Our refined prototype demonstrated how sonification, spatialized audio, and adaptive aggregation can support orientation, accuracy, and learnability in surface and point plots. Beyond the artifact, our work contributes (1) a transferable co-design protocol for translating tactile expertise into digital interaction, and (2) concrete guidance for designing accessible analytic workflows around volumetric data. At the same time, our findings are bounded by a small expert user group, a limited set of datasets, and a cross-sectional evaluation. Future work should expand evaluation with a broader BLV population, investigate alternative input modalities for more fluid navigation, refine sonification with perceptually salient cues, and extend analytic scope to comparative and embodied interfaces. Taken together, these directions chart a concrete, evidence-based path toward equitable access to 3D scientific visualization.
ACKNOWLEDGMENTS This work was supported in part by JooYoung Seo’s Faculty Startup Fund, which provided funding for project materials, including lowfidelity prototypes and shipping costs. We would also like to extend our sincere gratitude to Tad Schroeder, an overall superhero, day-jobbing as the Assistant Director of Facilities at the School of Information Sciences, University of Illinois Urbana-Champaign for his invaluable support in coordinating the shipping logistics for our low-fidelity prototype.
REFERENCES [1] Knight Lab Studio [n. d.]. Uncharted Territory: Diving in to Data Visualization in Virtual Reality. Knight Lab Studio. https://studio.knightlab.com/r esults/exploring-data-visualization-in-vr/uncharted-territorydatavis-vr/ [2] Ali Abdolrahmani, Kevin M. Storer, Antony Rishin Mukkath Roy, Ravi Kuber, and Stacy M. Branham. 2020. Blind Leading the Sighted: Drawing Design Insights from Blind Users towards More Productivity-oriented Voice Interfaces. ACM Transactions on Accessible Computing 12, 4 (Jan. 2020), 18:1–18:35. https: //doi.org/10.1145/3368426 [3] Nancy E Adams. 2015. Bloom’s taxonomy of cognitive learning objectives. Journal of the Medical Library Association : JMLA 103, 3 (July 2015), 152–153. https://doi.org/10.3163/1536-5050.103.3.010 [4] Dragan Ahmetovic, Cristian Bernareggi, João Guerreiro, Sergio Mascetti, and Anna Capietto. [n. d.]. AudioFunctions.Web: Multimodal Exploration of Mathematical Function Graphs. In Proceedings of the 16th International Web for All Conference (New York, NY, USA, 2019-05-13) (W4A ’19). Association for Computing Machinery, 1–10. https://doi.org/10.1145/3315002.3317560 [5] Dragan Ahmetovic, Cristian Bernareggi, João Guerreiro, Sergio Mascetti, and Anna Capietto. 2019. AudioFunctions.web: Multimodal Exploration of Mathematical Function Graphs. In Proceedings of the 16th International Web for All
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
Conference (W4A ’19). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3315002.3317560 [6] Safinah Ali, Laya Muralidharan, Felicia Alfieri, Monali Agrawal, and Jacob Jorgensen. 2020. Sonify: Making Visual Graphs Accessible. In Human Interaction and Emerging Technologies, Tareq Ahram, Redha Taiar, Serge Colson, and Arnaud Choplin (Eds.). Springer International Publishing, Cham, 454–459. https: //doi.org/10.1007/978-3-030-25629-6_70 [7] Ahmed Amer and Phillip Peralez. 2014. Affordable altered perspectives: Making augmented and virtual reality technology accessible. In IEEE Global Humanitarian Technology Conference (GHTC 2014). 603–608. https://doi.org/10.110 9/GHTC.2014.6970345 [8] Robert Ball and Chris North. 2007. Realizing embodied interaction for visual analytics through large displays. Computers & Graphics 31, 3 (June 2007), 380– 400. https://doi.org/10.1016/j.cag.2007.01.029 [9] Cynthia L. Bennett, Erin Brady, and Stacy M. Branham. 2018. Interdependence as a Frame for Assistive Technology Research and Design. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’18). Association for Computing Machinery, New York, NY, USA, 161– 173. https://doi.org/10.1145/3234695.3236348 [10] Tigmanshu Bhatnagar, Albert Higgins, Nicolai Marquardt, Mark Miodownik, and Catherine Holloway. 2023. Analysis of Product Architectures of Pin Array Technologies for Tactile Displays. Database of Pin Array Technologies included for the Analysis of Product Architectures of Pin Array Technologies for Tactile Displays 7, ISS (Nov. 2023), 432:135–432:155. https://doi.org/10.1145/3626468 [11] Ivy Bui, Arunabh Bhattacharya, Si Hui Wong, Harinder R. Singh, and Arpit Agarwal. 2021. Role of Three-Dimensional Visualization Modalities in Medical Education. Frontiers in Pediatrics 9 (Dec. 2021), 760363. https://doi.org/10 .3389/fped.2021.760363 [12] Matthew Butler, Leona M Holloway, Samuel Reinders, Cagatay Goncu, and Kim Marriott. [n. d.]. Technology Developments in Touch-Based Accessible Graphics: A Systematic Review of Research 2010-2020. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2021-05-07) (CHI ’21). Association for Computing Machinery, 1–15. https: //doi.org/10.1145/3411764.3445207 [13] Matthew Butler, Leona M Holloway, Samuel Reinders, Cagatay Goncu, and Kim Marriott. 2021. Technology Developments in Touch-Based Accessible Graphics: A Systematic Review of Research 2010-2020. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.114 5/3411764.3445207 [14] Francesco Cafaro. [n. d.]. Using Embodied Allegories to Design Gesture Suites for Human-Data Interaction. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (New York, NY, USA, 2012-09-05) (UbiComp ’12). Association for Computing Machinery, 560–563. https://doi.org/10.1145/2370216. 2370309 [15] Jinho Choi, Sanghun Jung, Deok Gun Park, Jaegul Choo, and Niklas Elmqvist. 2019. Visualizing for the Non-Visual: Enabling the Visually Impaired to Use Visualization. Computer Graphics Forum 38, 3 (2019), 249–260. https://doi.org/10.1111/cgf.13686 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.13686. [16] Pramod Chundury, Biswaksen Patnaik, Yasmin Reyazuddin, Christine Tang, Jonathan Lazar, and Niklas Elmqvist. 2022. Towards Understanding Sensory Substitution for Accessible Visualization: An Interview Study. IEEE transactions on visualization and computer graphics 28, 1 (Jan. 2022), 1084–1094. https: //doi.org/10.1109/TVCG.2021.3114829 [17] James M. Coughlan and Joshua Miele. 2017. AR4VI: AR as an Accessibility Tool for People with Visual Impairments. ... IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) 2017 (Oct. 2017), 288–292. https://doi.org/10.1109/ISMAR-Adjunct.2017.89 [18] Chris Creed, Maadh Al-Kalbani, Arthur Theil, Sayan Sarcar, and Ian Williams. 2024. Inclusive Augmented and Virtual Reality: A Research Agenda. International Journal of Human–Computer Interaction 40, 20 (Oct. 2024), 6200– 6219. https://doi.org/10.1080/10447318.2023.2247614 _eprint: https://doi.org/10.1080/10447318.2023.2247614. [19] Caitlin de Villiers. 2023. Embodied Knowledge in 4IR-Oriented Design Practice – Autoethnographic Approaches to Experiential Futuredirected Ways of Knowing and Learning in Selected Case Studies - ProQuest. Ph. D. Dissertation. University of Johannesburg, South Africa. https://www.proquest.com/docview/322 4565125 [20] Yehor Dzhurynskyi, Volodymyr Mayik, and Lyudmyla Mayik. 2024. Enhancing Accessibility: Automated Tactile Graphics Generation for Individuals with Visual Impairments. Computation 12, 12 (Dec. 2024), 251. https://doi.org/10.339 0/computation12120251 [21] Salla Eilola, Kaisa Jaalama, Petri Kangassalo, Pilvi Nummi, Aija Staffans, and Nora Fagerholm. 2023. 3D visualisations for communicative urban and landscape planning: What systematic mapping of academic literature can tell us of their potential? Landscape and Urban Planning 234 (June 2023), 104716. https:
CHI ’26, April 13–17, 2026, Barcelona, Spain
//doi.org/10.1016/j.landurbplan.2023.104716 [22] Frank Elavsky, Cynthia Bennett, and Dominik Moritz. 2022. How accessible is my visualization? Evaluating visualization accessibility with Chartability. Computer Graphics Forum 41, 3 (2022), 57–70. https://doi.org/10.1111/cgf.14522 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14522. [23] Kajetan Enge, Elias Elmquist, Valentina Caiola, Niklas Rönnberg, Alexander Rind, Michael Iber, Sara Lenzi, Fangfei Lan, Robert Höldrich, and Wolfgang Aigner. 2024. Open Your Ears and Take a Look: A State-of-the-Art Report on the Integration of Sonification and Visualization. Computer Graphics Forum 43, 3 (June 2024), e15114. https://doi.org/10.1111/cgf.15114 arXiv:2402.16558 [cs]. [24] Danyang Fan, Alexa Fay Siu, Wing-Sum Adrienne Law, Raymond Ruihong Zhen, Sile O’Modhrain, and Sean Follmer. 2022. Slide-Tone and Tilt-Tone: 1DOF Haptic Techniques for Conveying Shape Characteristics of Graphs to Blind Users. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–19. https://doi.org/10.1145/3491102.3517790 [25] Danyang Fan, Olivia Tomassetti, Aya Mouallem, Gene S-H Kim, Shloke Nirav Patel, Saehui Hwang, Patricia Leader, Danielle Sugrue, Tristen Chen, Darren Reese Ou, Victor R Lee, Lakshmi Balasubramanian, Hariharan Subramonyam, Sile O’Modhrain, and Sean Follmer. 2025. Promoting Comprehension and Engagement in Introductory Data and Statistics for Blind and Low-Vision Students: A Co-Design Study. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, 1–20. https://doi.org/10.1145/3706598.3713333 [26] Anis Farshian, Markus Götz, Gabriele Cavallaro, Charlotte Debus, Matthias Nießner, Jón Atli Benediktsson, and Achim Streit. 2023. Deep-Learning-Based 3-D Surface Reconstruction—A Survey. Proc. IEEE 111, 11 (Nov. 2023), 1464–1501. https://doi.org/10.1109/JPROC.2023.3321433 [27] Emilie Francis-Auton, Colleen Cheek, Elizabeth Austin, Natalia Ransolin, Lieke Richardson, Mariam Safi, Nematullah Hayba, Luke Testa, Reema Harrison, Jeffrey Braithwaite, and Robyn Clay-Williams. 2024. Exploring and Understanding the ‘Experience’ in Experience-Based Codesign: A State-of-The-Art Review. International Journal of Qualitative Methods 23 (Jan. 2024), 16094069241235563. https://doi.org/10.1177/16094069241235563 [28] Beth Fylan, Justine Tomlinson, David K. Raynor, and Jonathan Silcock. 2021. Using experience-based co-design with patients, carers and healthcare professionals to develop theory-based interventions for safer medicines use. Research in Social and Administrative Pharmacy 17, 12 (Dec. 2021), 2127–2135. https://doi.org/10.1016/j.sapharm.2021.06.004 [29] David Geary, Jon Francombe, Kristian Hentschel, and Damian Murphy. 2022. Design Dimensions of Co-Located Multi-Device Audio Experiences. Applied Sciences 12, 15 (Jan. 2022), 7512. https://doi.org/10.3390/app12157512 [30] Nils Graber, Nina Canova, Denise Bryant-Lukosius, Glenn Robert, Blanca Navarro-Rodrigo, Lionel Trueb, George Coukos, Manuela Eicher, Tourane Corbière, and Sara Colomer-Lahiguera. 2024. Reflections on the opportunities and challenges of applying experience-based co-design (EBCD) to phase 1 clinical trials in oncology. Health Expectations 27, 4 (2024), e14068. https://doi.org/10.1111/hex.14068 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/hex.14068. [31] Theresa Green, Ann Bonner, Laisa Teleni, Natalie Bradford, Louise Purtell, Clint Douglas, Patsy Yates, Margaret MacAndrew, Hai Yen Dao, and Raymond Javan Chan. 2020. Use and reporting of experience-based codesign studies in the healthcare setting: a systematic review. BMJ Quality & Safety 29, 1 (Jan. 2020), 64–76. https://doi.org/10.1136/bmjqs-2019-009570 [32] Silvia Grimaldi, Steven Fokkinga, and Ioana Ocnarescu. 2013. Narratives in design: a study of the types, applications and functions of narratives in design practice. In Proceedings of the 6th International Conference on Designing Pleasurable Products and Interfaces. ACM, Newcastle upon Tyne United Kingdom, 201–210. https://doi.org/10.1145/2513506.2513528 [33] Paul Grimm, Wolfgang Broll, Rigo Herold, Johannes Hummel, and Rolf Kruse. 2022. VR/AR Input Devices and Tracking. In Virtual and Augmented Reality (VR/AR): Foundations and Methods of Extended Realities (XR), Ralf Doerner, Wolfgang Broll, Paul Grimm, and Bernhard Jung (Eds.). Springer International Publishing, Cham, 107–148. https://doi.org/10.1007/978-3-030-790622_4 [34] N. C. Harte, D. Obrist, M. Versluis, E. Groot Jebbink, M. Caversaccio, W. Wimmer, and G. Lajoinie. 2024. Second order and transverse flow visualization through three-dimensional particle image velocimetry in millimetric ducts. Experimental Thermal and Fluid Science 159 (Dec. 2024), 111296. https://doi.org/10.101 6/j.expthermflusci.2024.111296 [35] Tingying He, Maggie McCracken, Daniel Hajas, Sarah Creem-Regehr, and Alexander Lex. 2025. Using Tactile Charts to Support Comprehension and Learning of Complex Visualizations for Blind and Low-Vision Individuals. https://doi.org/10.48550/arXiv.2507.21462 arXiv:2507.21462 [cs]. [36] Bridger Herman, Cullen D. Jackson, and Daniel F. Keefe. [n. d.]. Touching the Ground: Evaluating the Effectiveness of Data Physicalizations for Spatial Data Analysis Tasks. 31, 1 ([n. d.]), 875–885. https://doi.org/10.1109/TVCG.2
CHI ’26, April 13–17, 2026, Barcelona, Spain
024.3456377 [37] Leona Holloway, Swamy Ananthanarayan, Matthew Butler, Madhuka Thisuri De Silva, Kirsten Ellis, Cagatay Goncu, Kate Stephens, and Kim Marriott. 2022. Animations at Your Fingertips: Using a Refreshable Tactile Display to Convey Motion Graphics for People who are Blind or have Low Vision. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3517428.3544797 [38] Leona M Holloway, Cagatay Goncu, Alon Ilsar, Matthew Butler, and Kim Marriott. 2022. Infosonics: Accessible Infographics for People who are Blind using Sonification and Voice. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3491102.3517465 [39] Md Naimul Hoque, Md Ehtesham-Ul-Haque, Niklas Elmqvist, and Syed Masum Billah. 2023. Accessible Data Representation with Natural Sound. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–19. https: //doi.org/10.1145/3544548.3581087 [40] Stacy Hsueh, Beatrice Vincenzi, Akshata Murdeshwar, and Marianela Ciolfi Felice. 2023. Cripping Data Visualizations: Crip Technoscience as a Critical Lens for Designing Digital Access. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’23). Association for Computing Machinery, New York, NY, USA, 1–16. https: //doi.org/10.1145/3597638.3608427 [41] Janet Yi-Ching Huang, Stephan Wensveen, and Mathias Funk. 2023. Experiential speculation in vision-based AI design education: Designing conventional and progressive AI futures. (2023). https://doi.org/10.57698/V17I2.01 [42] Ying Ying Huang, Jonas Moll, Eva-Lotta Sallnäs, and Yngve Sundblad. 2012. Auditory feedback in haptic collaborative interfaces. International Journal of Human-Computer Studies 70, 4 (April 2012), 257–270. https://doi.org/10.1 016/j.ijhcs.2011.11.006 [43] Najwa Ayuni Jamaludin, Farhan Mohamed, Vei Siang Chan, Mohd Shahrizal Sunar, Ali Selamat, Ondrej Krejcar, and Andres Iglesias. [n. d.]. Answering Why and When?: A Systematic Literature Review of Application Scenarios and Evaluation for Immersive Data Visualization Analytics. 25, 1 ([n. d.]), 1–29. https://doi.org/10.4018/JCIT.323799 [44] Najwa Ayuni Jamaludin, Farhan Mohamed, Vei Siang Chan, Mohd Shahrizal Sunar, Ali Selamat, Ondrej Krejcar, and Andres Iglesias. 2023. Answering Why and When?: A Systematic Literature Review of Application Scenarios and Evaluation for Immersive Data Visualization Analytics. Journal of Cases on Information Technology (JCIT) 25, 1 (Jan. 2023), 1–29. https://doi.org/10.4 018/JCIT.323799 [45] Seungwoo Je, Hyunseung Lim, Kongpyung Moon, Shan-Yuan Teng, Jas Brooks, Pedro Lopes, and Andrea Bianchi. 2021. Elevate: A Walkable Pin-Array for Large Shape-Changing Terrains. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3411764.3445454 [46] Chutian Jiang, Yinan Fan, Junan Xie, Emily Kuang, Kaihao Zhang, and Mingming Fan. 2024. Designing Unobtrusive Modulated Electrotactile Feedback on Fingertip Edge to Assist Blind and Low Vision (BLV) People in Comprehending Charts. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA, 1–20. https://doi.org/10.1145/3613904.3642546 [47] Elise Johnson, S. Sandra Bae, and Ellen Yi-Luen Do. [n. d.]. Supporting Data Visualization Literacy through Embodied Interactions. In Proceedings of the 15th Conference on Creativity and Cognition (New York, NY, USA, 2023-06-19) (C&C ’23). Association for Computing Machinery, 346–348. https://doi.org/10.1 145/3591196.3596607 [48] Peter Jones. 2018. Contexts of Co-creation: Designing with System Stakeholders. In Systemic Design, Peter Jones and Kyoichi Kijima (Eds.). Vol. 8. Springer Japan, Tokyo, 3–52. https://doi.org/10.1007/978-4-431-55639-8_1 Series Title: Translational Systems Sciences. [49] Shakila Cherise S Joyner, Amalia Riegelhuth, Kathleen Garrity, Yea-Seul Kim, and Nam Wook Kim. 2022. Visualization Accessibility in the Wild: Challenges Faced by Visualization Designers. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–19. https://doi.org/10.1145/3491102.3517630 [50] Sanchita S. Kamath, Aziz N. Zeidieh, and JooYoung Seo. 2025. Explore, Listen, Inspect: Supporting Multimodal Interaction with 3D Surface and Point Data Visualizations. https://doi.org/10.1145/3663547.3759765 arXiv:2508.08554 [cs]. [51] Alexander S. Kaplitz and Kevin A. Schug. 2023. Gas chromatography—vacuum ultraviolet spectroscopy in petroleum and fuel analysis. Analytical Science Advances 4, 5-6 (2023), 220–231. https://do i . o r g / 1 0 . 1 0 0 2 / a n s a . 2 0 2 3 0 0 0 25 _eprint: https://chemistryeurope.onlinelibrary.wiley.com/doi/pdf/10.1002/ansa.202300025. [52] Tiffany Karalis Noel, Aiko Minematsu, and Nikki Bosca. 2023. Collective Autoethnography as a Transformative Narrative Methodology. International
Kamath et al.
Journal of Qualitative Methods 22 (Oct. 2023), 16094069231203944. https: //doi.org/10.1177/16094069231203944 [53] Adnan Khan, Alireza Choubineh, Mai A. Shaaban, Abbas Akkasi, and Majid Komeili. 2025. TactileNet: Bridging the Accessibility Gap with AI-Generated Tactile Graphics for Individuals with Vision Impairment. https://doi.org/ 10.48550/arXiv.2504.04722 arXiv:2504.04722 [cs]. [54] Hyeok Kim, Yea-Seul Kim, and Jessica Hullman. 2024. Erie: A Declarative Grammar for Data Sonification. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–19. https://doi.org/10.1145/3613904.36 42442 arXiv:2402.00156 [cs]. [55] N. W. Kim, S. C. Joyner, A. Riegelhuth, and Y. Kim. 2021. Accessible Visualization: Design Space, Opportunities, and Challenges. Computer Graphics Forum 40, 3 (2021), 173–188. https://doi.org/10.1111/cgf.14298 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14298. [56] Bongshin Lee, Eun Kyoung Choe, Petra Isenberg, Kim Marriott, and John Stasko. 2020. Reaching Broader Audiences With Data Visualization. IEEE Computer Graphics and Applications 40, 2 (March 2020), 82–90. https://doi.org/10.1 109/MCG.2020.2968244 [57] Benjamin Lee, Xiaoyun Hu, Maxime Cordeil, Arnaud Prouzeau, Bernhard Jenny, and Tim Dwyer. [n. d.]. Shared Surfaces and Spaces: Collaborative Data Visualisation in a Co-located Immersive Environment. 27, 2 ([n. d.]), 1171–1181. https://doi.org/10.1109/TVCG.2020.3030450 [58] Daniel Leithinger, Sean Follmer, Alex Olwal, and Hiroshi Ishii. 2015. Shape Displays: Spatial Interaction with Dynamic Physical Form. IEEE Computer Graphics and Applications 35, 5 (Sept. 2015), 5–11. https://doi.org/10.110 9/MCG.2015.111 [59] Richard Qi Li. [n. d.]. Taoist Data Visualization: An Embodied Aesthetic Approach to Data Visualization through Gesture-Based Technology. 34, 1 ([n. d.]), 67–91. https://hta.ac/ojs/htr/article/view/taoist-datavisualization [60] Qiuyu Liu, Jingwen Pan, and Gang Li. 2020. Research of Immersive Geospatial Data Visualization based on Embodied Cognition. In 2020 International Conference on Innovation Design and Digital Technology (ICIDDT). 358–362. https://doi.org/10.1109/ICIDDT52279.2020.00071 [61] Richen Liu, Min Gao, Lijun Wang, Xiaohan Wang, Yuzhe Xiang, Aolin Zhang, Jiazhi Xia, Yi Chen, and Siming Chen. 2022. Interactive Extended Reality Techniques in Information Visualization. IEEE Transactions on Human-Machine Systems 52, 6 (Dec. 2022), 1338–1351. https://doi.org/10.1109/THMS.202 2.3211317 [62] Alan Lundgard, Crystal Lee, and Arvind Satyanarayan. 2019. Sociotechnical Considerations for Accessible Visualization Design. https://doi.org/10.4 8550/arXiv.1909.05118 arXiv:1909.05118 [cs]. [63] Alan Lundgard and Arvind Satyanarayan. 2021. Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content. ht tps://doi.org/10.1109/TVCG.2021.3114770/ arXiv:2110.04406 [cs]. [64] Andrei Lăpus, teanu, Anca Morar, Alin Moldoveanu, Maria-Anca Bălut, oiu, and Florica Moldoveanu. 2024. A review of sonification solutions in assistive systems for visually impaired people. Disability and Rehabilitation. Assistive Technology 19, 8 (Nov. 2024), 2818–2833. https://doi.org/10.1080/17483107.2024. 2326590 [65] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, and Leah Findlater. 2021. What Do We Mean by “Accessibility Research”? A Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 371, 18 pages. https://doi.org/10.1145/3411 764.3445412 [66] Alexander Marquardt. 2023. Multisensory Guidance Under Sensory Constraints in Augmented Reality. Ph. D. Dissertation. Universitaet Bremen, Germany. https: //www.proquest.com/docview/3226986651?pq-origsite=gscholar&from openview=true&sourcetype=Dissertations%20&%20Theses [67] Kim Marriott, Bongshin Lee, Matthew Butler, Ed Cutrell, Kirsten Ellis, Cagatay Goncu, Marti Hearst, Kathleen McCoy, and Danielle Albers Szafir. 2021. Inclusive data visualization for people with disabilities: a call to action. interactions 28, 3 (April 2021), 47–51. https://doi.org/10.1145/3457875 [68] Roberto Martinez-Maldonado, Abelardo Pardo, Negin Mirriahi, Kalina Yacef, Judy Kay, and Andrew Clayphan. 2015. LATUX: an iterative workflow for designing, validating, and deploying learning analytics visualizations. Journal of Learning Analytics 2, 3 (2015), 9–39. https://doi.org/10.18608/jla.2 015.23.3 [69] Mille Nabsen Marwaa, Susanne Guidetti, Charlotte Ytterberg, and Hanne Kaae Kristensen. 2023. Using experience-based co-design to develop mobile/tablet applications to support a person-centred and empowering stroke rehabilitation. Research Involvement and Engagement 9, 1 (Aug. 2023), 69. https://doi.org/ 10.1186/s40900-023-00472-z [70] Claire Morley, Kim Jose, Sonj E. Hall, Kelly Shaw, Deirdre McGowan, Martina Wyss, and Tania Winzenberg. 2024. Evidence-informed, experience-based codesign: a novel framework integrating research evidence and lived experience
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
in priority-setting and co-design of health services. BMJ Open 14, 8 (Aug. 2024), e084620. https://doi.org/10.1136/bmjopen-2024-084620 [71] Claire Morley, Kim Jose, Sonj E Hall, Kelly Shaw, Deirdre McGowan, Martina Wyss, and Tania Winzenberg. 2024. Evidence-informed, experiencebased co-design: a novel framework integrating research evidence and lived experience in priority-setting and co-design of health services. BMJ Open 14, 8 (2024). https://doi.org/10.1136/bmjopen- 2024- 084620 arXiv:https://bmjopen.bmj.com/content/14/8/e084620.full.pdf [72] Omar Moured, Sara Alzalabny, Thorsten Schwarz, Bastian Rapp, and Rainer Stiefelhagen. 2023. Accessible Document Layout: An Interface for 2D Tactile Displays. In Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA ’23). Association for Computing Machinery, New York, NY, USA, 265–271. https://doi.org/10.114 5/3594806.3594811 [73] Mukhriddin Mukhiddinov and Soon-Young Kim. 2021. A Systematic Literature Review on the Automatic Creation of Tactile Graphics for the Blind and Visually Impaired. Processes 9, 10 (Oct. 2021), 1726. https://doi.org/10.3390/pr91 01726 [74] Nicola O’Brien, Ben Heaven, Gemma Teal, Elizabeth H Evans, Claire Cleland, Suzanne Moffatt, Falko F Sniehotta, Martin White, John C Mathers, and Paula Moynihan. 2016. Integrating Evidence From Systematic Reviews, Qualitative Research, and Expert Knowledge Using Co-Design Techniques to Develop a Web-Based Intervention for People in the Retirement Transition. Journal of Medical Internet Research 18, 8 (Aug. 2016), e210. https://doi.org/10.219 6/jmir.5790 [75] Zoe J. Oliver, Filipe Cristino, Mark V. Roberts, Alan J. Pegna, and E. Charles Leek. 2018. Stereo Viewing Modulates Three-Dimensional Shape Processing During Object Recognition: A High-Density ERP Study. Journal of Experimental Psychology. Human Perception and Performance 44, 4 (April 2018), 518–534. https://doi.org/10.1037/xhp0000444 [76] Arnaud Prouzeau, Maxime Cordeil, Clement Robin, Barrett Ens, Bruce H. Thomas, and Tim Dwyer. [n. d.]. Scaptics and Highlight-Planes: Immersive Interaction Techniques for Finding Occluded Features in 3D Scatterplots. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow Scotland Uk, 2019-05-02). ACM, 1–12. https://doi.org/10.1145/ 3290605.3300555 [77] Graham Pullan. [n. d.]. Visualization of Large Datasets of Simulations - An Infinite Canvas Approach. In AIAA SCITECH 2025 Forum. American Institute of Aeronautics and Astronautics. https://arc.aiaa.org/doi/abs/10.2514/6 .2025-1576 [78] Min Qi, Bao-lin Zhang, Guang-he Liang, Jie Wang, and Xin-ping Cai. 2007. 3D Modeling and Visualization of Geology Volume based on Geophysical Field Data. Data Science Journal 6, 0 (Oct. 2007). https://doi.org/10.2481/dsj.6.S652 [79] Lauren Race, Chancey Fleet, Danielle Montour, Lindsay Yazzolino, Marco Salsiccia, Claire Ferrari, Sheri Wells-Jensen, and Amy Hurst. 2023. Designing While Blind: Nonvisual Tools and Inclusive Workflows for Tactile Graphic Creation. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’23). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3597638.3614546 [80] Agis Abhi Rafdhi. 2024. Integrating Web Technologies with Augmented Reality. International Journal of Research and Applied Technology (INJURATECH) 4, 2 (Dec. 2024), 239–252. https://ojs.unikom.ac.id/index.php/injuratec h/article/view/16431 [81] M. Rautenhaus, M. Kern, A. Schäfler, and R. Westermann. 2015. Threedimensional visualization of ensemble weather forecasts – Part 1: The visualization tool Met.3D (version 1.0). Geoscientific Model Development 8, 7 (July 2015), 2329–2353. https://doi.org/10.5194/gmd-8-2329-2015 [82] David K. Raynor, Hanif Ismail, Alison Blenkinsopp, Beth Fylan, Gerry Armitage, and Jonathan Silcock. 2020. Experience-based co-design—Adapting the method for a researcher-initiated study in a multi-site setting. Health Expectations : An International Journal of Public Participation in Health Care and Health Policy 23, 3 (June 2020), 562–570. https://doi.org/10.1111/hex.13028 [83] Samuel Reinders, Matthew Butler, Ingrid Zukerman, Bongshin Lee, Lizhen Qu, and Kim Marriott. 2025. When Refreshable Tactile Displays Meet Conversational Agents: Investigating Accessible Data Presentation and Analysis with Touch and Speech. IEEE Transactions on Visualization and Computer Graphics 31, 1 (Jan. 2025), 864–874. https://doi.org/10.1109/TVCG.2024.3456358 arXiv:2408.04806 [cs]. [84] David Rojas, Bill Kapralos, Andrew Hogue, Karen Collins, Lennart Nacke, Sayra Cristancho, Cristina Conati, and Adam Dubrowski. 2013. The effect of sound on visual fidelity perception in stereoscopic 3-D. IEEE transactions on cybernetics 43, 6 (Dec. 2013), 1572–1583. https://doi.org/10.1109/TCYB.2013.2269712 [85] Robert E. Roth. 2021. Cartographic Design as Visual Storytelling: Synthesis and Review of Map-Based Narratives, Genres, and Tropes. The Cartographic Journal 58, 1 (Jan. 2021), 83–114. https://doi.org/10.1080/00087041.2019.1633 103 _eprint: https://doi.org/10.1080/00087041.2019.1633103. [86] Mustafa Saifee. [n. d.]. VR-Viz: Visualization System for Data Visualization in VR. ([n. d.]). https://aaltodoc.aalto.fi/handle/123456789/36113
CHI ’26, April 13–17, 2026, Barcelona, Spain
[87] Elizabeth B.-N. Sanders and Pieter Jan Stappers. 2008. Co-creation and the new landscapes of design. CoDesign 4, 1 (March 2008), 5–18. ht tps://doi.org/10.1080/15710880701875068 _eprint: https://doi.org/10.1080/15710880701875068. [88] JooYoung Seo, Sanchita S. Kamath, Aziz Zeidieh, Saairam Venkatesh, and Sean McCurry. 2024. MAIDR Meets AI: Exploring Multimodal LLM-Based Data Visualization Interpretation by and with Blind and Low-Vision Users. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’24). Association for Computing Machinery, New York, NY, USA, 1–31. https://doi.org/10.1145/3663548.3675660 [89] JooYoung Seo, Yilin Xia, Bongshin Lee, Sean Mccurry, and Yu Jun Yam. 2024. MAIDR: Making Statistical Visualizations Accessible with Multimodal Data Representation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA, 1–22. https://doi.org/10.1145/3613904.3642730 [90] Ather Sharif, Sanjana Shivani Chintalapati, Jacob O. Wobbrock, and Katharina Reinecke. 2021. Understanding Screen-Reader Users’ Experiences with Online Data Visualizations. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’21). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/34 41852.3471202 [91] Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and Jacob O. Wobbrock. 2022. VoxLens: Making Online Data Visualizations Accessible with an Interactive JavaScript Plug-In. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–19. https://doi.org/10.1145/3491102. 3517431 [92] Mark Simpson, Jiayan Zhao, and Alexander Klippel. [n. d.]. Take a Walk: Evaluating Movement Types for Data Visualization in Immersive Virtual Reality. ([n. d.]). [93] Alexa F. Siu, Mike Sinclair, Robert Kovacs, Eyal Ofek, Christian Holz, and Edward Cutrell. 2020. Virtual Reality Without Vision: A Haptic and Auditory White Cane to Navigate Complex Virtual Worlds. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.114 5/3313831.3376353 [94] Chris Snider, Aman Kukreja, Christopher Michael Jason Cox, James Gopsill, and Lee Kent. 2024. Mixed reality prototyping: a framework to characterise simultaneous physical/virtual prototyping. Proceedings of the Design Society 4 (May 2024), 775–784. https://doi.org/10.1017/pds.2024.80 [95] Hyemi Song, Sai Gopinath, and Zhicheng Liu. 2025. Intents, Techniques, and Components: a Unified Analysis of Interaction Authoring Tasks in Data Visualization. https://doi.org/10.48550/arXiv.2409.01399 arXiv:2409.01399 [cs]. [96] Maximilian Speicher, Katy Lewis, and Michael Nebeling. 2021. Designers, the Stage Is Yours! Medium-Fidelity Prototyping of Augmented & Virtual Reality Interfaces with 360theater. Proc. ACM Hum.-Comput. Interact. 5, EICS (May 2021), 205:1–205:25. https://doi.org/10.1145/3461727 [97] Ryo Suzuki, Abigale Stangl, Mark D. Gross, and Tom Yeh. 2017. FluxMarker: Enhancing Tactile Graphics with Dynamic Tactile Markers. https://doi.or g/10.48550/arXiv.1708.03783 arXiv:1708.03783 [cs]. [98] Anders Høyland Syvertsen. 2022. Tangible Scalar Fields. Master’s thesis. The University of Bergen. https://bora.uib.no/bora-xmlui/handle/11250/3 004277 Accepted: 2022-07-08T23:39:01Z. [99] Andrew Vande Moere and Stephanie Patel. [n. d.]. The Physical Visualization of Information: Designing Data Sculptures in an Educational Context. In Visual Information Communication (Boston, MA, 2010), Mao Lin Huang, Quang Vinh Nguyen, and Kang Zhang (Eds.). Springer US, 1–23. https://doi.org/10.1 007/978-1-4419-0312-9_1 [100] J. A. Wagner Filho, C.m.d.s. Freitas, and L. Nedel. [n. d.]. VirtualDesk: A Comfortable and Efficient Immersive Information Visualization Approach. 37, 3 ([n. d.]), 415–426. https://doi.org/10.1111/cgf.13430 [101] Qiming Wang and Bonita Saunders. [n. d.]. Web-Based 3D Visualization in a Digital Library of Mathematical Functions. In Proceedings of the Tenth International Conference on 3D Web Technology (New York, NY, USA, 2005-0329) (Web3D ’05). Association for Computing Machinery, 151–157. https: //doi.org/10.1145/1050491.1050513 [102] Qiming Wang and Bonita Saunders. 2005. Web-based 3D visualization in a digital library of mathematical functions. In Proceedings of the tenth international conference on 3D Web technology (Web3D ’05). Association for Computing Machinery, New York, NY, USA, 151–157. https://doi.org/10.1145/1050491.1050513 [103] Xiyao Wang, Lonni Besançon, Mehdi Ammi, and Tobias Isenberg. [n. d.]. Understanding Differences between Combinations of 2D and 3D Input and Output Devices for 3D Data Visualization. 163 ([n. d.]), 102820. https: //doi.org/10.1016/j.ijhcs.2022.102820 [104] Zhuohao Zhang, John R Thompson, Aditi Shah, Manish Agrawal, Alper Sarikaya, Jacob O. Wobbrock, Edward Cutrell, and Bongshin Lee. 2024. ChartA11y: Designing Accessible Touch Experiences of Visualizations with Blind Smartphone
CHI ’26, April 13–17, 2026, Barcelona, Spain
Users. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’24). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3663548.3675611 [105] Zhuohao (Jerry) Zhang, Haichang Li, Chun Meng Yu, Faraz Faruqi, Junan Xie, Gene S-H Kim, Mingming Fan, Angus Forbes, Jacob O. Wobbrock, Anhong Guo, and Liang He. 2025. A11yShape: AI-Assisted 3-D Modeling for Blind and Low-Vision Programmers. In Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’25). Association for Computing Machinery, New York, NY, USA, 1–20. https://doi.org/10 .1145/3663547.3746362 [106] Jack Zhao and Andrew Vande Moere. 2008. Embodiment in data sculpture: a model of the physical visualization of information. In Proceedings of the 3rd international conference on Digital Interactive Media in Entertainment and Arts (DIMEA ’08). Association for Computing Machinery, New York, NY, USA, 343– 350. https://doi.org/10.1145/1413634.1413696 [107] Kaixing Zhao, Sandra Bardot, Marcos Serrano, Mathieu Simonnet, Bernard Oriola, and Christophe Jouffrais. 2021. Tactile Fixations: A Behavioral Marker on How People with Visual Impairments Explore Raised-line Graphics. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3411764.3445578 [108] Liang Zhou, Mengjie Fan, Charles Hansen, Chris R. Johnson, and Daniel Weiskopf. 2022. A Review of Three-Dimensional Medical Image Visualization. Health Data Science 2022 (April 2022), 9840519. https://doi.org/10.3 4133/2022/9840519 [109] Jonathan Zong, Crystal Lee, Alan Lundgard, JiWoong Jang, Daniel Hajas, and Arvind Satyanarayan. 2022. Rich Screen Reader Experiences for Accessible Data Visualization. https://doi.org/10.48550/arXiv.2205.04917 arXiv:2205.04917 [cs]. [110] Jonathan Zong, Isabella Pedraza Pineros, Mengzhu (Katie) Chen, Daniel Hajas, and Arvind Satyanarayan. 2024. Umwelt: Accessible Structured Editing of MultiModal Data Representations. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA, 1–20. https://doi.org/10.1145/3613904.3641996
A
BRAINWRITING SESSION PROMPTS AND TASK-BASED QUESTIONS A.1 Session 1: Brainwriting Prompts (1) Which elements of the tactile experience feel essential for understanding the surface? This question was designed to focus the team on the core affordances of the physical prototype, identifying what made it effective for building a spatial mental model. (2) How might these translate into a digital interface? This prompts a direct translation of the previously identified essential traces of the physical prototype into potential digital features, bridging the gap between the physical and virtual. (3) What features or feedback would help reproduce the tactile clarity in a web interface?The aim was to focus on the specific quality of “clarity”, encouraging co-designers to think about the fidelity and type of feedback needed to make the digital experience as unambiguous as the physical one. (4) What risks do we face if these features aren’t included? This prompt helped us consider what would be lost in a poor translation. Our aim was to brainstorm the most critical design requirements by forcing a consideration of failure modes.
A.2
Session 1: Extended Task-Based Questions
Throughout Session 1’s probe exploration and brainwriting exercise, co-designers engaged with the following task-based questions, which were presented conversationally to elicit their exploration strategies, confusions, and insights: 3D Orientation Tasks.
Kamath et al.
• How do you know where you are in the plot when using the tactile probe versus the digital prototype? • What cues help you understand the axes in each version? • When you lose your orientation, what helps you recover your bearings? • How would you differentiate between the X, Y, and Z axes nonvisually? Landmark and Peak Finding Tasks. • How would you locate the highest point on the surface using the tactile probe? • How would you match this and locate the highest point using the digital prototype? • Can you find where the data changes most dramatically? What strategies do you use? • What features or feedback would help you quickly jump to salient points of interest? Comparing Local Maxima versus Global Trends. • How do you differentiate a single peak from an overall rising trend in the tactile probe? • What would help you compare multiple high points in the digital interface (key-binding or continuous sonification)? • When exploring the surface, how do you maintain awareness of the overall shape while examining local features? • What feedback mechanisms would support comparison between different regions? Gradient Tracing Tasks. • Can you follow the path of steepest ascent on the physical probe? How? • How would you trace a ridge or valley in the digital version? • What auditory or tactile cues would best indicate the direction and steepness of slopes? • How do you distinguish between gradual slopes and sharp cliffs? Identifying Occluded or Partially Hidden Features (for those with Residual Vision). • How do you explore areas that might be hidden from one perspective on the tactile probe? • What strategies help you uncover features that aren’t immediately apparent in the digital interface? • How would you know if there are features "behind" or "underneath" other parts of the surface? • What feedback would indicate overlapping or complex spatial relationships?
A.3
Session 2: Brainwriting Prompts
(1) Which tools, techniques, or mediums can help preserve embodied cognition digitally? This question was intentionally broad, using the term “embodied cognition” to encourage the team to think beyond simple keyboard interactions and consider how physical movement and presence could be integrated. (2) How can sound, text or haptics replicate the spatial clarity of the physical prototype - shifting from a web-based prototype to a spatially adaptive interface to support
Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
embodied cognition? This prompt focuses on the specific modalities (sound, text, haptics) and the goal of creating a “spatially adaptive interface”, pushing the team to ideate on how the system could respond to a user’s physical context. (3) What’s the most feasible and understandable step to move toward these interfaces? After the expansive ideation prompted by the first two questions, the idea was to ground the discussion in pragmatism, asking the team to identify a concrete, actionable first step toward their more ambitious, long-term visions.
A.4
Session 2: Extended Task-Based Questions
During Session 2’s feature validation phase, co-designers were asked to evaluate the newly implemented features through the following task-based questions: Reference Sonification and Orientation (DG3). • Does the fixed origin sound help you maintain orientation during exploration? • How useful is the replay mechanism when you become disoriented? • Can you use the reference point to navigate back to previously explored regions? • What additional reference points or landmarks would be helpful? Spatial Audio for Position and Depth (DG4). • How effectively does stereo panning convey horizontal position (X-axis)? • Can you perceive depth (Z-axis) through volume changes alone? • What makes spatial audio cues more or less salient during exploration? • How would additional audio cues (e.g., reverb, echo) enhance depth perception?
CHI ’26, April 13–17, 2026, Barcelona, Spain
• What physical input devices would best support 3D navigation? • How could haptic feedback enhance your understanding of data density or slope? • In a mixed reality environment, how would you use your body to explore the data space? • What’s the most important first step toward creating a spatially adaptive interface?
B
AI CHAT ASSISTANT IMPLEMENTATION
The multi-threaded AI chat assistant (viewable in Figure 5) leverages Gemini 2.5 Pro to support interactive data exploration. Since the WebGL plot supports cursor-based interactivity, a screenshot of the user’s current view was passed to the model alongside the raw dataset used by the VisualizationEngine.js from the Engine Layer. This allows the model to provide contextually relevant explanations or respond to targeted questions about specific data points and regions. To ensure robustness, a system of static responses was also implemented, enabling the assistant to provide fallback answers based on pre-computed dataset statistics if the live model connection fails. For users with access to the system repository, if they chose to run it locally rather than through the web, the tool supported integration with Ollama, allowing queries to be directed to a user’s local language model, including fine-tuned versions. The assistant was designed to automatically detect the most recent available model and establish a connection to facilitate seamless responses. Additionally, four example prompts were provided to guide users in framing queries to the assistant.
Configurable Buffer for Comparison (DG5). • How intuitive is the buffer selection process? • Does the aggregated playback help you understand regional patterns? • Can you effectively compare a saved buffer region to your current focus point? • What improvements would make the buffer more useful for analytical tasks? • How might the buffer support focused queries to the AI assistant? Multi-Perspective Auto-Play and Jump-to-Peak (DG1, DG2). • How do the auto-play overviews help you build an initial mental model? • Does jump-to-peak functionality help you efficiently locate salient features? • What additional automated exploration strategies would be valuable? • How do these features balance efficiency with user control? Future Embodied Interface Concepts.
Figure 5: AI Chat Interface