In Terms of Explainability: Refining Requirements for Self-Explainable Systems

In Terms of Explainability: Refining Requirements for Self-Explainable Systems Arno Leue

, Akhila Bairy

, Maike Schwammberger

arXiv:2608.06049v1 [cs.SE] 6 Aug 2026

Institute of Information Security and Dependability (KASTEL) Karlsruhe Institute of Technology Karlsruhe, Germany arno.leue@student.kit.edu, {akhila.bairy, schwammberger}@kit.edu

Abstract—Autonomous and software-intensive systems have been growing in occurrence, complexity, and assumed responsibility. Due to the high complexity of these systems, properties like transparency and explainability must be a focus of investigation. To date, no universally applicable definition and guide for the development of (self-)explainable systems exists. A need for explainability standards has already been recognized in the EU AI Act and the IEEE Transparency Standard 7001-2021. To address this need, we propose unified definitions in explainability based on an analysis and combination of existing definitions. Additionally, we present structured explainability requirements that are necessary to build (self-)explainable systems. By analysing the resulting taxonomy, we propose the incorporation of explanation goodness and thus correctness of explanations into the unified definitions. With our approach, we support the development of formal standards for (self-)explainable systems. Establishing such a uniform taxonomy also is a beneficial step towards certifying or auditing explainable systems. Index Terms—explanations, explainability, explainability engineering, explainability requirements, requirements engineering, explainable systems, taxonomy

I. I NTRODUCTION Intelligent and semi-automated systems are deployed in more and more application domains. These application domains range from smart factories and smart home applications to (semi-)automated cars or automated delivery robots. At the same time, these software-intensive systems become larger, more complex and more opaque [1], e.g. also due to an increase in Artificial Intelligence (AI) components. Due to this lack of transparency [2], it becomes even more difficult for stakeholders to understand aspects of system-behaviour [3]. Amongst other issues, a lack of understanding can contribute to unfairness or other negative impacts on the user [4]. Consequently, it is beneficial and essential for complex systems to explain their behaviour to ensure non-functional requirements, such as trust and understanding [3], [5]. The term explainability [6] is closely connected to the terms interpretability, comprehensibility, understandability, or transparency [5], [7]. For clarity, we will stick to the differentiation of these terminologies made by Arrietta et al. [8]. However, we generalise their focus on AI systems to more general types of software-intensive systems with the following notions: Understandability as the ability of a system to make a human understand (parts of) its behaviour and reasoning.

Comprehensibility as the ability of a system to present information in a human-understandable format. Interpretability as the ability of a system to provide meaning in understandable terms to a human. Explainability: as the ability of a system to provide information that helps a human understand a system. Explainability being implemented into a system of interest, results in a self-explainable system. Self-explainability itself is a crucial non-functional requirement that must be considered within system development processes [5]. Such requirements describe perceived needs of stakeholders or capabilities a system should have, and thus form the basis of quality software. In order to facilitate the engineering of the non-functional requirement of explainability, it is crucial to formally define the term explainability. The first contribution of the paper at hand is an analysis of existing explainability definitions and a synthesis of unified explainability definitions from this investigation. Moreover, it is evident that explainability is not a simple, isolated, requirement; instead, additional requirements must be considered that enable or support the concept of explainability. Our second contribution therefore is a structured analysis of requirements that support the nonfunctional requirement of explainability. We demonstrate our combined taxonomy and explainability requirements with a traffic management case study. This paper is structured as follows; in Sect. II, we investigate relevant state in different aspects of explainability. In Sect. III, we sketch our general approach. Our central contribution is twofold: a) We review explainability terminology and provide definitions for terms used in this paper in Sect. IV. b) We then derive and structure requirements that are necessary for integrating explainability into system design in Sect. V. We demonstrate our findings from Sects. IV and V within a case study from the traffic management domain in Sect. VI. We close this paper in Sects. VII, VIII and IX with a discussion, limitations, and a conclusion. II. S TATE OF THE A RT We investigate the current state of the art on explainability definitions, aspects, quality, and standardisation efforts. a) Definitions in Explainability: The research field of explainability has brought forward different definitions of

explanations and explainable systems [1], [4]–[6], [9]. Köhl et al. [6] explore explainability from the perspective of Requirements Engineering, based on which Chazette et al. [4], [5] propose a more detailed explainability definition. Bersani et al. [1] propose increasing levels of explainability and a metric for quantitatively measuring a system’s explainability at a certain level. Based on this, Schwammberger et al. [9] explore the integration of explainability levels into a framework for engineering run-time explainability. The existing definitions will be compared in more detail in Sect. IV. b) Aspects of Explainability: Sadeghi et al. [10] propose explanation categories and a taxonomy of situations that they refer to as cases that need explanations. Additionally, they provide examples for these types of cases in order to be used to guide the requirements elicitation for the explanation capabilities of systems. Furthermore, the proposed categories are envisioned as a means to extract explanation situations at the design time of a system. Schwammberger and Klös [11] investigate explanation models and propose an extraction and refinement process for generating such models from system and environment models. Bairy et al. [12] address the timing of explanations as a crucial aspect. They envision finding the optimal timing of an explanation, either before, during or after an explanandum, based on the attention level of the recipient. c) Explanation Quality and Evaluation: Brunotte et al. [13] observe that establishing standards for explainability allows for the creation of quality models for certification and auditing. To that end, Chazette et al. [2] propose a framework that links dependencies, characteristics, and evaluation methods for explainability requirements. Their results show that it leads to the construction of explanations that increase usage, acceptance and satisfaction [2]. Hoffman et al. [14] examine goodness of explanations and provide a checklist for it, based on which Schwammberger [15] proposes a definition for explanation correctness and goodness. The latter argues that trustworthiness of explanations can only be reached through a holistic Explainability Engineering process, which further complicates the achievement of goodness in explanations. d) Explainability Taxonomies: Similar to the quality framework by Chazette et al. [2], proposals for taxonomies also exist in the field of explainability. Kahn et al. [16], among others, propose a taxonomy for information. Building on this, Nunes and Jannach [17] and Nauta et al. [18] propose taxonomies for explanations. Several taxonomies for explainability have also been developed [19]–[21]; however, mainly with a focus on explainable artificial intelligence (XAI). Speith [22] examines the differences between taxonomies in XAI, and thus raises the need for standards again. e) Explainability Standardisation: Current standardisation efforts for intelligent and AI-supported software-intensive systems also include the need to integrate transparency and explainability mechanisms into such systems. This can, e.g., be found in the EU AI Act [23] and the IEEE standard 70012021 on transparency [24]. Apart from the need of integrating explainability into intelligent and software-intensive systems, a call for standards on how to integrate explainability into

systems can be observed in recent literature. As Brunotte et al. [13] conclude, one of the significant challenges related to explainability is the absence of a definitive and operational definition. Langer et al. [25] argue that the establishment of a parametrized definition or a meta-model of explainability could assist in defining aspects of explainability that are measurable and evaluable. Walke et al. [26] conducted an analysis of the EU AI Act, leading to the conclusion that it provides only abstract regulations, thereby making it challenging to define specific metrics for achieving explainability. III. R ESEARCH G OAL AND D ESIGN We motivate and structure the key contribution of this paper in this section. A key finding from the research landscape is that explainability is a crucial non-functional requirement that must be integrated into software development processes [23], [24], [27]. Furthermore, a wealth of different explainability definitions, aspects, and taxonomies has been introduced so far. However, the different taxonomies and definitions that we mention in Sect. II d) have their merits. The presence of numerous aspects of explainability in taxonomies and knowledge catalogues is of significant relevance. From a practical perspective, particularly from that of a software developer, functionalities such as explainability should adhere to established standards. The need for explainability also directly implies a need for some standards for integrating explainability into softwareintensive systems. Such a standard should combine insights from different research directions. Thus we conclude the following two research needs RN1 and RN2. RN1: (Self-) Explainability of software-intensive systems RN2: Standardisation in engineering explainability To address the research needs RN1 and RN2, we venture towards a standardised and combined explainability terminology that conforms with existing explainability requirements. The following research questions RQ1 – RQ3 guide our endeavour. RQ1: What are suitable definitions of explanations and explainability? RQ2: What explainability requirements are needed for the implementation of (self-)explainable systems? RQ3: How can insights from explainability requirements engineering (RQ2) be used to improve the proposed definitions? To answer our research questions, we first identify, combine and analyse existing terminologies in depth in Sect. IV-A. Based on existing definitions of explanations and explainable systems we develop unified explainability definitions in Sects. IV-B, IV-C, IV-D. In Sect. V-A, we then identify, compare and analyse existing explainability requirements. Based on these, we generate a taxonomy in Sect. V-B that emphasises interconnections between requirements for explanations based on definitions from existing explainability research and our own proposed definitions from Sect. IV.

We investigate whether our proposed taxonomy meets the demands of explainability through an exemplary case-study in Sect. VI. We use the insights from Sects. IV, V and VI to introduce a combined explainability definition in Sect. VI-B. To illustrate concepts within the paper, we present the following example scenario that serves as a running example for the remainder of the paper: Consider a smart coffee machine which is integrated into a smart home system. Residents can use the coffee machine via a smartphone application. Users can query status information and communicate demands to the machine both through the application or manually at it. IV. T ERMINOLOGY In Sect. II, we provide an overview of existing explainability terminologies. In this section, we delve deeper into these terminologies and derive a unified definition from them. In its simplest form, an explanation involves an explainer providing information about a matter that is of interest to an explainee, as visualised in Figure 1. The matter that is explained is usually referred to as the explanandum.

Fig. 1. A minimal representation of the connections between constituents in the context of an explanation, which consists of an explainer, an explanandum, and an explainee. The explainer is the one who explains, the explanandum is what is being explained, and the explainee is the one to whom is explained. (a) is the matter of interest, (b) the explainer realising this and therefore (c) explaining it (d) so that the matter is understood better.

Although this concept is only an abstraction of the process of explaining, it captures the essence of the key elements in an explanatory context. Applied to the example, the explainer and the explanandum are the two systems: usually, the explainer would be the smart home app, and the explanandum the smart coffee machine. Depending on the situation, the relationship of the explainer and the explanandum can also be switched around. Next, we take a closer look at the constituents of an explanation to clarify the terminology used in this paper. A. Constituents of an Explanation According to IEEE standard 729-1983, a system S is a collection of components organised to accomplish specific functionality [28]. Systems can be classified according to various aspects: one such aspect is the type of components involved (e.g. software or physical components). The combination of both software and physical components constitutes the class of cyber-physical systems (CPS). In more detail, a CPS is a collection of independently interacting components that primarily transmutes how we interact with the physical world [29]. Although systems in general are not limited to CPS, many explanation cases can be identified for CPSs. Example domains include robotics, autonomous driving and smart factories. The smart coffee machine can be classified as

a CPS. This is due to its ability to autonomously and discretely decide on the software level, as well as to produce coffee in real-time with hardware using sensors and actuators. The water tank filling with water and emptying (even at the same time) would be an example for a continuous CPS aspect. An explanandum X, also known as a phenomenon [30], is a matter to be explained. This can be a system in general, or a specific aspect of a system (e.g. its reasoning processes, inner logic, intention, behaviour, decision procedures or knowledge) [4]. Within our running example, an explanandum X could refer to the functionality of its internal decisionmaking algorithm, a specific aspect of coffee production, a self-cleaning routine, or a detail of its interaction with users. The need N for an explanation of an explanandum X originates in a lack of understanding of it by an entity [6]. Thus, an explainee, also known as addressee A, is a recipient of explanation [3], [6]. They are a subset of stakeholders H, where a stakeholder is anyone who has a vested interest in the entity. All addressees are stakeholders, but not all stakeholders are addressees, since not all stakeholders are concerned with receiving an explanation. Given that such a need N for an explanation is often expressed more than once, and potentially by several different individual explainees in a similar manner, these explainees are often classified into target groups G. Explanations that address a common need for an explanation are adapted to fit to a specific representative R of target group G that is equipped with the background knowledge and processing capabilities characteristic of the group [6]. Specific target groups for the smart coffee machine could be consumers, maintenance workers, and software developers. A specific representative of a group could be the owner of the coffee machine or an individual maintenance worker. Explainers refer to a system or specific parts of a system that supply its explainees with the needed information [4]. In more general terms, means M characterize the ability of an entity to provide and produce an explanation [1], [6], [9]. In addition to explanandum, explainer, and explainee, a context C and a goal of the explanation need to be considered. The context C is sometimes also referred to as the environmental entities [1] or ’environment’ in general [4]. In connection with systems as explainers, the context C of an explanation is set by entities from the environment that interact with the system, influence its behaviour and provide measurable factors that influence the observed explananda [9]. Thus, the explainee and the explanandum are also included in the explanation context C. A goal θ of an explanation for a stakeholder group is a target to be reached by the process of explaining to the stakeholder [9]. The goal might also be included in the context. In our example, the entities that are comprised in the context C are primarily human stakeholders, environmental conditions and the dimension of time as additional factor. Equally, the smart coffee machine system and the smart home system are also included in the explanation context. Goals could relate to consumer satisfaction or a consumers trust in the system. Economic and ecological optimisation, as well as minimising production delays, could also be (sub-)goals.

TABLE I C OMPARISON OF EXISTING DEFINITIONS OF E XPLANATIONS . Source

Definition: An explanation E for an explanandum X is a piece (of information I) that ...

Köhl et al. [6] Bersani et al. [1] Chazette [5] Schwammberger et al. [9]

... processed in context C makes any representative R of target group G understand X with respect to aspect Y. ... makes X interpretable by a target group G. ... contributes to the addressee A’s understanding of X in context C. ... makes X understandable by a target group G with respect to an explainability goal θ.

B. Definition of Explanation The literature that we analysed and compared in the previous section contains different definitions of explanations and explainability [1], [4]–[6], [9]. Additionally, we also indicated that different phrases are used for similar concepts. We briefly summarise and compare the core concepts of these different definitions of explanations in Table I. Different opinions exist among researchers w.r.t. the objectives that should be pursued with an explanation. As Miller [31] already pointed out, researchers from philosophy, social and cognitive sciences have tried to give an answer to the nature of explanations for ages. Some consensus seems to root in the fact that for most cases, there is not just one piece of information [32], [33] that, once transmitted, clears the explanation need completely [34]. With this, explaining is understood as a social process in which information I is transmitted multiple times in both ways, until the explanation need is sufficiently fulfilled [35]. As such, explainability is not a technical concept but tightly coupled to human understanding [6], and what makes a system explainable is the access to explanations [4]. Based on this, we adhere to the phrasing of a single explanation as a contribution to understanding as used by Chazette [5]. Note that this phrasing does not exclude the possibility that the goal of an explanation is to fully explain something. We propose to keep the aspect of such a goal θ in our definitions, as suggested by Schwammberger et al. [9]. This leads to a very central proposition in this paper: the definition of an explanation E. We derive the following proposition from the definitions of Köhl et al. [6], Schwammberger et al. [9] and Chazette [5] and visualise it in Figure 2. Proposition 1 (Explanation): Consider a representative R of a target group G. An explanation E is a piece of information I that contributes to the representative R understanding an explanandum X in a context C with an explainability goal θ.

Fig. 2. A visualisation for the proposed definition of an explanation E. The arrows are intended to be read as ’information’ I contributes to the understanding of an ’explanandum’ X by a ’representative’ R.

C. Definition of Explainable System In the following, we enhance the definition of an explanation E to that of an explainable system SE . Explainability refers to the ability to describe a system to other systems or humans, and thus can also be referred to as explainable systems. Hence, we use explainability and explainable systems interchangeably. The concept of explainability can be defined in such a manner that its application is not exclusive to a particular entity, but that it is accessible to any entity. We focus on a definition of explainability that is limited to systems, which is in line with the definitions from [1], [4], [6], [9]. We briefly summarise the existing definitions in Table II. In order to add the ability to explain to systems, the related works define a means M as an asset that enables producing explanations [1], [4], [6], [9]. Similarly, we adhere to the phrasing. The definition by Köhl et al. [6] differs from the other papers in that it specifies a particular aspect Y of the explanandum that is addressed by an explanation. In the other papers, ’aspect’ and ’explanandum’ are used interchangeably and referred to as either term [1], [4], [9]. Our definition aligns with the latter in that it encompasses the concept of an explanandum as either a system in general or a specific aspect of it. Analogue to three of the four definitions [1], [6], [9], our approach also entails the utilisation of the definition of explanations within the definition of explainability. Our definition diverges from using the term ’information’ by Chazette et al. [4], as we opt for the term ’explanation’ to articulate the content being communicated. In addition, it is necessary to address the utilisation of an explainability goal, as previously defined in the context of explanations. The following differentiation represents a change from all existing definitions: In the definition of explainability of systems, the explaining system and the explained system do not necessarily have to be the same. In the following, we therefore refer to an explanation system SE and an explanandum system SX . Since these two systems are each located in a context that does not necessarily have to be the same either, we also define the term of an explanation context CE and an explanandum context CX . Using our example, we illustrate the difference between the explanation and explanandum context. In the explanation context CE , Bob tries to get a coffee from the smart coffee machine by entering his order at the designated terminal. After several unsuccessful attempts, he gives up. Later, he receives a notification on his smart home app informing him that the cof-

TABLE II C OMPARISON OF EXISTING DEFINITIONS OF E XPLAINABLE S YSTEMS . Source

Definition: S is explainable if and only if ...

[6] [4] [1] [9]

... means M is able to produce an E in context C such that E is an explanation of X with respect to aspect Y, for target group G in context C. ... an entity E by giving information I enables an addressee A to understand an aspect X of S in context C. ... it is able by a means M to produce an explanation E of an explanandum X for a target group G in context C. ... it is able by a means M to produce an explanation E of an explanandum X for a target group G, with an explainability goal θ in context C.

fee order failed because the machine was still being cleaned. This situation provides the explanandum context CX . The interactions with the explanation system SE and explanandum system SX took place in two different environments, at differed times and with different entities involved. So the respective contexts are not the same. The approach of differentiating the two systems and contexts ensures that the definition of explainable systems is as general as possible. Additionally, it provides a suitable basis for defining self-explainable systems and the timing of explanations. We phrase the definition of an explainable system in Proposition 2 and provide a visualisation of it in Figure 3.

an explanation precedes the timing tX of its explanandum, this relation describes an ante-hoc explanation. We express this through tE < tX . Analogously, an explanation occurring posthoc (>) or during (=) an explanandum can be defined. Figure 4 illustrates the time relation between tE and tX . We provide the definition for this explanation timing in Proposition 3 and give an exemplary visualisation in 4. Proposition 3 (Explanation Timing): The explanation E of an explanandum X of system S can be timed either before (tE < tX ), during (tE = tX ) or after (tE > tX ) X happened.

Proposition 2 (Explainable System): A system SE in a context CE is able by a means M to produce an explanation E of an explanandum X of a system SX in a context CX for a target group G with an explainability goal θ.

Fig. 3. The proposed definition of explainable systems (XS), visualised. (a) is the observation of the explanandum, (b) refers to the expression of an explanation need. (c) is the process of delivering an explanation, after which (d), the understanding of the explanandum will be improved.

Based on Proposition 2, the definition of explainability now extends to a few special cases concerning the relation of the explanation system SE and the explanandum system SX . If both systems are identical, this is referred to as ’self-explainability’ (Proposition 2.1). Even though it is not necessarily the case, with a self-explainable system, the contexts CE and CX may also be identical. One could refer to this as a ’live’ selfexplainable system. Proposition 2.1 (Self-Explainable System): A system SX is self-explainable iff SX is explainable and SE = SX . D. Timing of Explanations As we indicated in Sect. II, explanations can be provided before (ante-hoc), during or after (post-hoc) an event occurred. Consequently, the timing of an explanation can be defined through the relationship between the explanation context CE and the explanandum context CX . In case that the timing tE of

Fig. 4. A sequential representation of a post-hoc explanation. Time is represented along the vertical axis, increasing from top to bottom. Both the explanandum context CX and the explanation context CE are depicted, as are the explanandum time tX and the explanation time tE . Due to tE > tX in this example, it is a post-hoc explanation. The interactions correspond to arrows (a)–(d) as shown in Fig. 3.

V. E XPLAINABILITY R EQUIREMENTS In previous research, a wealth of requirements for explainability were identified with different levels of detail [2], [16]–[18], [36], [37]. Based on these requirements, Leue [38] identified a subset of them regarding the explainability of software systems with a focus on the MAB-EX framework. It is evident that a substantial proportion of the requirements identified are of an optional nature and, consequently, are not universally necessary for self-explainable systems. Accordingly, we distilled nine general requirements through an analysis their results and discuss them in this section. Requirement R0, explainability in systems, is a high-level meta-requirement; all others are sub-requirements of this one. R0 Explainability: The explanandum system SX must be explainable for a target group G in a context C with respect to an explanandum X [6].

A. Requirements Elicitation In the following, we elicit the requirements that emerged from distilling the requirements proposed by Leue [38]. They are presented individually at this stage, without any interpretative structure being imposed. Thus, their enumeration is merely for reference. Interdependencies between the requirements are discussed in the subsequent Sect. V-B. An important abstract requirement for explainability is the existence of a dedicated means for the purpose of producing information. Depending on the context, such means may be subsystems or entire systems. This requirement can be referred to as explanation producibility (R1). R1 Explanation Producibility: The explanation system SE must have means M to be able to produce an explanation E [1], [4], [6], [9]. Furthermore, the product must also contribute to the understanding of a target group in the explanandum in order to fulfil the goals of an explanation. This is described in explanation understandability (R2). We will examine what it means to understand an explanandum in more detail in Sect. V-B. R2 Explanation Understandability: The information I produced by the explanation system SE must contribute to the understanding of the explanandum X by any representative R of the target group G [4], [6], [9]. Two aspects of information that are already present in existing taxonomies of information and explanations are the type of presentation (R3) (i.e., syntax) and the type of content (R4) (i.e., semantics). Both should be adaptable by the explanation system to a stakeholder in order to contribute as much as possible to the understanding of the explanandum. Necessary attributes that implicitly play a role here are the comprehension of presentation and the interpretation of content. To ensure these, more detailed aspects of information play a role: In terms of syntax, these are properties such as the type of modality used [2], [17], the amount of the types of modalities [2] and the general amount of the information [16], [18] as well as its coherence [18] and consistency [16], [18]. In terms of semantics, these include the relevance of information for the explanation goal [16], [18], the contrastivity [2], [18], complexity [16], [18], and completeness [16], [18] of information, and the style [2], [17] of it. However, in order to remain at a meta-level here, we have not listed these individually as sub-requirements. R3 Explanation Presentation: The presentation (syntax) of the explanation E produced by the explanation system SE must be comprehensible by any representative R of the target group G in order to have any contribution to its understanding [2], [3], [5], [16]–[18]. R4 Explanation Content: The content (semantics) of the explanation E produced by the explanation system SE must be interpretable by any representative R of the target group G in order to have any contribution to its understanding [2], [16]–[18].

Another property of explanations that could also be considered part of the adaptability of content is correctness. Providing correct explanations for all explananda that need explanation is not possible due to models and idealisations. In our case, we are dealing with explananda of systems that correspond to models from the ground up and can therefore be explained completely correctly by them. Explanation correctness (R5) is thus an achievable characteristic. It is also desirable, since only correct information ultimately provides the basis for fulfilling explanation goals such as trust. R5 Explanation Correctness: The explanation E produced by the explanation system SE must be deducible from provably correct system models and context models [15]. In addition to correctness, an explanation should also contribute measurably to the understanding of an explanandum as described above. This enables monitoring whether an explanation goal has already been achieved or whether further information is needed. Combined, these two characteristics can be defined as explanation goodness (R6). R6 Explanation Goodness: The explanation E produced by the explanation system SE must be correct and measurably help any representative R of target group G in understanding an explanandum X [14], [15]. Two further dimensions that play a role in the provision of explanations are the choice of the right moment (timing, R7) and the choice of the information included (scope, R8). The former can also be considered a subcategory of the presentation of information, while the latter can be understood as a subcategory of content. However, both have a considerable influence on the understanding of stakeholders, hence we list them here. R7 Explanation Timing: The explanation E produced by the explanation system SE should be adaptable in its timing, i.e. its provision before, during, or after explananda [12]. R8 Explanation Scope: The explanation E produced by the explanation system SE should be adaptable in its scope, i.e. providing local or global information on explananda [1]. Furthermore, a necessary aspect for the production of explanations is obtaining underlying information. In order to extract this, models (R9) of relevant entities involved in an explanation are required, e.g. users, systems, context, or explanation history. A system model can be an artefact from Software Engineering processes, e.g. architecture diagrams, communication protocols, etc, a context model describes the operating context of a system [11]. R9 Explanation Modelability: The representative R, context C, explanandum system SX and the explanation E itself should be modelable [3], [9], [11], [15]. While provably correct models of systems are achievable, the complete correctness of context models poses a problem in terms of idealisations. Either way, correctness is highly dependent on the type of model and should therefore be chosen carefully.

Fig. 5. Taxonomy of abstract explanation requirements. Edges represent connections between the requirements themselves and also to the definition of explainability. All requirements have ’Explanation’ as a prefix. The operator ⊎ is used as an indication that the following attribute is a property of the requirement it is mentioned with. The connections are not hierarchical, though the graph can be read from left to right, e.g., ”for ’Explanation Goodness’, ’Explanation Correctness’ is needed”, based on the definition of goodness as mentioned in this chapter. The requirements ’Explanation Timing’ and ’Explanation Scope’ are dashed, because they could also be modelled as attributes or sub-requirements of ’Explanation Presentation’ and ’Explanation Content’ respectively.

B. Requirements Taxonomy To find interconnections between the elicited requirements, we analysed conditions based on their wording or, if available, their definition itself. Starting with explainability (R0), its definition calls for means for producing an explanation and thus for the producibility (R1) of explanations. In addition, its transitive use of the definition of explanations requires the product (information) to contribute to the understanding of it, with which it implies understandability (R2). As already briefly touched on in Section IV, there is no uniform definition of understanding. However, as understanding is referred to as a process, we can make use of an abilities-based notion of understanding as proposed by Speith et al. [36], which can be conditionalised as: Condition 1 (Understanding): A representative of a target group understands an explanandum iff they possess one or several understanding-related abilities. With this, we can exploit at least two understandingrelated abilities that are useful in understanding information: comprehensibility and interpretability, i.e. understanding the presentation and the content of information. Additionally to their notion of understanding, Speith et al. [36] also proposed clusters of the understanding-related abilities, spanning from ’only’ recognising content all the way to designing it. From our point of view, the comprehension of information is located in the recognising cluster, and the interpretation of it somewhere in between the assessing and the design cluster. Thus, as a beneficial differentiation, it makes sense to highlight that for understandability (R2), it is necessary to be able to adapt the presentation (R3) and content (R4) of the explanation. There are more relevant aspects that are not directly connected to the understanding of an explanation, but also have an impact on how fast, good or complete it is understood. The timing (R7) and scope (R8) of an explanation are such that they are also interconnected with understandability (R2).

Another property of explanations that is of relevance is the correctness of the information that is used for an explanation, as only correct explanations are really wanted by stakeholders. In combination, both explanation correctness (R5) and understandability (R2) are required for explanations to be classified as ’good’. This is defined in explanation goodness (R6), which implies all three requirements to be connected as well. As a final requirement, the ability of systems to build models to derive explanations from is needed [3], [9], [11], [15]. This requirement, the explanation modelability (R9), is interconnected to several other requirements, as different models are proposed to be of use for different steps in the explanation derivation process [11], [15]. For the appropriate presentation of the syntax and semantics of content for a stakeholder, user, system, and context models are needed. Additionally, the correct timing of an explanation also implies the use of all these three models. Explanation correctness, and with that goodness as well, also needs systems and context models in order to deduce correct information, as explicitly stated in its definition. All of these interconnections are visualised in Figure 5. We explicitly do not want to highlight any hierarchy between them, although the connections can be interpreted as [left] needs [right]. Explainability (R0) is intentionally placed at the border of requirements for explanations, as it can be seen as a high-level meta requirement. We decided to nevertheless place it into the figure, in order to highlight the direct connections to other requirements. The set of requirements presented is meant to be a very high-level way of describing dimensions beneficial for explanations, and not as a complete set of requirements for the implementation of explainable systems. VI. C ASE -S TUDY We investigate whether our proposed taxonomy meets the demands of explainability through a case example in the following. For this purpose, we use the taxonomy of Sadeghi et al. [10], which structures explanation needs into different

categories. We do not use the different explanation categories to point out additional ones, but merely to highlight different user requirements for explanations. The setting of our exemplary case-study is a traffic management system at an urban intersection. The system controls traffic flow at the intersection and explains different situations and decisions to different involved stakeholders. A. Case Elicitation Overall, we investigate two different scenarios: in scenario S-1, we consider an autonomous vehicle with a passenger, Alice, and in scenario S-2, we consider an emergency vehicle, driven by Bob: S-1: On her way to work, Alice receives a notification in her autonomous vehicle. The management system informs her that due to construction works on her usual route, it would be faster to take a different route. Due to this, the car changes from the currently slower route to the new faster route. The system decided to contact Alice before she were to notice the problem herself so that time would be saved and a cognitive overload for Alice would be avoided. S-2: On his way to the hospital, the emergency vehicle driver Bob approaches an intersection where the traffic situation is not clearly perceivable from a distance. He contacts the management system to get a quick and reliable explanation of how to get through the junction as fast and safe as possible. In order to save time, the system generates the information as a route rendered into the routing service of his vehicle. Additionally, the system provides real-time images of the junction for Bob to be able to verify the planned route. This can help to increase Bob’s ability to trust into the traffic management system. B. Case Analysis The requirements identified for these two explanation cases can be classified into three categories: (a) technical requirements (e.g., the presence of an explainer and an explainee), which are not relevant for a broad approach to explainability, (b) abstract requirements, which describe generally relevant aspects, and (c) sub-requirements of those in b), that are not relevant at a high-level. As such, in the following, we focus on requirements that were categorised in b). Analysing the difference in presentation (R3) and timing (R7) of information provided in both cases, results in the requirement that both properties have to be dynamically adaptable to the stakeholder at runtime. This also implicitly highlights that information has to be producible (R1). It is evident that the content (R4) of explanations should be adaptable in a way that it should be understandable (R2) for the stakeholders involved. What is particularly emphasised in S-2 is that the explanation explicitly requested by the stakeholder must be based on correct information and should therefore also be correct itself.

The main reason why an explanation is requested in S-2 is Bob’s intent to cross the intersection safely. In order to enable and support trust in the system’s decisions, it must be able to prove that the information and the explanations are correct. This confirms the need for the requirement for correctness (R5) and, in combination with the required understandability (R2), also the goodness (R6) of explanations. In S-1 and S-2, it becomes evident that the explanation system (i.e. the traffic management system) has access to a model of the user to adapt the explanation in terms of presentation and content. This adaptation of the explanation can help to save mental space for the explainee. Additionally, the scope (R8) of the explanation must be adaptable. Furthermore, it is suggested that the environment and the explanandum system, or at least its explananda, are captured within corresponding models. Consequently, the utilisation of a model (R9) is necessary. VII. E XTENDING THE TERM OF E XPLANATION Comparing the results of the case study with the taxonomy of explainability requirements proposed in Sect. V-B, all of the requirements mentioned there are at least confirmed, especially the correctness of information. Our definition of explainability in Sect. IV is part of a basis for building connections between the requirements in the taxonomy. Despite the existence of a need for explanation goodness, and thus also for explanation correctness, explanation goodness is not directly linked to the definition of explainability. However, we emphasise that this link should be present. We therefore integrate explanation correctness and the measurability of understanding, and thus overall goodness, into the definition of explanations. In a transitive way, correctness and goodness are included in the definition of explainable systems. The definition of explanations is therefore phrased as in Proposition 1.1. Proposition 1.1 (Explanation): Provably correct Information I that measurably contributes to a representative R of a target group G understanding an explanandum X in a context C with an explainability goal θ. VIII. L IMITATIONS AND T HREATS TO VALIDITY Our work is exclusively based on qualitative data analysis. Consequently, there is a possibility that the results are affected by subjectivity during analysis. Thus, it is possible that some literature was not encountered when eliciting explainability and its requirements. To eliminate this uncertainty, one could collect data on the basis of a systematic literature research. Next, we briefly discuss threats to the validity of the main parts of our research in more detail. a) Terminology: In order to compare existing definitions of explainability without the need for a detailed semantic analysis, we limited our selection to definitions that focus on formal terms. The objective of this paper is not to provide a comprehensive comparison of all notions of explainability; as such, this would have been outside the scope. Nonetheless, our

structured requirement analysis serves as a frame for including explainability capabilities into system design processes. b) Explainability Requirements: In this paper, we propose an abstract set and taxonomy of requirements. However, we don’t focus on the completeness of all requirements for explanations. Instead, our main focus was on outlining the aspects in explainability, and identifying the structural connections between them. c) Case Study: Due to the small number of cases in the analysis of requirements from the stakeholder’s perspective in Sect. VI, not all requirements that could be used for self-explainable systems were analysed. However, a complete coverage of explanation requirements was not the goal in this research, but rather highlighting the importance of explanation correctness. IX. C ONCLUSION AND F UTURE W ORK By comparing and analysing existing terminology, we developed a unified terminology for explanations (Prop. 1) and explainable systems (Prop. 2), which we propose as a starting point for developing standards in explainability. In addition, we propose to integrate the requirements for explanation goodness into the definition of explanations (Prop. 1.1), even though they are not necessary for explanations. Namely, we consider explanations based on false information to be undesirable, not least in terms of the goals of explainability. By distilling requirements on explainable systems proposed by Leue [38] and comparing them with existing taxonomies on information and explanations, we propose a taxonomy of requirements (Figure 5). The requirements represent a minimal set that is needed for the implementation of (self-)explainable systems. Additionally, the structure of the taxonomy highlights the logical interconnections between requirements, that could help as an abstract guideline for future research. By analysing the structure of our requirements taxonomy (Figure 5), we highlight the need for explanation goodness to be incorporated into the definition of explanations. This need was validated in a brief case-study (Sect. VI), analysing explanation needs of stakeholders interacting with a management system. In a similar way, more research on explainability requirements could be used to further refine definitions in explainability. Based on the proposed definitions, future research should address further aspects of explanations in order to advance the development of explainability standards and requirements. To this end, a systematic literature review could be conducted to identify or include further aspects. Aspects from other disciplines should also be included, for example, from philosophy and the social sciences. ACKNOWLEDGMENTS The authors would like to thank the Ministry of Science, Research and Arts of the Federal State of Baden-Württemberg for the financial support of the project within the Innovation Campus Future Mobility.

D ECLARATION OF AI-A SSISTED T ECHNOLOGIES During the preparation of this work, the authors used DeepL (https://deepl.com) for translation from German to English and for editing assistance to improve grammar and readability. The authors reviewed and revised all outputs from the tool and take full responsibility for the final content of the work. R EFERENCES [1] M. M. Bersani, M. Camilli, L. Lestingi, R. Mirandola, M. Rossi, and P. Scandurra, “A Conceptual Framework for Explainability Requirements in Software-Intensive Systems,” in 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW), 2023, pp. 309–315. [2] L. Chazette, V. Klös, F. Herzog, and K. Schneider, “Requirements on Explanations: A Quality Framework for Explainability,” in 2022 IEEE 30th International Requirements Engineering Conference (RE), 2022, pp. 140–152. [3] M. Blumreiter, J. Greenyer, F. J. C. Garcia, V. Klös, M. Schwammberger, C. Sommer, A. Vogelsang, and A. Wortmann, “Towards self-explainable cyber-physical systems,” in Proceedings of the 22nd International Conference on Model Driven Engineering Languages and Systems Companion, ser. MODELS ’19 Companion. IEEE Press, 2021, p. 543–548. [Online]. Available: https://doi.org/10.1109/MODELS-C. 2019.00084 [4] L. Chazette, W. Brunotte, and T. Speith, “Exploring Explainability: A Definition, a Model, and a Knowledge Catalogue,” in 2021 IEEE 29th International Requirements Engineering Conference (RE), 2021, pp. 197–208. [5] L. Chazette, “Requirements engineering for explainable systems,” Ph.D. dissertation, Leibniz Universität Hannover, 2023. [Online]. Available: https://repo.uni-hannover.de/items/ e822e738-42fd-4ccf-99c7-4a289de1a652 [6] M. A. Köhl, K. Baum, M. Langer, D. Oster, T. Speith, and D. Bohlender, “Explainability as a Non-Functional Requirement,” in 2019 IEEE 27th International Requirements Engineering Conference (RE), 2019, pp. 363–368. [7] M. C. Buiten, L. A. Dennis, and M. Schwammberger, “A Vision on What Explanations of Autonomous Systems are of Interest to Lawyers,” in 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW), 2023, pp. 332–336. [8] A. Barredo Arrieta, N. Dı́az-Rodrı́guez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, and F. Herrera, “Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI,” Information Fusion, vol. 58, pp. 82–115, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1566253519308103 [9] M. Schwammberger, R. Mirandola, and N. Wenninghoff, “Explainability Engineering Challenges: Connecting Explainability Levels to Run-Time Explainability,” in Explainable Artificial Intelligence, L. Longo, S. Lapuschkin, and C. Seifert, Eds. Cham: Springer Nature Switzerland, 2024, pp. 205–218. [10] M. Sadeghi, V. Klös, and A. Vogelsang, “Cases for Explainable Software Systems: Characteristics and Examples,” in 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), 2021, pp. 181–187. [11] M. Schwammberger and V. Klös, “From Specification Models to Explanation Models: An Extraction and Refinement Process for Timed Automata,” Electronic Proceedings in Theoretical Computer Science, vol. 371, p. 20–37, Sep. 2022. [Online]. Available: http://dx.doi.org/10.4204/EPTCS.371.2 [12] A. Bairy, W. Hagemann, A. Rakow, and M. Schwammberger, “Towards Formal Concepts for Explanation Timing and Justifications,” in 2022 IEEE 30th International Requirements Engineering Conference Workshops (REW), 2022, pp. 98–102. [13] W. Brunotte, L. Chazette, V. Klös, and T. Speith, “Quo Vadis, Explainability? – A Research Roadmap for Explainability Engineering,” in Requirements Engineering: Foundation for Software Quality, V. Gervasi and A. Vogelsang, Eds. Cham: Springer International Publishing, 2022, pp. 26–32.

[14] R. R. Hoffman, S. T. Mueller, G. Klein, and J. Litman, “Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance,” Frontiers in Computer Science, vol. Volume 5 2023, 2023. [Online]. Available: https://www.frontiersin.org/journals/ computer-science/articles/10.3389/fcomp.2023.1096257 [15] M. Schwammberger, “From Explanation Correctness to Explanation Goodness: Only Provably Correct Explanations Can Save the World,” in Bridging the Gap Between AI and Reality, B. Steffen, Ed. Cham: Springer Nature Switzerland, 2025, pp. 307–317. [16] B. K. Kahn, D. M. Strong, and R. Y. Wang, “Information quality benchmarks: product and service performance,” Commun. ACM, vol. 45, no. 4, p. 184–192, Apr. 2002. [Online]. Available: https://doi.org/10.1145/505248.506007 [17] I. Nunes and D. Jannach, “A systematic review and taxonomy of explanations in decision support and recommender systems,” User Modeling and User-Adapted Interaction, vol. 27, no. 3, pp. 393–444, Dec 2017. [Online]. Available: https://doi.org/10.1007/ s11257-017-9195-0 [18] M. Nauta, J. Trienes, S. Pathak, E. Nguyen, M. Peters, Y. Schmitt, J. Schlötterer, M. van Keulen, and C. Seifert, “From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI,” ACM Comput. Surv., vol. 55, no. 13s, Jul. 2023. [Online]. Available: https://doi.org/10.1145/3583558 [19] G. Vilone and L. Longo, “Classification of Explainable Artificial Intelligence Methods through Their Output Formats,” Machine Learning and Knowledge Extraction, vol. 3, no. 3, pp. 615–661, 2021. [Online]. Available: https://www.mdpi.com/2504-4990/3/3/32 [20] D. Minh, H. X. Wang, Y. F. Li, and T. N. Nguyen, “Explainable artificial intelligence: a comprehensive review,” Artificial Intelligence Review, vol. 55, no. 5, pp. 3503–3568, Jun 2022. [Online]. Available: https://doi.org/10.1007/s10462-021-10088-y [21] M. Langer, D. Oster, T. Speith, H. Hermanns, L. Kästner, E. Schmidt, A. Sesing, and K. Baum, “What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research,” Artificial Intelligence, vol. 296, p. 103473, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0004370221000242 [22] T. Speith, “A Review of Taxonomies of Explainable Artificial Intelligence (XAI) Methods,” in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, ser. FAccT ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 2239–2250. [Online]. Available: https://doi.org/10.1145/ 3531146.3534639 [23] The European Parliament and of the Council, “Regulation (EU) 2024/1689 of laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act),” OJ, vol. L 1689, June 2024. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2024/1689 [24] IEEE Computer Society, “IEEE Std 7001-2021 - IEEE Standard for Transparency of Autonomous Systems,” Institute of Electrical and Electronics Engineers, pp. 1–54, 2022. [Online]. Available: https://ieeexplore.ieee.org/document/9726144 [25] M. Langer, K. Baum, K. Hartmann, S. Hessel, T. Speith, and J. Wahl, “Explainability Auditing for Intelligent Systems: A Rationale for Multi-

Disciplinary Perspectives,” in 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), 2021, pp. 164–168. [26] F. Walke, L. Bennek, and T. J. Winkler, “Artificial Intelligence Explainability Requirements of the AI Act and Metrics for Measuring Compliance,” in Wirtschaftsinformatik, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:264974754 [27] L. Chazette and K. Schneider, “Explainability as a nonfunctional requirement: challenges and recommendations,” Requirements Engineering, vol. 25, no. 4, pp. 493–514, Dec 2020. [Online]. Available: https://doi.org/10.1007/s00766-020-00333-1 [28] IEEE Computer Society, “IEEE Std 729-1983 - Standard Glossary of Software Engineering Terminology,” ANSI/ IEEE Std 729-1983, pp. 1–40, 1983. [Online]. Available: https://ieeexplore.ieee.org/document/ 7435207 [29] C. Konstantinou, M. Maniatakos, F. Saqib, S. Hu, J. Plusquellic, and Y. Jin, “Cyber-physical systems: A security perspective,” in 2015 20th IEEE European Test Symposium (ETS), 2015, pp. 1–8. [30] J. WOODWARD, “Scientific Explanation,” The British Journal for the Philosophy of Science, vol. 30, no. 1, pp. 41–67, 1979. [Online]. Available: https://doi.org/10.1093/bjps/30.1.41 [31] T. Miller, “Explanation in artificial intelligence: Insights from the social sciences,” Artificial Intelligence, vol. 267, pp. 1–38, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0004370218305988 [32] D. S. Krull and C. A. Anderson, “The Process of Explanation,” Current Directions in Psychological Science, vol. 6, no. 1, pp. 1–5, 1997. [33] M. El-Assady, W. Jentner, R. Kehlbeck, U. Schlegel, R. Sevastjanova, F. Sperrle, T. Spinner, and D. Keim, “Towards XAI: Structuring the Processes of Explanations,” in Proceedings of the Human-Centered Machine Learning Perspectives Workshop, HCML 2019, 05 2019, p. 12pp. [34] D. Mindlin, F. Beer, L. N. Sieger, S. Heindorf, E. Esposito, A.-C. Ngonga Ngomo, and P. Cimiano, “Beyond one-shot explanations: a systematic literature review of dialogue-based xAI approaches,” Artificial Intelligence Review, vol. 58, 01 2025. [35] K. J. Rohlfing, P. Cimiano, I. Scharlau, T. Matzner, H. M. Buhl, H. Buschmeier, E. Esposito, A. Grimminger, B. Hammer, R. HäbUmbach, I. Horwath, E. Hüllermeier, F. Kern, S. Kopp, K. Thommes, A.-C. Ngonga Ngomo, C. Schulte, H. Wachsmuth, P. Wagner, and B. Wrede, “Explanation as a Social Practice: Toward a Conceptual Framework for the Social Design of AI Systems,” IEEE Transactions on Cognitive and Developmental Systems, vol. 13, no. 3, pp. 717–728, 2021. [36] T. Speith, B. Crook, S. Mann, A. Schomäcker, and M. Langer, “Conceptualizing understanding in explainable artificial intelligence (XAI): an abilities-based approach,” Ethics and Information Technology, vol. 26, no. 2, p. 40, Jun 2024. [Online]. Available: https: //doi.org/10.1007/s10676-024-09769-3 [37] K. Sokol and P. Flach, “Explainability fact sheets: a framework for systematic assessment of explainable approaches,” in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, ser. FAT* ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 56–67. [Online]. Available: https://doi.org/10.1145/ 3351095.3372870 [38] A. Leue, “Validation of the MAB-EX Framework through Requirements Analysis and a Case Study,” Bachelor’s thesis, Karlsruhe Institute of Technology, Karlsruhe, Germany, June 2025. [Online]. Available: https://doi.org/10.5445/IR/1000190826

Related documents

• Subsumption in $\mathcal{FL}_{\bot \mathit{reg}}$ with TBoxes Is in ExpTime#1123087
• Specification Oracles#1071205
• Tell Me Why: Designing an Explainable LLM-based Dialogue System for Student Problem Behavior Diagnosis#134605
• Constructing Theorem-Level Questions: A Declaration Discipline for Bounded Inquiry#689598
• Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization#229570