Usability Analysis of Configurator User Interfaces with Multimodal Large Language Models
arXiv:2605.29456v1 [cs.SE] 28 May 2026
Sebastian Lubos1 , Alexander Felfernig1 , Damian Garber1 , Adnan Kraljić1 , Tarik Kraljić1 , Viet-Man Le1 , Thi Ngoc Trang Tran1 , Gerhard Leitner2 , Julian Schwazer2 , Doris Suppan3 , Reinhard Willfort3 , Ivan Dukic4 , Jeremias Fuchs4 , and Manuel Henrich5 Graz University of Technology, Inffeldgasse 16b/II, 8010 Graz, Austria [email protected] 2 University of Klagenfurt, Universitätsstraße 65-67, 9020 Klagenfurt, Austria 3 isn – innovation service network GmbH, Grabenstraße 231, 8045 Graz, Austria 4 Morgendigital GmbH, Jahnstraße 18, 6020 Innsbruck, Austria 5 UNiQUARE Software Development GmbH, Lannerweg 9, 9201 Krumpendorf, Austria 1
Abstract. Configuration is a key technology for tailoring complex software systems, services, and products. A successful application of configurators not only depends on technical correctness, performance, and domain modeling but also on their usability. While general usability heuristics are widely used, configurator-specific criteria and tool support for systematic user interface (UI) analysis are limited. This paper explores the use of multimodal large language models (MLLMs) for scalable and semi-automated usability analysis of configurator UIs. We synthesize 18 configurator-specific usability criteria from the literature and apply these criteria in an MLLM-based analysis of 16 real-world configurators. Each criterion is assessed individually to generate severity ratings for usability issues and actionable improvement suggestions. A review of the results confirms that MLLMs can reliably identify configurator-specific usability issues and provide domain-aware improvement recommendations. Although human validation remains necessary, this approach has the potential to significantly reduce the required effort to analyze configurator usability. Keywords: Configuration · Usability Analysis · Large Language Models · Multimodal Large Language Models · User Interface Evaluation
1
Introduction
Configurators are key components in software product lines and mass customization [5, 4]. They enable customers to tailor complex software, services, and products to individual user needs and often serve as the primary entry point for placing orders [1]. However, even if a configurator produces technically correct configurations, it may still fail to meet the expectations of its intended users [26].
2
S. Lubos et al.
Since customization is often performed by end users with limited domain expertise, the usability of configurator user interfaces (UIs) has to be assured [18]. Configuration-related research often focuses on the aspects of correctness, performance, and domain knowledge modeling [6, 29, 30]. However, the UI, as the primary interaction layer with end users, receives less attention. Configurator UIs help users to understand available options, express preferences, explore alternatives, construct valid solutions, and complete orders [18]. These functionalities are highly relevant for the successful completion of configuration sessions. According to ISO 9241-11:2018, usability is the extent to which users achieve specified goals effectively, efficiently, and satisfactorily [12]. In human–computer interaction (HCI), usability is a central design and evaluation concern [11]. To analyze the usability of software systems, usability heuristics, such as Nielsen’s heuristics [23], help to reveal potential usability issues. However, these heuristics address general issues but not configurator-specific challenges, such as large option spaces or complex compatibility constraints. Also, heuristic analyses require substantial manual effort and expertise. The lack of configurator-specific criteria and related tool support [1] is a probable reason for the absence of a systematic evaluation of configurator usability in practice. Recent advances in multimodal large language models (MLLMs) [33] offer new opportunities for scalable and semi-automated usability analysis. MLLMs can process textual descriptions alongside visual representations. Initial studies report promising results in general usability evaluations [9, 20, 25, 35], but their potential for assessing configurators remains unexplored. This paper addresses this gap. We review existing work to derive configuratorspecific usability criteria and apply them in an MLLM-based analysis of realworld configurators. We aim to understand whether a state-of-the-art MLLM can accurately detect configurator-specific issues and suggest related improvements. The contributions of this paper are the following: – We present a set of configurator-specific usability criteria grounded in existing research. – We introduce an MLLM-based usability-analysis approach applied to realworld configurators. – We evaluate the strengths, limitations, and implications of our approach for improving configurator UIs.
2
Usability Criteria for Configurators
Multiple guidelines describe rather general criteria for good usability in interactive software systems, including the ISO 9241-11:2018 standard [12], Nielsen heuristics [23], and platform-dependent guidelines such as the Human Interface Guidelines by Apple 6 or the UI Design best practices for Android 7 . While these 6 7
https://developer.apple.com/design/human-interface-guidelines https://developer.android.com/design/ui
Usability Analysis of Configurator UIs with Multimodal LLMs
3
criteria are also applicable to configurator UIs, they lack configurator-specific aspects. As no guideline considers the specific requirements of configurator users, we summarize the related literature on this topic and extract relevant usability criteria for configurators. To identify relevant studies, we conducted keyword searches in Google Scholar 8 and the proceedings of software engineering conferences in the ACM Digital Library 9 , using related terms such as “configurator usability”, “configurator user interfaces”, and “interactive product configurator ”. We screened the retrieved papers for relevance and applied backward snowballing by reviewing their references to find additional relevant studies. While this literature review was not systematic, it provides a solid basis for deriving configurator-specific usability criteria grounded in existing research. Rogoll and Piller [28] compared 12 configurators for mass customization in the apparel industry from the user (customer) perspective to identify good and bad practices. The authors emphasize that configurators must fulfill three major tasks for customers. Firstly, risk reduction and trust building, which is related to different aspects of uncertainty on the customer side. This involves a lack of knowledge about which solution fulfills their needs and what their needs actually are. Additionally, users do not have the opportunity to evaluate the configuration before placing their order. Secondly, the visualization of the configuration can directly mitigate this uncertainty by illustrating the design of the software, service, or product through drafts, renderings, or descriptions. Thirdly, the usability is directly responsible for the success or failure of a configurator, as it determines whether the configuration process can be completed successfully or will be aborted. Overall, a configurator must support users and reduce their uncertainty by building confidence and demonstrating competence, which in turn increases customer trust and the likelihood of ordering. Rabiser et al. [26] reviewed the features of academic and commercial configurators to identify key capabilities that guide users based on their needs. These identified features were implemented in a configurator, which was evaluated in a user study to derive general implications for tool developers. Related insights involve measures to tackle complexity challenges, for example, by hiding certain options depending on the current configuration step or by enabling option filtering. Additionally, navigation is supported by guiding users through the configuration in a predefined yet flexible process, allowing them to choose arbitrary configuration options at any time to avoid a feeling of forced selection. Also, the state of configuration should be immediately reflected by visualizing changes and ensuring the validity of the configuration throughout the process. Explanations help users to understand the process, while suggestions for resolving problems help them to comprehend and address dependency errors. Abbasi et al. [1] reviewed 111 real-world web sales configurators from various industries to highlight good and bad practices. Based on their review, they suggested guidelines for the efficient engineering of configurators and considered 8 9
https://scholar.google.com https://dl.acm.org
4
S. Lubos et al.
three essential dimensions, including rendering configuration options, constraint handling, and configuration process support. They highlight the need for selfexplanatory configurators that provide clear guidance during the configuration, including explanations of constraints and transparent communication of changes when dependencies are propagated. Additionally, they argue for the need of immediate consistency checks and flexible stateful navigation, which allows users to control the configuration process and keep selected properties unchanged. Trentin et al. [31] discuss the capabilities of sales configurators that help to avoid the product variety paradox, i.e., the risk that offering more product variety and customization results in decreased sales. These capabilities include: focused navigation, flexible navigation, benefit-cost communication, easy comparison, and user-friendly product space description. Focused navigation is about supporting an efficient configuration process that narrows down the search space fast. If users can start with options they are most certain about, they can invest more time in learning and deciding about uncertain properties. Flexible navigation aims to minimize the effort required to modify a configuration by allowing changes to previous steps without losing progress. Benefit-cost communication should explain the consequences of the available choice options in terms of what the customer gets and in terms of what the customer has to give (monetary and nonmonetary costs, e.g., delivery time). This helps to assess the utility of a configuration. Providing easy comparison of configured variants supports more efficient decision-making. A user-friendly product space description should focus on the needs and abilities of different potential customers and contexts of use. The positive effect of these capabilities on the creative and hedonic value perceived by users of the configurator was confirmed in a user study [24]. Leitner et al. [18] summarize five key design principles for configurator UIs: (i ) Customize the customization process by personalizing the configuration environment based on the background of the current user, for example, by providing a parameter-based interface for direct configuration of technical properties by domain experts, and a needs-based interface for non-experts that translates customer requirements into technical properties. (ii ) Provide starting points to enable users to begin with predefined configurations, and not only from scratch. (iii ) Support incremental refinement by highlighting tradeoffs between properties and alternatives to support users in identifying their preferences during the configuration process. (iv ) Exploit prototypes to avoid surprises by visualizing the configuration. (v ) Teach the consumer to expand their domain knowledge by explaining parameters, highlighting the constraints, and explaining conflicting requirements if no solution can be found. Leclercq et al. [17] reviewed car configurators, focusing on HCI principles and design flaws to identify the most frequent usability problems in these systems. Besides the violation of basic HCI principles (e.g., information overload, missing feedback), the most frequent problem was the inappropriate implementation of configurator-specific functionalities. Examples of identified issues were related to the navigation within the configurator and included missing state information, unrecognizable controls to proceed, and a lack of visualization. The configu-
Usability Analysis of Configurator UIs with Multimodal LLMs
5
ration itself lacked validity checks, which allowed incompatible user selections. Transparent explanations and suggestions to repair errors were missing. A user study by Leclercq et al. [16] identified that product validity and adequacy have the highest priorities for users of configurators. Usability and realtime visualization were the most important factors. Based on these insights, the authors suggest corresponding UI practices to support these priorities. These include systematically organizing large configuration spaces to avoid crowded UIs if many options are available, avoiding disabled configuration options, notifying users by highlighting changes and communicating dependencies, assisting and teaching users appropriately by explaining the impact of choices, providing parametrized product visualization by offering multiple visualization options, and customizing the configuration process to support different user expectations. Building on these foundations, we summarize the usability criteria that are relevant to achieving a good user experience with configurators. We organize these criteria into four categories, summarized in Tables 1–4. For all criteria, we provide a description that can be used to evaluate the extent to which a configurator application fulfills the criterion. The criteria in Table 1 summarize aspects related to the configuration process itself, in which users define their preferences and specify the relevant options. This involves aspects related to the presentation and selection of configuration options, comparison of alternatives, and prevention of invalid configurations.
Table 1: Evaluation criteria of the configuration process category. ID Criterion C1 Customized options
Description
Refs.
Does the configurator adapt the available options to [16, 18, meet different user profiles (e.g., expert vs. non-expert), 31] such as presenting a needs-based view or a parameter view, with a clear way to select the profile? C2 Organized Does the configurator help users with large option [16, 26] configuration spaces by utilizing explicit mechanisms (e.g., grouping, space search/filtering, multi-step configuration) to keep the number of simultaneously visible choices manageable? C3 Availability Can users revisit and change previously set options [1, 16, 26] of options without losing their current configuration state? C4 Auto-completion Does the configurator offer user-triggered auto- [1] completion that fills in the remaining required options with defaults? C5 Variant Can users keep multiple variants and compare them [18, 31] comparison (e.g., ranking by properties, side-by-side, or highlighting differences) to review trade-offs? C6 Error prevention Does the configurator prevent users from ending up [1, 17, 26] with an invalid configuration (e.g., by disabling incompatible options, auto-resolving conflicts, or requiring conflict resolution before proceeding)?
6
S. Lubos et al.
Table 2 summarizes criteria related to explanations within the system, which help users to understand the product domain, dependencies, and give support if errors occur. In Table 3, criteria related to the navigation within the configurator are summarized. These criteria assess whether users can navigate efficiently, transparently, and flexibly, based on their individual needs. Table 4 collects criteria regarding the visualization of the configuration, which help users to evaluate whether a configuration adequately fulfills their needs.
Table 2: Evaluation criteria of the explanation category. ID Criterion
Description
E1 Providing domain knowledge E2 Transparency of dependencies
Does the configurator provide in-context explanations [1, 18, 28, of options (e.g., tooltips or examples) that clarify mean- 31] ing and help users make informed choices? Does the configurator explain dependencies of options [17, 18, at decision time (e.g., ‘choosing X requires Y’, ‘this re- 31] moves Z’, impact on price/delivery) to highlight consequences of choices? Does the configurator explain why a configuration is in- [1, 16– consistent (e.g., highlighting which selected options are 18, 26] in conflict) using actionable, non-technical language? Does the configurator suggest actionable repair options [1, 18, 26] (e.g., ‘change X to Y or Z’ or ‘remove X’) to assist users in resolving errors with inconsistent constraints?
E3 Transparency of errors E4 Repair suggestions
Refs.
Overall, these usability criteria encompass a broad range of aspects that can help to improve the quality of user interaction with configurators. While these criteria can be used for the manual evaluation of configurator UIs, we utilize them to conduct an automated usability analysis with MLLMs to identify usability issues and generate explicit improvement suggestions. Some of the usability criteria are tightly coupled to specific features of a configurator, e.g., the auto-completion of configurations (see C4 in Table 1). However, they can be evaluated beyond identifying whether these features are present by considering how intuitive they are to use. In this sense, an issue can still be present if a feature is available but does not sufficiently support users in their interactions. This aligns with ISO 9241-11:2018 [12], which defines usability as the extent to which users achieve specified goals effectively, efficiently, and satisfactorily. While general usability criteria, such as Nielsen heuristics [23], remain helpful for identifying usability issues in configurators, combining them with domain-specific criteria helps to address the specific user needs for specialized configurator applications.
Usability Analysis of Configurator UIs with Multimodal LLMs
7
Table 3: Evaluation criteria of the navigation category. ID Criterion
Description
N1 Focused navigation
Does the configurator provide a clear, task-aligned se- [1, 17, 26, quence of steps that helps users reach a valid result 28] efficiently (e.g., using a relevant order of decisions or avoiding unnecessary steps)? Does the configurator support moving for- [1, 26] ward/backward using consistently placed, clearly labeled controls? Can users modify or undo earlier selections without los- [1, 26, 31] ing the current configuration state? Does the configurator clearly indicate the current step, [1, 17] remaining steps, and completion status (e.g., stepper or progress bar)? Can users save, name, or restore previous full configu- [31] ration variants during or after the session (e.g., version history or saved configurations)? Does the system offer practical starting points to sup- [18, 31] port different user needs (e.g., starting from predefined configurations or selecting properties to start with)?
N2 Manual step transition N3 Flexible navigation N4 Progress indication N5 Variant persistence N6 Starting points
Refs.
Table 4: Evaluation criteria of the visualization category. ID Criterion
Description
V1 Product preview
Does the configurator provide a product preview that [1, 17, 18, is continuously updated after changes to reflect the cur- 26, 28] rent configuration? Can users switch between preview modes (e.g., 2D/3D [16, 28] images or a textual summary), with a consistent relation to the current configuration?
V2 Customized preview
3
Refs.
MLLM-based Usability Analysis
Initial approaches for automated usability analysis used AI-based methods, such as heuristic checkers and rule-based approaches [2, 22]. These are typically limited in terms of the considered evaluation aspects and context-dependent issues [15]. More recently, multimodal large language models (MLLMs), i.e., models that combine textual and visual understanding [33], have shown new possibilities for usability analysis. The general idea is to provide screenshots of an application to be analyzed, together with instructions and specific evaluation criteria, to assess how well these criteria are satisfied [3, 9, 19, 20, 25, 35]. In many cases, the analysis generated by the MLLM had good overlap with assessments of usability experts. Nevertheless, their role remains supportive, and UI experts are still needed for further evaluation and validation [9, 20, 25, 35].
8
S. Lubos et al.
Up to now, most studies have focused on general usability criteria, for example Nielsen’s heuristics [23], for the analysis of different user interfaces, including design mockups [3], mobile apps [25, 35], and web applications [9, 20]. The usage of application-specific usability criteria was investigated in [19], where the analysis was tailored to specific criteria for recommender systems. It showed promising results in identifying and explaining related usability issues in user interfaces of recommender systems. In contrast to prior work, this work focuses specifically on configurator user interfaces and on usability criteria that reflect the specific challenges users face when interacting with them. Furthermore, we use screen recordings as the evaluation context, whereas previous studies rely on static screenshots [9, 19, 20, 25, 35]. Using videos instead of screenshots helps preserve interaction dynamics and UI state transitions. This enables a more comprehensive usability analysis. In line with previous work, we see significant potential in applying MLLMs to analyze configurator user interfaces and identify relevant usability issues. An automated analysis can potentially reduce the manual effort for these evaluations. The underlying idea is to provide an MLLM with all the data needed to analyze the usability of a configurator user interface. This includes example user interactions, specified evaluation criteria, and explicit task instructions. For the configurator application, a representative screen recording of a user interaction is provided as visual context. To include the evaluation criteria, we use the descriptions of the usability criteria described in Tables 1–4. While it would be possible to evaluate multiple criteria at once, we consider one criterion per prompt to ensure more focused and precise results. The task instruction provides detailed guidance and clearly defines the expected output. In many cases, the usability analysis of criteria is not a binary decision, but rather identifies issues with different levels of severity. To account for this, the MLLM is instructed to provide a severity rating on a 3-point scale (no issue, minor issue, major issue). Additionally, if an issue is identified, a description of the issue and an explanation of how it can be improved are requested. We split the instructions for the MLLM into a system prompt, which includes general instructions for the usability analysis (see Figure 1), and a user prompt template that defines the context for the current criterion (see Figure 2).
Fig. 1: System prompt for the usability analysis, providing general instructions.
Usability Analysis of Configurator UIs with Multimodal LLMs
9
Fig. 2: Prompt template for the usability analysis, where variable placeholders are indicated in braces “{}”. Result formatting instructions are omitted.
4
Methodology
4.1
Dataset
To evaluate our approach with real-world configurators, we selected 16 examples from the Cyledge configurator database [21]. We selected one configurator from each available industry category to obtain a broad sample across domains. Each configurator had to be publicly accessible, available in English, and still online at the time of the study. For each configurator, one of the authors interacted with the system10 to demonstrate its functionality and interaction flows. Although these recordings do not reflect full, realistic customer journeys, they illustrate the types of actions and interface elements that real users commonly engage with when configuring and ordering a product. As a basis for our usability analysis, we recorded these interactions as full-screen MP4 videos. Compared to static application screenshots, a screen recording captures interaction sequences and transitions, which provides a richer context for analysis. The selected configurator applications are summarized in Table 5, including the duration of each recorded interaction. 4.2
Model Selection
For the usability analysis, we used Google’s gemini-2.5-flash MLLM [8]. This model was suitable for evaluating recorded user interactions, as it supports the efficient processing of multimodal inputs, including video. Although larger models are available, we decided to use this model, as prior work indicated that 10
Using Google Chrome v143 on macOS v26
10
S. Lubos et al.
Table 5: Configurators used for the usability analysis, including the industry category, name, and duration of the recorded interaction in minutes. ID Industry
Name
1 Accessories Tie Creators 2 Apparel Clothoo 3 Beauty & Health eSalon 4 Electronics AimControllers 5 Food & Packaging Oreo 6 Footwear DIS 7 Games & Music Fender 8 House & Garden Ergohide 9 Industrial Goods Altrex 10 Kids & Babies Nuk 11 Motor Vehicles Aixam 12 Office & Merchandise Austrian Post 13 Paper & Books Packhelp 14 Pet Supplies 4Pets 15 Printing Platforms Namemaker 16 Sportswear & Equipment Aurum Bikes
Duration URL 05:22 05:47 04:19 08:44 04:44 05:49 04:17 03:41 01:23 04:02 02:49 03:20 07:39 01:13 01:32 02:15
tiecreators.com clothoo.com esalon.com aimcontrollers.com oreo.com designitalianshoes.com/en fender.com ergohide.com altrex.com/en nuk.de aixam.com onlineshop.post.at/en-AT packhelp.com 4pets-products.com/en namemaker.com aurumbikes.com
larger models do not necessarily yield more precise results for usability evaluation tasks [20]. Other models may produce different outcomes, and a performance comparison could reveal such differences. However, this paper focuses on demonstrating the feasibility of MLLM-based usability analysis for configurators, rather than optimizing model choice. A systematic comparison is out of scope. We accessed the model in Python via the Gemini Developer API.11 To obtain more deterministic outputs, we set the temperature to 0.0. The screen recordings were passed directly to the model using the default settings, which extract one frame per second.12 This choice balances cost and detail, as fewer frames reduce token usage13 and speed up analysis, but involve the risk of omitting relevant interactions, whereas more frames increase detail at higher cost. Context-token limits could become a problem for excessively long recordings, though the practical risk is low. With the default sampling rate, gemini-2.5-flash can process videos of at least one hour before reaching the token limit, which is sufficient for typical user interactions. However, if the limit is exceeded, a lower-quality tokenization or a lower frame sampling rate could be applied, or recordings could be split into smaller segments. The full implementation of the usability analysis, dataset with video samples, and complete analysis and evaluation results are available in our repository.14 11
https://ai.google.dev/gemini-api/docs https://ai.google.dev/gemini-api/docs/video-understanding 13 Each extracted frame contributes 258 tokens under default settings. 14 https://anonymous.4open.science/r/configurator-usability-analysis-2206/ 12
Usability Analysis of Configurator UIs with Multimodal LLMs
4.3
11
Study design
We used the recorded samples from the dataset (see Section 4.1) as input for the MLLM and obtained analysis results for the predefined usability criteria following the procedure outlined in Section 4.2. In the following, we present the MLLM-based analysis results descriptively and discuss detailed evaluation results. This study concentrates on the plausibility of MLLM-identified usability issues and related improvement recommendations. This aims to assess the practical utility of this approach. To assess the correctness and plausibility of the identified issues and suggested improvements, we engaged six reviewers with several years of academic and professional experience in software engineering to evaluate the analysis results. The descriptions of each MLLM-identified usability issue and suggested improvement were independently evaluated by exactly three reviewers. For every sample, the reviewers watched the screen recording and assessed the generated outputs against the relevant usability criterion using two binary judgments: – Issue description plausibility: Does the described issue constitute a genuine usability problem related to the criterion and the given configurator? – Improvement recommendation plausibility: Is the proposed improvement appropriate to fix the identified issue? We aggregate the three independent judgments by majority vote and report the inter-rater reliability to quantify agreement among reviewers.
5
Results
In total, 288 MLLM-based usability analyses were generated by evaluating 18 usability criteria across 16 configurators. In 140 cases, the results indicated no issue, while 73 minor issues and 75 major issues were identified. The execution times of the MLLM-based analysis ranged from 3.4 to 52.8 seconds per criterion (14.4 seconds on average), depending on the video duration. This corresponds to 135.6 to 636.7 seconds per configurator (258.4 seconds on average). Compared to manual usability analysis, these results indicate that MLLM-based analysis can be performed more efficiently. Across configurators, the number of identified issues (minor and major issues) ranged from 5 to 13 (6.75 on average). Figure 3 visualizes the distribution of issue severity across all analyzed usability criteria for each configurator. These results indicate that usability issues are common in configurator UIs. Additionally, the number of issues varies across configurators. This means that the MLLM considers the differences between configurators and provides individual results. Figure 4 summarizes how often issues were identified for each usability criterion. Certain criteria were fulfilled by most configurators, including availability of options (C3), transparency of errors (E3), manual step transitions (N2), starting points (N6), and product preview (V1). This indicates that these usability aspects are well established among configurator providers.
12
S. Lubos et al.
Fig. 3: Identified issues per configurator across all usability criteria. However, other criteria were frequently violated across configurators, especially variant comparison (C5) and variant persistence (N5). These closely related criteria appear to be commonly overlooked. It is possible that product comparison is often not perceived as part of the core configuration task. Nevertheless, improving this aspect could enhance the user experience. All analysis results in which the model reported minor or major issues were reviewed to evaluate their plausibility as described in Section 4.3. To assess whether the plausibility assessment was consistent across the three reviewers per issue, we computed the inter-rater reliability and report the results in Table 6. The observed agreement (Po ) [32] between reviewers was high and indicated substantial agreement. However, the Fleiss’s Kappa metric [7], which accounts for chance agreement, indicates poor agreement (κ ≈ 0). The reason for this discrepancy is the Kappa Paradox [14], in which extremely skewed ratings toward one option inflate the expected agreement by chance. The majority vote in our evaluation leads to 88.5% of issue descriptions considered plausible and 98.0% of improvement recommendations (see Table 7). Therefore, minor disagreement is heavily affecting the Fleiss’s Kappa values. Consequently, we also report Gwet’s AC1 [10], which is more robust to such skewed distributions and treats extremely dominant choices as intentional rather than by chance. These values indicate moderate agreement (0.4 < κ ≤ 0.6) for issue descriptions and almost perfect agreement (κ > 0.8) for improvement recommendations. Overall, the majority of issue descriptions and improvement recommendations were considered plausible based on a majority vote as reported in Table 7. The plausibility values did not show a notable difference between the assess-
Usability Analysis of Configurator UIs with Multimodal LLMs
13
Fig. 4: Identified issues per usability criterion across all configurators.
Table 6: Inter-Rater Reliability metrics for the review of MLLM-generated usability issue descriptions and improvement suggestions. Issue Improvement Description Recommendation Observed Agreement (Po ) Fleiss’ Kappa Gwet’s AC1
0.694 0.06 0.546
0.842 -0.007 0.813
ment of major and minor issues. Full agreement among all three reviewers on the same judgment was observed for 54.1% of the issue descriptions and 76.6% of the improvement recommendations. In the following, we discuss different usability analysis results in detail to further develop an understanding of their usefulness and current limitations. In our experiments, the MLLM consistently distinguished the severity of identified issues across all samples. If a criterion was only partially fulfilled, it was marked as minor issue. In contrast, if the criteria were completely unfulfilled, they were marked as major issues. Although this behavior is reasonable, assigning severity levels remains inherently subjective, as it depends on individual judgment rather than strictly objective criteria. Therefore, these labels should be used as a guide for grouping issues, not as a strict prioritization for issue resolution. Table 8 exemplifies some descriptions and suggested improvements of analysis results. Overall, these show that the generated issue descriptions were mostly
14
S. Lubos et al.
Table 7: Overall plausibility of MLLM-generated usability issue descriptions and improvement suggestions based on majority voting of reviewers. Issue Improvement Description Recommendation Minor Issues (n = 73) Major Issues (n = 75) All Issues (n = 148)
0.877 0.893 0.885
0.973 0.987 0.980
nuanced and tailored to the specific configuration domain. They mention specific details of the configurators, like “the ’Add LED light with epoxy diffusion’ checkbox lacks in-context explanations or tooltips” which allows users to easily locate the issue. Also, the descriptions reflect specific usability criteria, for example, “it doesn’t support managing multiple saved designs or a version history,” which is clearly related to the variant persistence (N5) criterion. The suggested improvements directly address the described issues and provide explicit instructions on how to resolve them, for example, “implement a ’Save Design’ feature within the configurator ” or “add brief tooltips or small ’i’ icons.” These recommendations specify concrete UI changes and also explain the underlying usability rationale, such as “this would enhance engagement and help users understand how their choices impact the final product.” In this sense, the suggestions do not only mention surface-level observations, but also express design interventions that can be implemented by developers. A review of the MLLM-generated analysis outputs that were considered implausible showed several common reasons that could guide future improvements of the analysis approach: – Unrecognized or misinterpreted controls: Sometimes, the MLLM failed to correctly recognize or interpret interface elements shown in the screen recordings. For example, an icon-only control for saving configuration variants was not recognized as such, and subtle changes in the product visualization were overlooked. This led to incorrect or unsupported conclusions. – Misinterpreted criteria: In some cases, the MLLM did not correctly apply the intended meaning of a usability criterion. For instance, an issue was reported for error prevention (C6) in the sense that incompatibility warnings were missing. However, the reason was that there were no incompatibilities in the example. This could be improved by extending the criteria descriptions with more detailed information on what should be observed. – Overly strict interpretations: In certain cases, the MLLM applied the criteria too literally and did not sufficiently include the configurator context. This was most evident for auto-completion (C4) and progress indication (N4). For example, the MLLM identified missing auto-completion buttons as an issue, even though users could add the configuration to the cart at any time and any remaining properties were automatically filled with default values,
Usability Analysis of Configurator UIs with Multimodal LLMs
15
Configurator Criterion Severity
Table 8: Example analysis results of configurator UIs and usability criteria, including issue severity, description, and improvement suggestions.
Description
Suggested Improvement
3 V1 Major The configurator lacks a continu- Integrate a dynamic product preissue ous product preview that updates view that visually reflects the user’s dynamically as the user makes se- selections in real-time or near reallections regarding their hair charac- time throughout the questionnaire. teristics (e.g., length, texture, oili- For example, as the user selects ness, condition). The product im- hair goals, the product bottles could age only appears towards the end visually highlight the chosen beneof the process for naming, and the fits or ingredients, or show a repfull personalized routine is only dis- resentation of the personalized forplayed on the final results page, sig- mula evolving. This would enhance nificantly delaying visual feedback engagement and help users understand how their choices impact the on the customization. final product. 4 N5 Major The configurator lacks any visi- Implement a ’Save Design’ feature issue ble functionality for users to save, within the configurator, allowing name, or restore different configu- users to store their custom configrations. While an ’Edit’ button in urations. Provide a ’My Saved Dethe cart allows returning to the con- signs’ section where users can view, figurator, it doesn’t support manag- name, edit, or load previously saved ing multiple saved designs or a ver- variants. Consider offering options sion history, which is crucial for to share or duplicate designs. a product with extensive customization options. 8 E1 Minor While detailed information is avail- Add brief tooltips or small ’i’ icons issue able for accessories via ’i’ icons, the next to the ’Board’ options and the initial ’Board’ options (e.g., ’With ’Add LED light with epoxy diffuextended cable space,’ ’With space sion’ checkbox. These should profor cables’) and the ’Add LED light vide concise explanations of what with epoxy diffusion’ checkbox lack each option entails, its benefits, or in-context explanations or tooltips. key characteristics (e.g., ’Epoxy difUsers must infer the differences fusion creates a soft, even light disfrom subtle visual changes or prior tribution’). knowledge, which can lead to uncertainty.
16
S. Lubos et al.
which aligns with the intended meaning of auto-completion. Similarly, the MLLM identified a missing progress bar as an issue, though the configurators indicated progress through step indicators. Adding more detail or examples to the criteria definitions could help to reduce such rigid interpretations. – Workflow mismatch: The MLLM generally applied the criteria more accurately to configurators that customize a single product, while the accuracy was weaker for configurators that filter multiple products to identify the best match for a user’s needs. In these cases, especially issues related to the product preview (V1) were incorrectly identified. Adapting the analysis instructions to better account for different configurator workflows could help mitigate this limitation. These observations also help to explain the counterintuitive finding that some improvement recommendations were considered plausible, even though the issue description was implausible (see Table 7). In these instances, the MLLM did not correctly recognize controls or interpreted the criteria too strictly, which means it did not capture a genuine issue in the description. Yet, the improvement recommendation was plausible and aligned with the criterion, such that it could be applied to further improve usability. Overall, our evaluation suggests that MLLMs offer promising support for usability analysis of configurators, as they effectively interpret screen recordings, apply domain-specific criteria, and generate detailed, context-aware descriptions and improvement recommendations. While further evaluations are still required, as not all results accurately describe genuine issues, the presented approach can substantially reduce evaluation effort.
6
Discussion
Our results demonstrate that MLLMs can meaningfully enhance the usability analysis of configurators by identifying diverse issues with relatively low effort. The model generated domain-aware explanations, provided differentiated assessments across systems, and offered explicit suggestions for improvement. This indicates that configurator-specific usability criteria can be analyzed using multimodal input context, which creates promising opportunities for integrating automated analysis into real-world development workflows. From a practical perspective, the findings highlight several implications for improving configurator UIs. Firstly, attention can be directed to frequently overlooked aspects, such as variant comparison and variant persistence. The automated detection can help providers identify these recurring gaps early and encourage them to provide additional support for users. Secondly, the detailed recommendations generated by the model suggest actionable refinements, such as adding clarifying tooltips, enhancing visual feedback, or providing clearer progress indicators. Such guidance can accelerate iteration cycles, especially in teams without dedicated usability experts.
Usability Analysis of Configurator UIs with Multimodal LLMs
17
Additional development is needed to further reduce implausible outputs. This includes refining prompt instructions, for example, by providing few-shot examples [27], specifying criteria more precisely with explicitly observable measures, and adapting to different configurator workflows more effectively. Structured improvements in these areas, combined with systematic evaluation, could reduce the number of false positives and further improve the quality of results. Automated usability analysis of configurators using MLLMs has the potential to create a substantial practical impact. These tools could continuously monitor usability throughout design and development, providing rapid feedback in each iteration to identify overlooked issues early. Keeping human validation in the loop ensures that automatically identified issues are interpreted correctly and that configurator quality is not compromised by false positives or misaligned recommendations. A current boundary is the need to manually generate screen recordings. While this approach still reduces the effort of the actual usability analysis, the use of intelligent agents to simulate user behavior and generate recordings could further automate the evaluation [34]. However, these approaches are not yet accurate enough for controlled study settings. While our current approach focuses on textual improvement suggestions, these improvements could be accompanied by concrete UI code changes if a configurator’s code repository is accessible [13]. This way, the MLLM acts as an assistant that translates usability findings into implementation-level guidance to support developers in a semi-automated way. Overall, the results suggest that MLLM-based usability analysis can evolve from feasibility to practical utility, enabling more accessible, scalable, and systematic improvements to configurator UIs.
7
Threats to Validity
Our analysis relies on 18 configurator-specific usability criteria derived from related studies. However, there may be additional relevant criteria that were missed in our literature review, for which the analysis does not work as expected. We attempted to mitigate this issue by employing a broad, diverse range of criteria aggregated from related literature. The screen recordings used as input represent simulated rather than real user interactions. Real interactions could raise other issues during analysis. We tried to mitigate this aspect by demonstrating a broad range of potential user interactions in the recordings, which should provide sufficient context for the analysis. Nevertheless, future studies should validate the results with interaction recordings of a more representative number of users. An assessment of completeness, i.e., whether all existing issues in the configurators were detected, was beyond the scope of this work, which focused on the correctness of the identified issues. Nevertheless, completeness is a complementary aspect for understanding the extent to which MLLMs identify usability issues compared to usability experts. We therefore plan to address this aspect in
18
S. Lubos et al.
future work by comparing MLLM outputs with additional evaluations conducted by usability experts. Finally, the study evaluated 16 configurators of different industries. Although this diverse sample provides a broad range of examples, it remains limited in size and may not capture the full variability of configurator designs, interaction paradigms, or domain conventions. A broader dataset would strengthen external validity and support more robust generalization of the findings.
8
Conclusions
This work investigated how multimodal large language models (MLLM) can support the usability analysis of configurators. Building on prior work, we derived 18 configurator-specific usability criteria and applied them to 16 real-world configurators using an MLLM and screen interaction recordings as contextual input. The model identified multiple issues per configurator. A review involving multiple software engineers found that most reported issues were plausible and accompanied by useful, domain-specific improvement suggestions. Practically, our results indicate that MLLMs offer a low-effort approach to analyzing the usability of configurators, although a follow-up expert-based validation remains necessary. With further refinement, such analyses could be integrated as continuous, automated checks in configurator engineering workflows. Our future work includes refining the criteria and prompts to further reduce the detection of implausible usability and further improve result quality. We also plan to expand the scope of our evaluations to better account for aspects of completeness and integration in software development processes.
References 1. Abbasi, E.K., Hubaux, A., Acher, M., Boucher, Q., Heymans, P.: The anatomy of a sales configurator: an empirical study of 111 cases. In: Proceedings of the 25th International Conference on Advanced Information Systems Engineering. p. 162–177. CAiSE’13, Springer-Verlag, Berlin, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38709-8_11 2. Castro, J.W., Garnica, I., Rojas, L.A.: Automated tools for usability evaluation: A systematic mapping study. In: Meiselwitz, G. (ed.) Social Computing and Social Media: Design, User Experience and Impact. pp. 28–46. Springer International Publishing, Cham (2022) 3. Duan, P., Warner, J., Li, Y., Hartmann, B.: Generating automatic feedback on ui mockups with large language models. In: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. CHI ’24, Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3613904.3642782 4. Felfernig, A., Hotz, L., Bagley, C., Tiihonen, J.: Knowledge-based Configuration – From Research to Business Cases. Morgan Kaufmann (2014) 5. Felfernig, A.: Standardized configuration knowledge representations as technological foundation for mass customization. IEEE Transactions on Engineering Management 54(1), 41–56 (2007). https://doi.org/10.1109/TEM.2006.889066
Usability Analysis of Configurator UIs with Multimodal LLMs
19
6. Felfernig, A., Falkner, A., Benavides, D.: Analysis of Feature Models, pp. 45–72. Springer International Publishing, Cham (2024). https://doi.org/10.1007/978-3031-61874-1_3 7. Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychological bulletin 76(5), 378 (1971) 8. Gemini Team Google: Gemini: A family of highly capable multimodal models (2024), https://arxiv.org/abs/2312.11805 9. Guerino, G., Rodrigues, L., Capeleti, B., Mello, R.F., Freire, A., Zaina, L.: Can gpt-4o evaluate usability like human experts? a comparative study on issue identification in heuristic evaluation. In: Human-Computer Interaction – INTERACT 2025: 20th IFIP TC 13 International Conference, Belo Horizonte, Brazil, September 8–12, 2025, Proceedings, Part III. p. 381–402. Springer-Verlag, Berlin, Heidelberg (2025). https://doi.org/10.1007/978-3-032-05005-2_20 10. Gwet, K.L.: Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology 61(1), 29–48 (2008). https://doi.org/https://doi.org/10.1348/000711006X126600 11. Hewett, T.T., Baecker, R., Card, S., Carey, T., Gasen, J., Mantei, M., Perlman, G., Strong, G., Verplank, W.: Human-Computer Interaction, pp. 5–29. ACM, New York, NY, USA (1992) 12. International Organization for Standardization: ISO/IEC/IEEE International Standard - Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts. ISO/IEC/IEEE 9241-11:2018(E) (2018) 13. Jiang, J., Wang, F., Shen, J., Kim, S., Kim, S.: A survey on large language models for code generation. ACM Trans. Softw. Eng. Methodol. 35(2) (Jan 2026). https://doi.org/10.1145/3747588, https://doi.org/10.1145/3747588 14. Konstantinidis, M., Le, L.W., Gao, X.: An empirical comparative assessment of inter-rater agreement of binary outcomes and multiple raters. Symmetry 14(2) (2022). https://doi.org/10.3390/sym14020262 15. Kuric, E., Demcak, P., Krajcovic, M., Lang, J.: Systematic literature review of automation and artificial intelligence in usability issue detection (2025), https: //arxiv.org/abs/2504.01415 16. Leclercq, T., Abbasi, E.K., Dumas, B., Remiche, M.A., Heymans, P.: Essential expectations of users of web configurators: An empirical survey. Proc. ACM Hum.Comput. Interact. 6(EICS) (Jun 2022). https://doi.org/10.1145/3534519 17. Leclercq, T., Cordy, M., Dumas, B., Heymans, P.: On studying bad practices in configuration uis. In: Joint Proceedings of the ACM IUI 2018 Workshops (2018), https://ceur-ws.org/Vol-2068/wii1.pdf 18. Leitner, G., Felfernig, A., Blazek, P., Reinfrank, F., Ninaus, G.: Chapter 8 - user interfaces for configuration environments. In: Felfernig, A., Hotz, L., Bagley, C., Tiihonen, J. (eds.) Knowledge-Based Configuration, pp. 89–106. Morgan Kaufmann, Boston (2014). https://doi.org/10.1016/B978-0-12-415817-7.00008-6 19. Lubos, S., Felfernig, A., Garber, D., Le, V.M., Tran, T.N.T.: Towards llm-based usability analysis for recommender user interfaces. In: Proceedings of the 12th Joint Workshop on Interfaces and Human Decision Making for Recommender Systems (IntRS 2025) (2025), https://ceur-ws.org/Vol-4027/paper7.pdf 20. Lubos, S., Felfernig, A., Garber, D., Leitner, G., Schwazer, J., Henrich, M.: Investigating multimodal large language models to support usability evaluation (2026), https://arxiv.org/abs/2508.16165 21. cyLEDGE Media: Configurator database (2022), https://www. configurator-database.com/, accessed: 04 December 2025
20
S. Lubos et al.
22. Namoun, A., Alrehaili, A., Tufail, A.: A review of automated website usability evaluation tools: Research issues and challenges. In: Soares, M.M., Rosenzweig, E., Marcus, A. (eds.) Design, User Experience, and Usability: UX Research and Design. pp. 292–311. Springer International Publishing, Cham (2021) 23. Nielsen, J.: Enhancing the explanatory power of usability heuristics. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. p. 152–158. CHI ’94, Association for Computing Machinery, New York, NY, USA (1994). https://doi.org/10.1145/191666.191729 24. Perin, E., Trentin, A., Forza, C.: The effect of sales configurator capabilities on the value perceived by the customer through the customization process. In: Proceedings of the 15th International Configuration Workshop (2013), https://ceur-ws.org/ Vol-1128/paper10.pdf 25. Pourasad, A.E., Maalej, W.: Does GenAI Make Usability Testing Obsolete? . In: 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). pp. 675–675. IEEE Computer Society, Los Alamitos, CA, USA (May 2025). https://doi.org/10.1109/ICSE55347.2025.00138 26. Rabiser, R., Grünbacher, P., Lehofer, M.: A qualitative study on user guidance capabilities in product configuration tools. In: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. p. 110–119. ASE ’12, Association for Computing Machinery, New York, NY, USA (2012). https://doi.org/10.1145/2351676.2351693 27. Reynolds, L., McDonell, K.: Prompt programming for large language models: Beyond the few-shot paradigm. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. CHI EA ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3411763.3451760 28. Rogoll, T., Piller, F.: Product configuration from the customer’s perspective: A comparison of configuration systems in the apparel industry. In: Proceedings of the International Conference on Economic, Technical and Organisational aspects of Product Configuration Systems (PETO 2004), Lyngby, Denmark. pp. 179–199 (2004) 29. Siegmund, N., Kolesnikov, S.S., Kästner, C., Apel, S., Batory, D., Rosenmüller, M., Saake, G.: Predicting performance via automated feature-interaction detection. In: Proceedings of the 34th International Conference on Software Engineering. p. 167–177. ICSE ’12, IEEE Press (2012) 30. Thüm, T., Apel, S., Kästner, C., Schaefer, I., Saake, G.: A classification and survey of analysis strategies for software product lines. ACM Comput. Surv. 47(1) (Jun 2014). https://doi.org/10.1145/2580950 31. Trentin, A., Perin, E., Forza, C.: Sales configurator capabilities to avoid the product variety paradox: Construct development and validation. Computers in Industry 64(4), 436–447 (2013). https://doi.org/https://doi.org/10.1016/j.compind.2013.02.006 32. Viera, A., Garrett, J.: Understanding interobserver agreement: The kappa statistic. Family medicine 37, 360–3 (06 2005) 33. Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., Chen, E.: A survey on multimodal large language models. National Science Review 11(12) (Nov 2024). https://doi.org/10.1093/nsr/nwae403 34. Yoon, J., Feldt, R., Yoo, S.: Intent-driven mobile gui testing with autonomous large language model agents. In: 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). pp. 129–139 (2024). https://doi.org/10.1109/ICST60714.2024.00020
Usability Analysis of Configurator UIs with Multimodal LLMs
21
35. Zhong, R., McDonald, D.W., Hsieh, G.: Synthetic heuristic evaluation: A comparison between ai- and human-powered usability evaluation (2025), https://arxiv. org/abs/2507.02306