Pretesting Flow Questionnaire Design Using Eye-Tracking: An Exploratory Study

The aim of this study is to evaluate an online self-administered questionnaire for measuring flow, using eye-tracking. We were specifically interested in objectively monitoring when, where and what individuals look at and also in quantifying their visual attention while completing an online flow questionnaire, using the direct and the indirect measurement approaches. Flow is the holistic sensation that people feel when they act with total involvement (Csikszentmihalyi, Larson & Prescott 1977). The main flow measurement methods involve direct and indirect approaches, using questionnaires. Eye-tracking has been used in the field of survey methodology by scholars to infer the cognitive processing of visual layout, instructions, and items. We ran an experiment with 43 participants, using Gazepoint GP3 eye-tracker. After eye-tracking data validation, we obtained 36 valid observations. A series of Wilcoxon Signed-ranks tests indicated a significant difference between direct and indirect flow measurement AOI, based on the time to first fixation, average fixation, time viewed, number of revisitors, and number of average revisit metrics. The main contribution of our study consists in outlining that the indirect measurement of flow requires more time and a higher cognitive effort to be processed than flow description used in direct measurement procedures. Thus, scholars should use the appropriate flow measurement approach and consider participants' willingness and ability to process the text. Finally, we conclude that eye-tracking is a useful method in pretesting self-administered questionnaires.