Cognitive bias benchmarks should incorporate insights from ecological rationality

Recent studies suggest that large language models (LLMs) exhibit a range of cognitive biases familiar from human cognition. This paper argues that cognitive bias benchmarks should incorporate insights from ecological rationality, a leading paradigm in the study of cognitive bias. Three strategies for resisting some existing allegations of cognitive bias are extracted from the ecological rationality paradigm. Each strategy is used to generate a concrete recommendation for improving cognitive bias benchmarks.