Official Review · ICLR 2025

This paper aims to demonstrate that transformer-based language models not only learns to process input sequences consisting of discrete tokens, but are also able to "reason" about the continuous interpolation of discrete tokens along the temporal and spatial dimensions. The paper first presents a continuous generalization of the classical transformer model to make it accept spatially and temporally interpolated continous inputs. It then shows via exemplar analyses that one can manipulate the "temporal duration" of input tokens to control model prediction on simple reasoning tasks; it also shows that the "spatial interpolation" of input tokens, formulated as their weighted averages, also elicit meaning outputs from the tested language models. The authors therefore claim that LLMs process natural language in a way that is fundamentally different from human speakers. 1. The continuous generalizations of key transformer modules such as multi-head self-attention seem novel and theoretically sound. 2. The presented examples of controlled input manipulation and the resulting model predictions in section 4 are interesting and intuitively make sense. 1. Despite the novelty of the main arguments and the proposed continuous generalization of transformers, the paper did not provide suffcient empirical evidence and quantitative evaluation to support them. In particular, the analyses in section 4 are based on only a couple of example input sequences, with a very special type of prompt template. It would make this section more convincing if the authors could curate and evaluate LLMs on a dataset of "temporally" or "spatially" interpolated input sentences, and define a more general evaluation metric to capture the variation of model prediction when manipulating the time durations or spatial weights of target tokens. 2. The claim that "LLMs reason about language (continuously)" seems also to be an overstatement given the input- and output-level simplicity of the evaluation tasks (e.g. counting the mentions of certain entity type in a short sentence). I'm curious to see whether the model could still produce fluent outputs on more complex text generation tasks (e.g. long-form QA) if we manipulate input tokens as suggested in section 3. If not, it would suggest that LLMs in fact do not assign meaningful representations for the spatio-temporally interpolated counterparts of input token embeddings. For the single-token time continuity test in section 4.1, what would happen if you use input prompts with different entity names from the same category? (e.g. In the sentence "apple banana pear orange", how many fruits are mentioned?)