Official Review · ICLR 2025

The paper titled "Reality Only Happens Once: Single-Path Generalization Bounds for Transformers" addresses a gap in the theoretical understanding of transformer models, particularly in non-i.i.d. settings commonly encountered in time-series data. Traditional statistical guarantees for transformers often rely on independent and identically distributed (i.i.d.) assumptions, which are seldom met in real-world applications such as natural language processing, finance, and reinforcement learning. This work extends the theoretical framework to single-path trajectories generated by bounded and exponentially ergodic Markov processes, providing non-asymptotic generalization bounds that converge at a rate of $\tetxrm{O}\left ( \frac{1}{\sqrt{N}}\right )$. The bounds are explicitly dependent on the transformer's architecture, including the activation function, number of self-attention heads, depth, width, and norm constraints. Addressing a Significant Gap: The paper tackles the under-explored area of providing statistical guarantees for transformers trained on non-i.i.d. data, which is highly relevant given the prevalence of sequential data in practical applications. Explicit Generalization Bounds: By deriving explicit bounds that depend on various architectural parameters of transformers, the paper offers valuable insights into how different components of the model influence its generalization capabilities. The paper situates itself well within the existing literature, contrasting its contributions with prior work on i.i.d. settings, in-context learning for linear transformers, and universal approximation theorems. This contextualization highlights the novelty and importance of the presented results. Although the bounds are explicit, the dependence on multiple architectural parameters may raise concerns about scalability. It would be useful to discuss how these bounds behave as transformers scale to very large sizes, which is common in modern applications. I would like to see more discussion about the limitations (or the lack of them) of the case $M=1$ in terms of how transformers are applied specifically to LLM data etc. The paper is done for the case $M=1$. For the case $M>1$, why can you apply the results of the paper for the augmented Markov chain $Y_n = ( X_{n-M+1}, \cdots , X_n )$. Here is a list of typos I found. However for the one in line 2462 I would appreciate a clarification. line 1124 the R^{(M)} should be R^M line 1371 the expectation should taken over \nu measure line 1409 On the statement of proposition there many typos, also in the proof of the proposition line 2462 (and can easily be seen to be Markovian since Brownian motion has the strong Markov property) line 2508 typos held, admitted -> holds admits.