When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?

There exists little mathematical theory explaining why Transformer architectures are empirically successful even for non-Markovian data. To this end, this paper studies next-token prediction via stochastic optimal control theoretically trained Transformers for such sources, where we consider either