Official Review · ICLR 2024

This paper introduces an innovative and parameter-efficient transformer construction, TINT, capable of simulating complex models, such as pre-trained transformers, and applying fine-tuning in-context via a single forward pass. 1. The paper details a novel transformer construction that enables forward and backward operations, as well as parameter updates, within a single inference pass, and without the need for weight adjustments in the TINT model. This method surpasses previous approaches in terms of parameter efficiency and the complexity of models it can simulate. 2. The TINT architecture's efficiency is corroborated through real-data evaluations. 1. The TINT model requires access to auxiliary model weights, which it uses in prefix embeddings. This dependency differs from some previous works where the transformer independently and implicitly learns the auxiliary model's weights (and architecture) from the provided in-context dataset. 2. Despite its parameter efficiency, the TINT model's reliance on prefix embeddings to access auxiliary weights may lead to longer input sequence, which could potentially reduce computational and memory efficiency during inference due to the transformer's sequence length dependency. 1. The capacity of the TINT structure in mimicking complex models like pre-trained transformers is well-documented, but is it equally adaptable to simpler auxiliary models, such as 2-layer MLPs? 2. Is the TINT's structure inherently dependent on the configuration of the auxiliary models it simulates, for instance in terms of number of layers, activation functions, or attention mechanisms? 3. The details of the Backward Module depicted in Figure 1 are unclear. Can you provide more insight into it, particularly regarding the connections between the Backward Module's inputs/outputs across layers and their relationship to the Forward Module's output?