[OpenAI's Latest Model Astra Utilizes ByteDance's Architecture: Repeated Computation in the Same Transformer Layer] Beating AI Newsflash: OpenAI's upcoming model Astra employs a recurrent depth architecture, also known as a looped transformer. In a standard Transformer, each token generation passes through a fixed sequence of layers. Astra, however, allows the same segment of information to repeatedly pass through the same set of layers, performing multiple rounds of computation before outputting results. This approach was publicly explored by ByteDance last year. The Ouro model released by the Seed team is also a type of Looped Language Model, where a set of Transformer layers operates in a loop, embedding more computation within the model itself. This enables smaller models to perform more computation without needing to generate excessively long reasoning chains, achieving results closer to those of larger models. However, this architecture introduces a safety concern. Some reasoning occurs within internal hidden states, making it impossible for humans to view a complete textual record, which complicates efforts to verify whether the model has violated any rules through reasoning chains. As a result, OpenAI has limited the extent to which Astra employs recurrent depth, ensuring it retains readable reasoning chains and is preparing to incorporate additional CoT (Chain-of-Thought) monitoring. [Original Link]