Continuing to study relevant papers for different approaches: TTT / Fast Weights: TTT-E2E ("End-to-End Test-Time Training for Long Context") https://arxiv.org/abs/2512.23675 Authors: Arnuv Tandon, Karan Dalal, Yu Sun et al. (Stanford, UC Berkeley, NVIDIA et al.) This model performs test-time updates (next-token prediction) on a sliding window Transformer and optimizes initialization through training-time meta-learning. The 3B model achieves near-full attention scaling over long contexts while maintaining constant inference latency (approximately 2.7 times faster with 128K context). In-Place Test-Time Training (《In-Place Test-Time Training》) https://arxiv.org/abs/2604.06169 Authors: Guhao Feng, Shengjie Luo, et al. (ByteDance Seed, Peking University) This projective directly uses the MLP output projection matrix of the existing Transformer as fast weights for drop-in updates, requiring no additional architectural modules. It aligns with next-token prediction and supports efficient chunk-wise updates. It offers improvements in long contexts (up to 128K). REFINE / ReFINE (《Reinforced Fast Weights with Next-Sequence Prediction》) https://arxiv.org/abs/2602.16704 This projective trains fast weights using next-sequence prediction + sequence-level rewards + RL (GRPO), instead of simply using next-token prediction. It can be applied to mid-training, post-training, and test-time training, improving long-range information preservation and tasks like LongBench. Falcon (Fast Weight Attention for Continual Learning) https://arxiv.org/abs/2608.27763 Authors: Yifan Zhang et al. (ByteDance Seed, Princeton et al.) Designs fast-weight attention from the perspective of online continuous learning, deriving a normalized delta rule and sliding window updates (Falcon-1/2/3 and their A variants) to improve the writing efficiency of fixed-size associated states. RoboTTT (RoboTTT: Context Scaling for Robot Policies) Authors: Yunfan Jiang et al. (NVIDIA, Stanford, UT Austin) https://arxiv.org/abs/2607.15275 Applies TTT/fast weights to robot VLA strategies, extending the visual-action context to 8K time steps (constant latency), improving long-term operations, single-video adaptation, and online error correction capabilities. Meta-learning (Meta-learning / Learning How to Learn): Nested Learning / Hope (《Nested Learning: The Illusion of Deep Learning Architectures》) Authors: Ali Behrouz, Vahab Mirrokni et al. (Google Research) NeurIPS 2025 / https://arxiv.org/abs/2512.24695 This approach views the model as multiple nested optimization loops with varying update frequencies: fast modules adapt to the current context, while slower modules preserve more stable knowledge. It proposes a Continuum Memory System + self-modifying modules. The Hope architecture performs better in language modeling, continuous learning, and long-context learning, mitigating catastrophic forgetting. SCoL (Self-Consolidating Language Models: Continual Knowledge Incorporation from Context) Authors: Zekun Wang et al. (Georgia Tech) arXiv: https://arxiv.org/abs/2605.07076 The model generates instructions on which layers to update based on the current context and is trained using meta-RL. It improves knowledge acquisition and retention in knowledge incorporation and long context consolidation, and tends to update layers with higher Fisher information. OPSD (On-Policy Self-Distillation) Article Description: Distills session experience back into the base model (guided by token distribution differences + trajectory, reward-free). Representative works include: Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (and subsequent variants, such as SR-OPSD, Purified OPSD, etc.). The core idea is to perform on-policy distillation on the same model under different conditions (students have no privileged information, teachers have reference solutions/trajectories, etc.) to transform experience into stable improvements.