Author: Hugo, Robonaissance

Translation: Deep Tide TechFlow

Deep Tide Guide: From early Atari game agents to large language models with millions of context windows, artificial intelligence consistently faces a fundamental physical contradiction when making continuous decisions over time: models can easily accommodate vast static human knowledge, but they cannot carry all past details intact in dynamic interactive environments.

Under the hard constraints of limited computing power and context windows, how to accurately extract the core state that determines the future while compressing the past is not only a breakthrough to solve “perception confusion,” but also the underlying logic for autonomous agents to achieve true persistent memory.

The State Paradox in a Frozen Frame

This is the third issue of the series “Memory, Forgetting, and Self in Machines”: when agents cannot carry all of the past with them, how does intelligence act across time.

Try to freeze a classic Pong game in any frame.

The scene only has two paddles, a ball, and the current score. At this moment, if you were to say where the ball is flying, you simply cannot, and even if you show the same static image to a machine, it also cannot determine.

Scholars Matthew Hausknecht and Peter Stone pointed out directly in their 2015 paper on learning to play Atari games with agents: “The current single-frame screen can only reveal the paddle and ball positions, yet it cannot reveal the ball's speed.”

They went on to explain why this is crucial: “Knowing the ball's direction of movement is key to determining the optimal position of the paddle.”

This frozen frame is precise, with every detail in the correct position. However, based solely on this single frame, it is still insufficient to play the game.

The sensor captures only an "Observation" at this moment.

The real global state that the agent's next action relies on is the "System State."

In Pong, the observation and the system state are severed in every single frame slice. Because the direction of the ball's movement is a fact about the past few moments, no isolated moment can contain it.

When the immediate input cannot satisfy the decision, a well-performing system must carry certain information from its past to the present.

However, what it carries can never be the complete past. What it carries is merely a summary, and this summary's selection entirely depends on what it needs to predict next.

This selection determines what kind of informational foundation is available in subsequent moments. Any detail omitted from the summary will forever hinder the ability to react based on it in future moments.

Memory Trade-off in Four Frames

The system that Hausknecht and Stone responded to was the DQN agent developed by Volodymyr Mnih and others at DeepMind, published in Nature magazine in 2015.

DQN can learn to play dozens of Atari games purely through screen visuals, and the authors faced the freeze-frame dilemma early in their development.

In the methods section of the paper, they wrote: “Since the agent can only observe the current screen, this task falls into the category of partially observable states, and many simulator states are perceptually aliased.”

In other words, relying solely on the current screen cannot adequately inform the agent of the complete situation.

A system that only looks at images is forced to interpret them as the same situation; if these two situations require completely opposite actions, the system will inevitably make a mistake in one of them.

Their remedy was very straightforward. The network no longer receives just a single screen image, but instead overlays several recent frames together as input.

With the default setting of 4 frames, although the algorithm can also adapt to 3 or 5 frames.

After placing four frames side by side, the ball's movement directly appears as a change in position. The direction of movement is no longer a concept requiring abstract memory but becomes a visual property that can be seen directly.

The paper did not refer to this stacked set of frames as memory. It is essentially just an input channel that is wider than a single moment.

Perception Confusion and Physical Hard Boundaries

This input width has an extremely rigid physical boundary, which Hausknecht and Stone accurately described.

They wrote: because DQN is trained with inputs comprising the four most recent states experienced by the agent, “if a game requires players to remember events from more than the past four screen frames, DQN will be unable to master the game.”

For DQN, the history beyond the fifth-to-last screen frame does not exist within its input range.

This fixed sliding window mechanism directly reveals the core contradiction of machine intelligence in processing time series: no matter how broad the window is, information beyond the boundary will be physically erased.

免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到[email protected]，本平台相关工作人员将会进行核查。