Author: Hugo, Robonaissance

Compilation: Shenchao TechFlow

Senchao guide: From early Atari game agents to large language models with millions of contextual windows, artificial intelligence always faces a fundamental physical contradiction when making continuous decisions over time: the model can easily accommodate the vast static knowledge that humans already possess, but it cannot carry all past details intact in a dynamic interactive environment.

Under the hard constraints of limited computing power and contextual windows, how to accurately extract the core state that determines the future while compressing the past is not only a technical breakthrough to resolve "perceptual confusion," but also the underlying logic for autonomous agents to achieve true persistent memory.

The State Paradox in a Still Frame

This is the third issue of the series "Memory, Forgetting, and Self in Machines": when agents cannot carry all their past, how does intelligence act across time?

Try to freeze-frame the classic Pong ball game.

The screen only shows two paddles, a ball, and the current score. At this moment, if you were asked where the ball is flying, you simply cannot answer, and even if the machine is shown the same static image, it cannot determine that either.

Scholars Matthew Hausknecht and Peter Stone pointed out directly in their 2015 paper on learning to play Atari games that "the current single-frame screen can reveal only the positions of the paddles and the ball but cannot reveal the ball's speed."

They then explained why this is crucial: "Knowing the direction of the ball’s movement is a key factor in determining the optimal position of the paddle."

This freeze-frame is precise; every detail in the image is where it ought to be. But based solely on this single frame, it is still insufficient to play the game.

What the sensor captures in this moment is merely an "Observation."

What the agent's next action depends on is the true global "State."

In the Pong game, observations and system states are completely separate on every single frame slice. This is because the direction of the ball's movement is a fact about the past few moments, and no isolated moment can encapsulate it.

When the immediate input cannot satisfy the decision, a well-performing system must carry some information from its past to the present.

However, what it carries can never be the complete past. What it carries is only a summary, and the trade-offs in that summary completely depend on what it needs to predict next.

This trade-off determines what kind of informational foundation the subsequent moments will have. Any detail omitted in the summary will forever prevent the subsequent moments from reacting based on it.

Four-Frame Memory Trade-off

The system that Hausknecht and Stone responded to was the DQN agent published by DeepMind's Volodymyr Mnih et al. in Nature in 2015.

The DQN is capable of learning to play dozens of Atari games purely through screen images, and the authors faced the challenge of freeze-frames early in development.

They wrote in the methods section of their paper: "Because the agent can only observe the current screen, this task falls under partially observable states, and many simulator states are perceptually aliased."

In other words, relying solely on the current screen image cannot inform the agent of the complete situation.

The term “perceptual aliasing” is an accurate academic descriptor of a common dilemma. Two entirely different scenarios (e.g., the ball is flying toward your paddle, or the ball is flying away from your paddle) can produce the exact same image in a static frame.

A system that only sees the image is forced to consider them as the same situation; if these two situations require completely opposite actions, the system will inevitably make an error in one of them.

Their remedy was very straightforward. The network no longer received just a single screen image but instead stacked together several recent frames as input.

In the default setting, this value is set to 4 frames, although the algorithm is also adaptable to values of 3 or 5 frames.

By placing four frames side by side, the motion of the ball is directly represented as a change in position. The direction of movement is no longer a concept that requires abstract memory but becomes a visual attribute that can be directly observed.

The paper does not refer to this set of stacked images as memory. It is essentially just a slightly wider input channel than that of a single moment.

Perceptual Confusion and Physical Hard Boundaries

This input width has exceedingly hard physical boundaries, which Hausknecht and Stone precisely described.

They wrote: Because DQN is trained using inputs consisting of the most recent four states the agent has experienced, "if a game requires the player to remember events beyond the last four screens, DQN will not be able to master that game."

For DQN, the history beyond the fifth-to-last screen simply does not exist within its input range.

This fixed sliding window mechanism directly reveals the core contradiction in machine intelligence regarding time series processing: no matter how wide the window is opened, information beyond the boundary will be physically erased.