The more merging, the more chaotic? Joint evaluation by Tsinghua and Jiaotong University reveals three fatal flaws in the memory system of large models According to Beating monitoring, Shanghai Jiao Tong University, Tsinghua University, MemTensor and other institutions jointly published a paper, which for the first time systematically evaluated 12 mainstream large-scale intelligent agent memory systems, including Mem0, Letta (formerly MemGPT) and Zep, from the perspective of data management. The research team proposed a four module analysis framework consisting of memory representation and storage, extraction, retrieval and routing, and maintenance, and quantified performance and cost on 11 datasets. The evaluation shows that currently there is no single memory architecture that can adapt to all workloads. Hybrid systems perform best in conversational question answering, while structured topological systems (such as graph/tree memory architectures) are the most reliable in single step fact recall, but difficult to handle temporal reasoning. Many memories that are only added face catastrophic decline during long-term operation; In time sensitive queries, the performance of original long context retrieval is even better than memory enhancement schemes, as standard semantic merging often destroys key timeline clues and triggers the illusion of the past. The experiment also revealed the performance of each component when disassembled. Traditional similarity retrieval experiences a sharp decline in accuracy as the time span lengthens. Although the fine-grained fact extraction of large models slightly improves retrieval accuracy, it can damage multi-step reasoning ability due to the gradual loss of information. Highly structured graph systems have several orders of magnitude higher index construction and query latency, but do not bring proportional improvements in accuracy. Research has shown that local maintenance is more cost-effective than global reconstruction, and conservative memory merging should be the default maintenance strategy. [Original link]