Why can the recent update of implicit caching in large models reduce cache prices and greatly improve hit rates. In traditional reasoning, every time you call the API, the GPU has to recalculate the entire input prompt. Fixed System Prompt, historical dialogue, RAG knowledge base, these duplicate contents are pre filled each time and charged at full price. What does implicit caching do? It automatically detects the common prefix of this input and previous requests. Once hit, load the previously calculated KV Cache directly from the SSD hard drive, and only recalculate the newly added portion. Repetitive labor, cut off directly. The effect is immediate. Multiple rounds of dialogue Agent、 In scenarios such as code completion, repeated prefixes often account for 70% -90% of the input token. After hitting, the input price will be directly discounted by 10%. The cost and memory pressure of reasoning through multiple conversations on long texts are greatly reduced. But here's a key question: why wasn't it possible before? KV Cache is too large. Under the traditional Multi Head Attention architecture, a KV with millions of tokens may require hundreds of GB of video memory. save? I/O latency is completely unacceptable. GPU must hold all KV throughout the entire process, and the cost cannot be reduced at all. DeepSeek's MLA architecture compresses KV Cache by 10-28 times. Million tokens have increased from 80-200GB to 4-10GB. This makes hard disk level caching possible, and compressed KV can be economically stored on distributed SSDs and quickly loaded when needed. Essentially, it changes the most expensive GPU memory from being fully occupied by the maximum context to being dynamically used based on actual newly added tokens, similar to a dynamic update mode. There is a counterintuitive point here: although the demand for high-speed memory has significantly decreased, it has not completely shifted towards hard drives. More like tiered storage, GPUs only retain hot data and offload cold prefixes to SSDs. Similar to CPU paging mechanism, but this time it occurred at the inference layer of the large model. DeepSeek was the first to enable this mechanism by default and offer the ultimate low price. Other manufacturers have to follow suit, as competition drives technology diffusion, which further lowers prices. What needs to be done to maximize cache hits? To maximize cost savings, try to place duplicate content at the beginning of the prompt and maintain prefix consistency, which will result in a higher hit rate. There is a prompt_cache_cit_token field in the API response, which can be directly used to check the hit rate. This round of price reduction is a real architectural innovation, and the cost reduction brought about by algorithm efficiency optimization. MLA compression KV, distributed SSD storage, and Radix Tree prefix indexing, these engineering breakthroughs have optimized the previously wasted computing power that required repetitive labor to the extreme Especially DeepSeek, can you still make money by lowering the price so much, or permanently lowering the price? This is a real skill. Was the previous original price just a joke for us? The first day using the original price API was a bit unfair.