The bigger the model, the stronger it is! A research team from New York University and Carnegie Mellon University recently published a cutting-edge paper titled "Requisitional Coding: Pushing the Limits of Model Compression with Self Generated Training Data". They proposed a novel model compression method called Recursive Coding. This research is not only an extremely efficient model compression technique, but may also be a new training paradigm that can make large models more powerful! It elegantly solves the generalization puzzle of large models from a new perspective of "information theory and compression", and provides solid theoretical support for the Scaling Law. 1、 The 'dead end' of existing model compression To understand the subtleties of this research, let's first take a look at why previous compression methods cannot explain the power of large models: Parameter based compression (such as 4-bit quantization, pruning): This type of method compresses the "body" (parameters) of the model. Its compression limit is highly physically bound to the number of model parameters. This means that even if a model with billions of parameters doesn't actually learn much useful knowledge, its parameters remain large after compression and cannot decouple the "parameter scale" from the "amount of information actually learned by the model". Pre sequential coding: This method does not compress parameters, but encodes the model by compressing the trajectory of the training data. But its fatal drawback is that it must compress the precise training dataset without loss. This results in a linear correlation between code length and dataset size, even if the model has learned the patterns, it still has to pay for the unpredictable random noise (high entropy parts) in the data. 2、 Innovative Requisitional Coding: Paying Only for 'Error Correction' In order to break through limitations, the author proposed Recursive Coding. Its core: instead of rigidly compressing real datasets, it is better to let the model "brainstorm" (autonomously generate) the data, and we only encode the areas where it "thinks wrong and needs to be corrected". How does it work? The student (target model) came up with a bunch of "multiple-choice questions and answers" (autonomously sampled candidate samples) based on their current cognition. Teachers (more powerful models) use Relative Entropy Coding (REC) to select the sample that best fits the true distribution from the samples supplemented by students' brains, and let students train and update it on it. Minimalist encoding: Since students and teachers share the same random number seed, the decoder (receiver) does not require large model parameters or redundant raw data, only needs to record which index the teacher accepted. It's like a smart student self-learning, the teacher doesn't need to teach word for word, just lightly tap (a very short error correction instruction) when the student deviates. The final compressed package consists only of these minimal correction instructions. 3、 Compression is intelligence: How can Requisitic training make large models stronger? Many people may ask, 'Is Requisitional Coding just a tool for optimizing the size of compressed packets after the fact?'? ” On the contrary. In the philosophy of Requisitic Coding, "better compression" and "stronger model" are actually two sides of the same coin. Through this "self generation teacher correction" loop training, the large model has achieved a transformative performance improvement in the following dimensions: 1. The ultimate "active denoising" allows the model to only learn useful patterns Real world data is filled with a lot of random noise, spelling errors, or meaningless trivial details. Traditional training forces models to fit all data without loss, resulting in the model wasting a lot of parameters to "memorize" these noises. In Requisitic training, the student model is only updated on key points that the teacher considers to have deviated from the essential laws. This is equivalent to installing a "denoising filter" on the model, forcing it to give up useless noise and only spend parameters on the cutting edge. 2. Stimulate "self generated" alignment, eliminate illusions and drift When deploying large models in practice, the biggest fear is the accumulation of illusions caused by "step-by-step mistakes" during autoregressive generation. Due to the fact that the core of Requisitic training is "student models generating data themselves", the model is already continuously responding to the data it generates and correcting itself during the training phase. This mechanism is similar to self alignment and can greatly enhance the robustness and anti-interference ability of the model in practical generation tasks. 3. The larger the model, the easier it is to compress Experiments have shown that, while maintaining the same prediction loss, the compressed code length of large and ensemble models is significantly smaller than that of small models. The large parameter space of a large model not only does not bring any burden, but also, due to its smooth local minima characteristic, makes it easier to grasp the global "focus" and essential laws. Because the big model can grasp the "big direction" on its own, the number of times teachers make corrections is significantly reduced. Large models use shorter underlying code in exchange for purer and stronger generalization ability. 4、 Scaling Law is theoretically 'indestructible' This paper incorporates the compression rate of Requential Coding into the famous PAC Bayes generalization boundary formula. The results are exciting: within the Compute optimal range, as the model scale expands, the calculated generalization guarantee boundary not only does not collapse, but also becomes increasingly tight! This mathematically refutes the argument that "big models are just rote memorization" and proves with data that the essence of Scaling Law is not that "models remember more scattered knowledge", but that "models have higher dimensional abstraction and simplification abilities, so they can map complex real worlds with shorter" underlying code ". 5、 Extended thinking: How will big models disrupt future scientific research? If 'the larger the model, the better and more concise compression methods can be found, and the stronger the ability to discover key points', then what does this mean for scientific research? The essence of scientific discovery is actually the process of humans' "lossy compression" of natural laws - extracting extremely short formulas such as $F=ma $or $E=mc ^ 2 $from countless complex physical phenomena. Traditional "small models" or human manual deduction often make overly simplified assumptions when facing high-dimensional complex systems such as biological genes and quantum chemistry due to the "bandwidth limitation" of the brain, which can easily be misled by experimental noise. The larger the scale of AI models, the stronger their ability to grasp the "essential focus" of the system. It can filter out massive amounts of impurity noise in high-dimensional chaos and be the first to point out a highly generalized correct physical path (such as AlphaFold's breakthrough in protein folding). The future paradigm of scientific research may evolve as follows: large models are responsible for "extreme compression" (discovering truth) in high-dimensional worlds, while human scientists are responsible for "decompression" (translating AI discovered dark knowledge into formulas that humans can understand). Conclusion Requisitic Coding shows us that size is not a burden, but rather the only path to minimalism and universality. The principle of Occam's Razor still shines in the modern era of deep learning: the simplest explanation is the best explanation. And the big model is the razor that can help us grind the fastest and most accurate in the vast cognitive understanding. ------------ Paper illustration Figure 4 (Loss vs. Cumulative Code Length) This figure compares the performance of Requisitic Coding with traditional compression methods such as PTQ quantization, Preliminary Coding, etc. on different datasets (OpenWebText, CIFAR-5M, FineWeb). It intuitively demonstrates that Requisitional Coding is perfectly located on the far left side of the Pareto front of "Loss vs. Compression Size". It even broke the limit of 4-bit quantization (PTQ) (the physical ceiling of FP32 parameter quantity), proving the dominant performance of the algorithm in model compression ratio.