[Google Pixel Deploys Zero-Copy MTP, Gemini Nano Inference Speeds Up by Over 50% While Saving Memory] According to monitoring by Beating Insights, Google has deployed a Multi-Token Prediction (MTP) architecture in its Pixel 9 and Pixel 10 series devices, directly accelerating the built-in Gemini Nano v3 model. By attaching a lightweight Transformer prediction head to the frozen tail of the main model, the new architecture boosts on-device inference speed by over 50% while fully preserving the original safety alignment and output quality. Traditional speculative decoding requires running a separate draft model to predict candidate tokens. This not only consumes additional device memory but also limits prediction accuracy since the standalone model cannot access the internal hidden states of the main model. The new architecture embeds the MTP head at the frozen tail of the main model, successfully reusing the feature activations already computed by the main model, significantly improving the prediction accuracy of candidate tokens. To avoid the repeated memory overhead caused by draft computation during autoregressive generation, Google has designed a zero-copy mechanism. In traditional approaches, the draft model needs to maintain an independent key-value cache (KV cache) memory when generating candidate words, whereas the zero-copy mechanism allows the external prediction head to directly read the existing cache of the main model via cross-attention. This not only eliminates the startup latency of draft predictions but also saves approximately 130MB of device memory. In practical Pixel applications such as notification summarization and text proofreading, the MTP architecture enables the model to successfully predict nearly 2 additional tokens per inference on average, reducing the frequency of the main processor being frequently awakened for verification, thereby saving system power consumption. In highly structured text generation tasks like smart replies, token acceptance rates have increased by up to 55%. [Original Link]