OpenAI discovers new method to halve inference cost According to a source familiar with the discussions, there is a previously undisclosed news: Earlier this month, OpenAI engineers told some colleagues that they had found a solution to reduce the inference cost of the model by more than half, relying on several newly developed optimization technologies. After applying this new technology to scenarios where visitors without free/paid accounts use ChatGPT, engineers once reduced the required number of Nvidia graphics processing units (GPUs) to only a few hundred - a shockingly low number. At present, it is not clear to the outside world the specific technical means by which OpenAI achieves this significant improvement in computing efficiency. Common optimization solutions in the industry generally include: quantization compression, key value caching, batch processing of user queries instead of individual calculations, and redirecting some requests to lighter models or model shards with lower power consumption for response.