Can you afford a graphics card, but can't afford the internet bill? Former world model manager of xAI exposes monthly million dollar "internet bill" According to Beating monitoring, pre training video models not only consumes a huge amount of graphics card computing power, but network transmission and data storage expenses are often underestimated by the industry. Ethan He, the former world model leader of xAI, disclosed in the Latent Space podcast the actual infrastructure cost measurement: take the Internet video corpus with a scale of 1 billion as an example, the physical storage demand of the original video is about 5PB. If the variable auto encoder (VAE) is used to compress into continuous Latent features, the size of the finally managed feature data will rapidly expand to tens of PB. On mainstream cloud services such as AWS, the monthly bill for static storage of 5PB of data is about $100000. However, the larger capital expenditure comes from the network bandwidth (egress/ingress) cost incurred by distributed training nodes when repeatedly pulling and synchronizing data. In multiple pre training iterations, the network traffic cost of pulling massive video features in one go can exceed $230000, resulting in a monthly comprehensive storage and internet bill quickly exceeding several million dollars. The massive amount of video features also makes the training process extremely limited by the input/output (IO bound) of the data. In order to prevent idle computing power caused by waiting for data loading in the Wanka graphics card cluster, the R&D team must invest significant engineering effort in optimizing block caching and pipeline data allocation. This is also the underlying physical limitation that the training efficiency of video models is much lower than that of large pure text models with the same number of parameters. [Original link]