… The company also noted speed improvements with the GB300 NVL72 v6.0 over v5.1, ranging from 1.21 times in the Llama 3.1 405B offline benchmark to 2.77 times for DeepSeek-R1 server test. “In just the past six months, we have been able to nearly triple our performance on DeepSeek-R1, which is a very … …
… If you need half as many GPUs, but they cost three or four times as much each, Nvidia gets to radically increase its revenues by selling at least twice as many devices, but your IT budget doesn’t go down and if your AI workloads are scaling – and they most certainly will be – then your IT budget in… …
… In the long run, the AI model builders will not rent huge amounts of capacity on the cloud. They will make the clouds sell them infrastructure on the cheap that they can amortize, thereby lowering the cost of tokens. …
… This may eventually be too costly for Meta Platforms to give it away as Google, OpenAI, and Anthropic most certainly do not. The Nemotron family has expanded significantly in the intervening years, with open AI models customized for particular industries that Huang pointed to during his keynote. …
… This is true given the current state of AI and the current cost of processing tokens. No one said it would remain true forever. …
… This competition is healthy, and will help drive the innovation that will keep driving the cost of AI processing – particularly AI inference processing – down and down and down until, at some point in the future, it will normalize at a price that doesn’t seem extreme compared to what it has cost in… …
… The amount of kilowatts you're consuming and tokens you're generating will become much more efficient.” The myth that AI data centers, because they're liquid cooled, are consuming water, is “not true, you can avoid it and still have that efficient data center.” Though local communities are likely t…
… In any event, in the table above, output speed is expressed in tokens per second, and the time to first token TTFT is expressed in seconds. The two smaller Llama 4 models beat OpenAI’s GPT 4o in terms of smarts by a little bit and cost 21X to 31X less per million tokens processed. …
… The second workload is called decode, and it means instead of chewing on tokens to either understand a query and its context or trying to find the statistical graph of linkages between tokens in a corpus of data, the system is spitting out tokens for the answers to a query in whatever modality the … …
… Which means the cost of inference has to scale down many orders of magnitude as the number of tokens processed as input and generated as output goes up by many orders of magnitude. If not, GenAI is dead in the water because even though prices have come down, they have not come down fast enough. …