NVIDIA's H100 GPUs & The AI Frenzy; an overview of the current situation

We are all well aware of the NVIDIA and AI "goldmine" that has recently taken everyone by storm. In the middle of it all are Team Green's H100 AI GPUs, which are easily the most sought-after hardware for AI at the moment, with everyone scrambling to get their hands on one to power their AI needs.

The NVIDIA H100 GPU is the best chip for artificial intelligence at the moment, and everyone wants more of them

This article is not particularly newsworthy, but enlightens readers on the current state of the AI industry, and how companies are revolving around the H100 GPUs for their "future".

Before we get into the heart of the article, giving a summary becomes a necessity. So at the beginning of 2022, everything went well with the usual development. But with November's arrival, a revolutionary application called "ChatGPT" emerged, establishing the foundation of the AI hype. While we cannot categorize "ChatGPT" as the founder of the AI boom, we can certainly say that it acted as a catalyst. With that came competitors like Microsoft and Google, who were forced into an AI race to unleash generative AI applications.

You might say, where does NVIDIA come in here? The backbone of generative AI involves intense LLM (Large Language Model) training periods, and the NVIDIA AI GPUs come in clutch here. We won't go into technical specifications and tidbits of facts as that makes things boring and not fun to read. But if you're interested in getting to know the details, we've dropped a table below, highlighting every AI GPU release from NVIDIA, dating back to Tesla models.

NVIDIA HPC / AI GPUs

The question is still not answered here, why H100s? Well, we'll get there. NVIDIA's H100 is the company's highest offering, offering massive computing capabilities. One could argue that the unevenness of performance results in higher costs, but companies tend to order large volumes, and "performance per watt" is the priority here. Compared to A100, Hopper "H100" brings in 3.5 times more 16-bit inference and 2.3 times 16-bit training performance, making it the obvious choice.

So now we hope the superiority of the H100 GPU is clear here. Now, moving on to our next segment, why is there a shortage? The answer to this involves several aspects, the first being the huge volumes of H100s needed to train a single model. An astonishing fact is that OpenAI's GPT-4 AI model required around 10,000 to 25,000 A100 GPUs (at that time H100 was not released).

Modern AI startups such as Inflection AI and CoreWeave have raised huge sums for H100s, with a total accounting value in the billions of dollars. This shows that a single company requires huge volumes, even to train a basic to decent AI model, which has meant that the demand has been huge.

If you question NVIDIA's approach, one might say, "NVIDIA can increase production to cope with demand." Saying this is much easier than actually implementing it. Unlike gaming GPUs, NVIDIA AI GPUs require extensive processes, with most of the manufacturing assigned to Taiwanese semiconductor behemoth TSMC. TSMC is the exclusive supplier of NVIDIA's AI GPU, leading all stages from wafer procurement to advanced packaging.

H100 GPUs are based on TSMC's 4N process, a revamped version of the 5nm family. NVIDIA is the biggest customer for this process since Apple previously used it for its A15 bionic chipset, but the A16 Bionic has replaced it. Of all the relevant steps, the production of HBM memory is the most complicated as it involves sophisticated equipment currently used by a few manufacturers.

HBM suppliers include SK Hynix, Micron and Samsung, while TSMC has limited its suppliers and we are unaware of who they are. However, apart from HBM, TSMC also faces problems in maintaining CoWoS (Chip-on-Wafer-on-Substrate) capability, a 2.5D packaging process and a crucial stage in the development of H100s. TSMC cannot match the demand from NVIDIA, due to which order backlogs have reached new highs, and are being delayed until December.

So when people use the word GPU shortage, they are talking about a lack of, or a lag in, some component on the board, not the GPU itself. There is only limited worldwide production of these things… but we anticipate what people will want and what the world can build.

-Charlie Doyle, NVIDIA's DGX VP and GM (via Computerbase.de)

We have left out many details, but going into detail would detract from our primary goal, which is to detail the situation to an average user. Although we currently do not believe that the shortage can reduce and in turn is expected to increase. However, we could see a landscape shift here following AMD's decision to consolidate its position in the AI market.

DigiTimes reports that "TSMC appears to be particularly optimistic about demand for AMD's upcoming Instinct MI300 series, saying it will account for half of Nvidia's total production of CoWoS packaged chips" It could split the workload across companies. Still, judging by Team Green's greedy policies in the past, such a thing would require a serious offer from AMD.

In summary, NVIDIA's H100 GPUs are taking the AI hype to new heights, which is why this frenzy surrounds them. We aimed to end our talk by giving the readers a general idea of the whole scenario. Thanks to GPU Utilis for the idea behind this article; be sure to look at their report as well.