Recent reports suggest that Amazon is struggling to gain traction for its own artificial intelligence (AI) chips among startups. According to an internal Amazon document from July obtained by Business Insider, several startups consider the Trainium 1 and Trainium 2 chips less powerful than Nvidia's H100 graphics processing units (GPUs). The document, marked confidential, describes the specific experiences of companies such as Cohere and Stability AI, which tested these chips in practice.
AI startup Cohere reported that the Trainium 1 and Trainium 2 chips do not match the performance of Nvidia's H100 GPUs. Access to Trainium 2 was also extremely limited, and service outages occurred frequently. Amazon states in the document that it is still investigating these performance issues together with its Annapurna Labs division, but progress has been limited. An Amazon spokesperson later said that the situation involving Cohere was no longer current and that the company values customer feedback, which helps improve its chips.
Stability AI, known for generating images using AI, had similar experiences. According to the document, these startups found that Trainium 2 lags behind the H100 in latency, resulting in longer task processing times. This makes Amazon's chips less competitive in terms of both speed and cost. Another startup, Typhoon, found that Nvidia's older A100 GPUs are up to three times more cost-efficient than AWS Inferentia 2 for certain workloads. The AI Singapore research group compared G6 servers with Nvidia GPUs and concluded that they offer a better price-to-performance ratio than Inferentia 2 in several cases. Inferentia chips are used primarily to run AI models (inference), while Trainium focuses on training them.
The Importance of Custom Chips for AWS
Amazon is relying on its own Trainium chips to boost growth in AI cloud services. Amazon Web Services (AWS) succeeded in the past by designing its own data center chips instead of purchasing expensive components from Intel. In the current era of generative AI, Amazon is trying to avoid the high cost of Nvidia GPUs while still providing customers with powerful services. If customers insist on using Nvidia chips, it could affect AWS's profitability because the company will have to pay more for these components.
AWS claims that its chips offer a 30% to 40% better price-to-performance ratio than the current generation of GPUs. The company has a talented chip design team and is working on new versions, including Trainium 3, which is expected to be introduced later this year. An Amazon spokesperson said that Trainium 2 is fully allocated and is being used by major customers such as Anthropic. Other companies, including Ricoh, Datadog, and Metagenomi, have achieved good results with Trainium and Inferentia. Amazon plans to expand availability to more customers with the arrival of Trainium 3.
AWS CEO Matt Garman and Amazon CEO Andy Jassy recently discussed growth. During the quarterly earnings call, Jassy said that Trainium 2 is a multibillion-dollar business growing by 150% quarter over quarter. Nevertheless, technical limitations that prevent customers from switching to Amazon's chips remain.
Market Position and Partnerships
Nvidia dominates the AI chip market with a share of more than 78%, followed by Google and AMD, each with over 4%, according to research firm Omdia. AWS chips rank sixth with just 2%. This illustrates how difficult it is for Amazon to gain traction.
The new $38 billion agreement between AWS and OpenAI includes cloud servers equipped exclusively with Nvidia GPUs, with no mention of Trainium. Analysts at Mizuho consider this disappointing and note that it makes sense to start with Nvidia because of its superior performance and the CUDA platform, with which developers are very familiar. This is important for large AI projects where reliability plays a crucial role.
Trainium's most prominent customer is Anthropic, the creator of the Claude models. AWS launched Project Rainier, a massive data center with half a million Trainium chips for training Anthropic's next generation of AI models. By the end of the year, Anthropic is expected to deploy more than one million Trainium 2 chips. Anthropic recently expanded its collaboration with Google on TPU chips, causing Amazon's stock to fall, but the company says it will continue using Trainium.
Jassy emphasized during the earnings call that AWS offers multiple chip options to meet different customer needs, just as it does in other areas of cloud computing. After the results were released, showing AWS revenue rising 20% to $33 billion, Amazon's stock gained. This is the fastest growth since 2022, although it remains slower than that of competitors such as Microsoft or Google Cloud.
Sources: businessinsider.com and theinformation.com
