Apple Reveals How It Trains Its Language Models
Imagine that Apple has just published a detailed technical report on its new foundation language models for Apple Intelligence. This document, titled “Apple Intelligence Foundation Language Models – Tech Report 2025,” describes how the company trained and optimized its models for both devices and servers. It covers architecture, data sources, pre-training, post-training, optimizations, and evaluation. And to spice things up, we will add four interesting points from an analysis by 9to5Mac that show how Apple has pushed the boundaries of efficiency and performance. Everything is based on actual data from the document.
Model Split into Two Blocks for Greater Efficiency
Apple’s on-device model, which has approximately 3 billion parameters, was cleverly split into two blocks. The first block contains 62.5% of all transformer layers, while the second block has the remaining 37.5%, but without key and value projections. This trick reduces memory usage for the KV cache (key-value cache) by 37.5% and also cuts the time to first token by approximately 37.5%. Apple claims that this preserves model quality, as confirmed by benchmarks such as MMLU (67.85 points), MMMLU (60.60 points), and MGSM (74.91 points). Comparisons with models such as Qwen-2.5-3B or Gemma-3-4B show that Apple’s model is competitive, although it lags behind larger ones such as Qwen-3-4B. This approach is reminiscent of an older Apple study from 2023, in which the company experimented with moving parts of the model between RAM and flash memory, but ultimately chose this more efficient structure.
Server Model Architecture for Private Cloud Compute
For its server model, Apple devised a new architecture called Parallel-Track Mixture-of-Experts (PT-MoE), tailored to the Private Cloud Compute platform. Instead of a single sequential stack of layers, it divides the model into multiple parallel tracks, with each track processing tokens independently and synchronizing only at block boundaries. Each track contains MoE layers (mixture of experts), where only the experts relevant to a given task are activated—for example, only those specializing in recipes for cooking-related tasks. This reduces synchronization overhead by 87.5% at a block depth of 4 and improves efficiency. The model also alternates between local and global attention layers, enabling it to process long sequences of up to 65,000 tokens without any loss of quality. In benchmarks, it scores 80.20 points in MMLU, 74.60 in MMMLU, and 87.09 in MGSM, which is comparable to LLaMA 4 Scout, though it lags behind larger models such as Qwen-3-235B or GPT-4o. This design makes the model faster and more efficient, making it ideal for cloud workloads.
275% Increase in Multilingual Support Thanks to Smarter Training
One of the biggest improvements is expanded language support. Apple increased the proportion of multilingual data used in training from 8% to 30%, an increase of 275%. This includes organic data, synthetic data, and machine-translated documents, with an emphasis on balance across languages. The tokenizer (a tool for breaking text into tokens) was expanded from 100,000 to 150,000 tokens, improving the representation of new languages. During post-training, including SFT (supervised fine-tuning) and RLHF (reinforcement learning from human feedback), the ratio of English to multilingual data was kept at 80:20. Evaluations showed significant improvements, for example in IFEval and AlpacaEval, where prompts written by native speakers were used to ensure natural language. As a result, features such as Writing Tools work more reliably in supported languages, including Portuguese, French, and Japanese, achieving a 16:9 win ratio over SFT versions in human evaluations.
From the Web to Synthetic Images, All with an Emphasis on Quality
Apple collected data responsibly, without using private user information. The largest portion comes from web crawling by Applebot, which respects robots.txt files to allow websites to opt out. The data includes hundreds of billions of pages, filtered for quality and stripped of profanity and harmful content. There is also licensed data from publishers, potentially including Condé Nast or Shutterstock, although details are not specified. Synthetic data plays a key role—Apple generated more than 5 billion image-caption pairs for visual understanding, including OCR (optical character recognition) for text-based images such as flyers or infographics. For images, more than 10 billion pairs with alt text were collected, along with 550 million images embedded in text. All of this helped the models handle tasks such as adding events to a calendar from a photo of a flyer. Overall, this leads to better performance in benchmarks such as MMLU, where the on-device model outperforms Qwen-2.5-3B.
This behind-the-scenes look shows how Apple is taking on the competition, even though it is still catching up. With an emphasis on privacy and efficiency through technologies such as Private Cloud Compute, this is a step forward for users who want smart features without compromise. If you are interested in the details, the technical report is an excellent read packed with charts and tables.
