Large language models have made quite a splash in recent years. They have dominated newspaper headlines, fascinated researchers, and powered chatbots and coding assistants around the world. But more and more companies are asking themselves a simple question: Do we really need that much power for every task? Increasingly, the answer is no. And this is where small language models (SLMs) come into play.
What are small language models?
SLMs are lightweight language models designed to operate efficiently even in resource-constrained environments. These include smartphones, embedded systems, and lower-performance computers.
While large models work with hundreds of billions or even trillions of parameters, SLMs typically range from one million to ten billion parameters. Despite this reduction, they retain core natural language capabilities: text generation, translation, document summarization, and question answering.
As Thomas Randall of the Info-Tech Research Group notes, it is not simply a matter of replacing large models with small ones. "A more accurate picture is one in which organizations are figuring out how to manage multiple models of different sizes across different deployment environments." The architecture works more like an intelligent division of labor: simple or precisely defined queries go to a small specialized model, while complex ones go to a large model.
How are models made smaller?
Several specific techniques are behind the compact size of SLMs, reducing model size without a significant loss of performance.
Knowledge distillation transfers capabilities from a larger “teacher” model to a smaller “student” model. The student learns not only to imitate outputs, but also the reasoning process itself. Pruning then removes unnecessary or redundant parameters from the neural network. And quantization converts values from high precision to lower precision, for example from 32-bit floating-point numbers to 8-bit integers. The result is less data, faster computation, and lower energy consumption.
In addition to these methods, there are approaches such as adapting a model to specific data (fine-tuning), using a knowledge base added at runtime (RAG), or the LoRA technique, which adds lightweight layers to the original model instead of retraining it.
Which models are classified as SLMs?
The market for small models is surprisingly diverse. The best-known include:
- Llama 3.2 from Meta, with 1-billion- and 3-billion-parameter variants optimized for edge devices
- Phi-3.5 Mini from Microsoft, with 3.8 billion parameters and a focus on reasoning and code generation
- Gemma from Google DeepMind, available in 2-billion-, 4-billion-, 7-billion-, and 9-billion-parameter versions, with multilingual and multimodal capabilities
- Qwen2.5 from Alibaba, with 1.5 billion parameters for multilingual use
- Granite from IBM, in 2-billion- and 8-billion-parameter variants, designed for enterprise environments, including cybersecurity and RAG tasks
- GPT-4o mini from OpenAI, a smaller, cost-effective variant of GPT-4o with multimodal capabilities
- DistilBERT, a lightweight version of Google's BERT that is 40% smaller and 60% faster while retaining 97% of the original model's capabilities
Mistral introduced the Ministral 3B and Ministral 8B models, with the eight-billion-parameter model outperforming its predecessor, Mistral 7B, in tests of knowledge, common sense, mathematics, and multilingual skills.
Why are companies embracing small models?
The main driver of interest is economic. Using a model with a trillion parameters for repetitive routine tasks, such as categorizing customer inquiries, generates unsustainable cloud costs. A specialized small model solves this problem.
An SLM excels when three conditions come together. The task is narrowly defined, repetitive and high-volume, and latency-sensitive. A small model does one thing well instead of doing everything adequately. And because it does not have to sift through the noise of the entire internet, it also reduces the risk of so-called hallucinations—that is, fabricated answers.
Privacy and security are also important. SLMs are small enough to run directly on devices or on a company's own infrastructure. This minimizes the risk of data leaks and provides better control over cyber threats. In sectors such as finance and healthcare, this is a crucial consideration.
Analyst firm Gartner predicts that by 2027, companies will use small, task-specific models three times more often than general-purpose large models. The diversity of tasks in business processes and the need for greater accuracy are driving the shift toward specialized models fine-tuned for specific functions or industry data.
Where do small models truly shine?
SLMs have a broader range of applications than it might seem. Chatbots and virtual assistants on mobile devices can respond in real time without an internet connection. Models such as Phi-3.5 Mini help developers write and debug code. Llama 3.2 can summarize a conversation on a smartphone and immediately create calendar entries.
In industry, SLMs are deployed directly on sensors and Internet of Things devices, where they analyze machine data and predict maintenance needs. Legal teams use them to identify contractual clauses, while finance departments use them to read transaction records and detect fraud. Nvidia researchers argue that SLMs are "powerful enough, inherently more suitable, and more economical for many calls in agentic systems, and are therefore the future of agentic artificial intelligence."
What can small models not handle?
Small models have their limits, and it would be naive to overlook them. The biggest trade-off lies in breadth of knowledge and reasoning ability. SLMs perform worse on tasks requiring contextual awareness or multi-step reasoning across unfamiliar domains. When a help desk receives a ticket that does not fit into any existing category, a small model may fail where a large one succeeds.
Microsoft itself acknowledges that its Phi-3 models "do not perform as well on factual knowledge benchmarks because the smaller model size results in a lower capacity for retaining facts."
Other weaknesses include bias: if the training data is not carefully prepared, a smaller dataset may amplify bias even further. SLMs are also more prone to errors in ambiguous situations or during complex, multi-stage social engineering attacks.
Small models can be fine-tuned for specific needs
One of the greatest strengths of SLMs is their adaptability. Exposing a model to industry-specific data can significantly improve its performance for a particular application. There are three main options. Full fine-tuning retrains all parameters on new data but requires greater computing power. LoRA (Low-Rank Adaptation) fine-tunes only a few layers, making it lighter and more efficient. Adapters and prompt tuning then add extra layers or optimize how the model receives instructions.
Companies should prioritize data preparation: collecting, managing, and organizing the data needed for fine-tuning. Enterprise data is becoming a key differentiator, and its quality, versioning, and structure determine how well a small model ultimately performs.
SLMs and large models as a team
Large language models will not disappear from the scene either. Both types will continue to grow within companies, with large models retaining an advantage in open-ended reasoning and breadth of knowledge.
A hybrid approach works like this: small models run directly at the customer's site or on the company's own infrastructure and, when a larger volume of data needs to be handled, turn to a large model in the cloud. Intelligent query routing determines which model receives which task. Simple requests go to a small model, while complex ones go to a large model.
Such systems allow companies to run SLMs on mobile devices using applications such as PocketPal or locally on a computer via the Ollama tool, which supports models such as Llama or Phi-3.5 Mini with minimal setup and no need for a cloud connection.
Sources: infoworld.com and ibm.com
