Small language models: compact models for laptops, phones and edge
Small language models (SLMs) are neural networks with a few million to a few billion parameters, designed for resource-constrained devices. They are built through techniques such as knowledge distillation, pruning, quantization and low-rank factorization. SLMs provide offline inference, lower latency and better privacy on edge hardware.
What are small language models
Small language models are artificial intelligence models that process and generate natural language, similar to large language models but with a much smaller scale. According to IBM's explanation, SLM parameters range from a few million to a few billion, in contrast to LLMs with hundreds of billions or trillions of parameters. This makes SLMs suitable for resource-constrained environments such as mobile apps and edge devices, and for scenarios where inference must happen offline.
Like their larger counterparts, SLMs are based on the transformer architecture, which uses encoders, a self-attention mechanism, and decoders to process input sequences and produce output. The reduced parameter count lowers memory and computational requirements, enabling deployment where full-scale LLMs would be impractical.
How small models are created
Model compression reduces a larger model to a smaller one while preserving accuracy. Common methods include pruning, which removes redundant parameters, quantization, which converts weights to lower precision, and low-rank factorization, which decomposes weight matrices. These techniques are often combined with fine-tuning to recover any loss in performance.
Knowledge distillation is another approach, where a smaller student model is trained to mimic a larger teacher model. The DistilBERT project applied distillation during pre-training and retained 97% of the original BERT model's language understanding while being 40% smaller and 60% faster, as described in the DistilBERT paper.
Some SLMs are trained from scratch on carefully curated data. Phi-1, a 1.3 billion parameter code model, was trained on a selection of textbook-quality data and synthetic textbooks, achieving high accuracy on coding benchmarks, according to the Textbooks Are All You Need paper. TinyLlama, with 1.1 billion parameters, builds on the Llama 2 architecture and tokenizer and was pretrained on around a trillion tokens, as reported in its technical report.
The SmolLM family from Hugging Face offers models at 135 million, 360 million, and 1.7 billion parameters, trained on a corpus that includes synthetic textbooks and educational web samples, as described in the SmolLM blog post.
Deployment on edge hardware
Microsoft's Phi model family is designed for edge deployment, with the ability to run locally on devices or in the cloud. Phi models are described as providing accelerated response times for low-latency applications and can integrate with existing infrastructure for on-device inference, as stated on the Azure Phi product page.
Running models on edge devices requires careful inference optimization. The NVIDIA technical blog explains that the decode phase of autoregressive generation is memory-bound, and techniques such as quantization, paged attention, and in-flight batching improve GPU utilization and throughput.
For laptops, the ability to run an SLM locally depends on memory, storage, and processor capabilities. A laptop buying guide can help compare specifications and features relevant to local AI workloads.
Performance and trade-offs
Smaller models trade some accuracy for efficiency. Benchmarks show that well-designed SLMs can outperform other models of comparable size, as demonstrated by SmolLM and TinyLlama evaluations. However, for tasks requiring extensive world knowledge or complex reasoning, larger models may still be necessary.
The choice of a small model depends on the specific use case, the target hardware, and the acceptable level of accuracy. The table below summarizes a few notable SLMs and their characteristics.
| Model | Parameter size | Key approach |
|---|---|---|
| DistilBERT | 40% smaller than BERT | Knowledge distillation |
| phi-1 | 1.3 billion | Trained on textbook-quality data |
| TinyLlama | 1.1 billion | Built on Llama 2 architecture |
| SmolLM | 135M, 360M, 1.7B | Curated training corpus |
Frequently asked questions
What is a small language model?
A small language model is a neural network with a few million to a few billion parameters, designed to process natural language and run on resource-constrained devices.
How are small language models created?
They are created through compression of larger models via pruning, quantization, low-rank factorization, and knowledge distillation, or by training from scratch on carefully selected data.
Can small language models run offline?
Yes, SLMs can run locally on devices without an internet connection, enabling inference in edge environments and preserving privacy.
Do small language models sacrifice accuracy?
Some accuracy may be traded for efficiency, but well-designed SLMs can retain a high percentage of the larger model's capabilities, as seen with DistilBERT.