Model serving: how trained models are deployed behind an API
Model serving places trained models behind APIs for inference. It involves creating endpoints, selecting deployment methods, optimizing performance, and scaling with distributed systems.
What is model serving
Model serving is the practice of making a trained machine learning model available to other software through an API. The model receives input data and returns predictions or other outputs, a process called inference.
This step moves a model from a development environment into production, where it can support applications, services, and automated decisions.
How serving systems work
An endpoint is a stable and durable URL that clients use to request a model's outputs. In Azure Machine Learning, an endpoint provides authentication, routing, and scaling, while a deployment holds the compute and code that actually runs the model. Microsoft's endpoint documentation
Serving software often organizes models in a repository. NVIDIA Triton Inference Server uses a model repository, per-model schedulers, and backends that handle different frameworks, allowing many models to run concurrently. NVIDIA's Triton user guide
Deployment methods
Models can be served in several modes. IBM describes real-time deployment for synchronous low-latency requests, batch deployment for offline processing of grouped data, streaming deployment for continuous calculations on data flows, and edge deployment for devices such as smartphones and wearables. IBM's model deployment overview
The choice of method depends on the latency and throughput requirements of the application. Real-time serving needs high-performance infrastructure, while batch processing can tolerate longer delays.
Optimizing inference
vLLM improves throughput by managing attention key and value memory with PagedAttention, and by continuously batching incoming requests and caching prefixes. vLLM's documentation
Hugging Face Text Generation Inference offers tensor parallelism for multi-GPU execution, token streaming with Server-Sent Events, and quantization for reduced memory use. Hugging Face's TGI documentation
NVIDIA TensorRT optimizes neural networks through quantization, layer and tensor fusion, and kernel tuning, then deploys them across data centers and edge devices. NVIDIA's TensorRT page
Serving at scale
For very large models, distributed inference spreads work across many nodes. Red Hat describes llm-d, a Kubernetes-native framework for distributed large language model inference, with a modular architecture that lets components work together or separately. Red Hat's llm-d overview
Such frameworks handle complex queries with large amounts of data by coordinating computation across clusters.
| Method | Description | Use case |
|---|---|---|
| Real-time | Synchronous low-latency requests for immediate predictions | Recommendation engines, chatbots |
| Batch | Offline processing of data grouped into batches | Document analysis, forecasting |
| Streaming | Continuous calculations on regular data streams | Fraud detection, IoT |
| Edge | Deployment on edge devices such as smartphones and wearables | Health monitoring, autonomous vehicles |
Frequently asked questions
What is the difference between online and batch endpoints?
Online endpoints handle real-time synchronous requests with low latency, while batch endpoints process large datasets asynchronously in jobs.
Why is continuous batching used in model serving?
Continuous batching groups incoming requests on the fly to increase throughput and reduce idle time, as implemented in vLLM.
How does quantization optimize inference?
Quantization compresses model weights to lower precision formats like INT8 or FP8, reducing latency and memory bandwidth while maintaining accuracy.
What is distributed inference?
Distributed inference spreads the computation of a large model across multiple nodes, coordinated by frameworks like llm-d, to handle complex queries faster.
What are the main model deployment methods?
The main methods are real-time, batch, streaming, and edge deployment, each suited to different latency and data processing needs.