AI accelerators: ASICs, FPGAs and other chips for neural networks
AI accelerators are specialized chips that speed up neural network workloads. They include GPUs, FPGAs, ASICs, and neural processing units. This article covers their architecture, notable examples, and software abstraction.
What an AI accelerator is
An AI accelerator, sometimes called an AI chip or neural processing unit, is a specialized hardware component for speeding up the training and inference of neural networks. According to IBM's AI accelerator topic, these chips are used in smartphones, PCs, robotics, autonomous vehicles, and edge computing. Their specialized parallel-processing capabilities allow them to perform billions of calculations at once, reducing latency and improving efficiency compared with traditional CPUs.
The broad category of AI chips includes several hardware families. IBM's AI chip page lists graphic processing units (GPUs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs) as common examples. ASICs are highly customizable and can be built for a specific AI function, while GPUs, though originally designed for rendering, have parallel architectures that suit many AI workloads.
The Tensor Processing Unit as a design reference
A well-documented example of an ASIC for neural network inference is the Tensor Processing Unit (TPU). The arXiv paper on the TPU describes a custom chip with a 65,536 8-bit multiply-accumulate (MAC) matrix unit and a peak throughput of 92 tera operations per second. It also has 28 MiB of software-managed on-chip memory and a deterministic execution model that meets the response-time requirements of production neural network applications.
The TPU's design avoids features such as caches and out-of-order execution that are common in CPUs and GPUs. This contributes to its small size and low power. The paper reports that the TPU was on average 15 to 30 times faster than a contemporary server-class Intel CPU and an Nvidia K80 GPU, with energy efficiency (operations per watt) 30 to 80 times higher.
Cloud and data center accelerators
Cloud providers offer purpose-built inference chips. The AWS Inferentia page states that the first-generation Inferentia chip has four NeuronCores, and an Inf1 instance can contain up to 16 of these chips. The second-generation Inferentia2 supports data types including FP32, TF32, and configurable FP8, and offers up to 32 GB of HBM per chip.
For training and high-performance computing, AMD's Instinct product page describes GPUs built on the CDNA architecture with matrix and vector cores. These accelerators support a broad range of datatypes, from the efficient OCP MXFP 8/6/4 formats to FP64 for double-precision workloads, and are used for training, inference, and simulation in data centers.
Software abstraction with execution providers
To deploy models across different accelerators, frameworks rely on abstraction layers. The ONNX Runtime execution provider documentation explains that the Execution Provider (EP) framework uses a GetCapability() interface to allocate specific nodes or sub-graphs of a model to an EP library, which then executes on the target hardware. This allows a model to run on CPU, GPU, FPGA, or specialized NPUs.
ONNX Runtime supports many execution providers. The documentation lists CPU providers such as the default CPU, Intel oneDNN, and OpenVINO; GPU providers like CUDA and TensorRT; and edge and mobile providers such as Android Neural Networks API and Apple CoreML. The same ONNX API works across these providers, so developers can switch hardware with minimal code changes.
| Category | Provider examples |
|---|---|
| CPU | Default CPU, Intel oneDNN, Intel OpenVINO, XNNPACK |
| GPU | NVIDIA CUDA, NVIDIA TensorRT, AMD ROCm |
| IoT/Edge/Mobile | Android NN API, Apple CoreML, Qualcomm QNN |
| Other | Xilinx Vitis-AI, Huawei CANN, Azure, WebGPU |
Frequently asked questions
What is an AI accelerator?
An AI accelerator is a hardware chip designed to speed up AI workloads, such as neural networks and deep learning. It includes GPUs, FPGAs, ASICs, and NPUs.
How do AI accelerators achieve higher performance?
They use parallel processing to perform many calculations at once and may use low-precision arithmetic to reduce energy consumption.
What is a Tensor Processing Unit?
It is a custom ASIC designed by Google for neural network inference, with a matrix multiply unit and on-chip memory.
How does ONNX Runtime help with different accelerators?
It provides an Execution Provider framework that abstracts hardware, allowing the same model to run on various chips by selecting providers.