Image classification: how neural networks label whole images
Image classification assigns one label to an entire image. Models learn from labeled examples using architectures such as convolutional networks or vision transformers. Evaluation relies on metrics like accuracy, recall, precision, and F1.
The task of image classification
Image classification assigns a single class to an entire image. Each image is expected to belong to exactly one category, and the model returns a prediction about that category, as described in the Hugging Face task page. This differs from object detection, which also provides location information.
The same page notes that stock photography uses classification to assign keywords to images, and image search services organize photo galleries by tags or categories. Image classification models are useful when the exact positions of objects are not required.
How a machine sees an image
To a computer, an image is a three-dimensional array of integers from 0 to 255, representing red, green, and blue values for each pixel. The Stanford CS231n notes state that a 248 by 400 pixel image contains 297,600 numbers. The task is to turn that array into a single label such as "cat".
Recognizing a category is hard for algorithms because of many variations. The same notes list viewpoint variation, scale variation, deformation, occlusion, illumination changes, background clutter, and intra-class variation as challenges that a model must handle.
Two approaches: rule-based and statistical
Rule-based classification uses human-defined rules and features. IBM's image classification article describes how a person might look for tires, doors, and mirrors to identify a car. This method relies on manual labeling and specialized knowledge.
Statistical classification instead trains models to learn patterns from data. IBM's article notes that convolutional neural networks use layers of increasing complexity to recognize parts of an image until the whole can be classified.
Datasets for learning and evaluation
Large datasets provide the labeled examples that statistical models need. ImageNet is an ongoing research effort that organizes images according to the WordNet hierarchy, aiming to provide about a thousand images for each concept. The ImageNet about page states that images are quality-controlled and human-annotated.
The same page notes that ImageNet does not own the images but compiles lists of web images; access is provided for non-commercial research and education. The scale and careful annotation make ImageNet a common benchmark for object recognition.
Deep architectures and modern practice
Deeper neural networks are harder to train. The arXiv paper on deep residual learning explains that reformulating layers as residual functions eases optimization and allows much deeper networks; a 152-layer residual network achieved 3.57 percent error on the ImageNet test set.
Vision transformers offer an alternative to convolutional networks. The arXiv paper on vision transformers shows that a pure transformer applied directly to image patches performs well on image classification after pre-training on large data.
In practice, one can fine-tune a pretrained vision transformer. The Hugging Face Transformers guide demonstrates loading a ViT checkpoint, preprocessing images into pixel values, and training on a food dataset. The same guide covers common evaluation metrics, including accuracy, recall, precision, and F1.
| Metric | Definition | Formula |
|---|---|---|
| Accuracy | Proportion of correct predictions among the total cases processed | Accuracy = (TP + TN) / (TP + TN + FP + FN) |
| Recall | Fraction of positive examples correctly labeled as positive | Recall = TP / (TP + FN) |
| Precision | Fraction of correctly labeled positive examples out of all examples labeled positive | Precision = TP / (TP + FP) |
| F1 | Harmonic mean of precision and recall | F1 = 2 * (precision * recall) / (precision + recall) |
Frequently asked questions
What is the difference between image classification and object detection?
Image classification assigns a single label to the whole image, while object detection locates and classifies multiple objects within an image, providing bounding boxes or other location information.
Why do images pose challenges for computer vision algorithms?
Images vary in viewpoint, scale, deformation, occlusion, illumination, background clutter, and the broad appearance of objects within a class, making recognition difficult for algorithms that rely on pixel values.
What is a convolutional neural network?
A convolutional neural network is a type of neural network used in statistical image classification that processes an image through layers of increasing complexity, from low-level edges to high-level parts, to determine the class.
What is ImageNet?
ImageNet is a large-scale image dataset organized according to the WordNet hierarchy, providing quality-controlled and human-annotated images for research in object recognition.
What is a vision transformer?
A vision transformer is a model that applies the transformer architecture directly to sequences of image patches. It can perform image classification well after pre-training on large datasets, without relying on convolutional layers.