Object detection: locating and labeling many objects in one image
Object detection is a computer vision task that finds and classifies objects in an image using bounding boxes. Models combine localization and classification, and are evaluated with metrics like average precision. This article covers the task, common architectures, and standard datasets.
What object detection is
Object detection is a computer vision task that identifies and locates objects in an image. According to IBM's overview of object detection, the task combines object localization, which draws bounding boxes around objects, and object classification, which assigns a semantic category. The output is a set of boxes with labels, each box marking one instance.
Object detection differs from image classification, which assigns a single label to the whole image, and from semantic segmentation, which marks each pixel with a category. IBM's page notes that detection is more precise than classification but less precise than segmentation, which works at the pixel level.
Object detection appears in many applications. Hugging Face's task description lists use cases such as autonomous driving, where vehicles detect pedestrians and road signs, sports tracking for monitoring players and balls, image search on smartphones, and counting objects in warehouses or crowds at events.
How object detection models are evaluated
Evaluation metrics measure how well a model locates and classifies objects. Hugging Face's object detection page describes Average Precision (AP) as the area under the precision-recall curve for a given class, and Mean Average Precision (mAP) as the average of AP values across all classes. These metrics are common in object detection benchmarks.
Another common metric is Intersection over Union (IoU), which compares the overlap between a predicted bounding box and the ground truth box. The Hugging Face page also mentions AP thresholds such as AP50 and AP75, where the threshold is the IoU value used to consider a detection correct.
Major model families
Early object detection systems reused classifiers, but the YOLO paper proposed a single neural network that predicts bounding boxes and class probabilities directly from a full image. The YOLO paper on arXiv states that this unified architecture runs at 45 frames per second for the base model, with a smaller version reaching 155 frames per second while achieving double the mAP of other real-time detectors.
Faster R-CNN introduced a Region Proposal Network that shares convolutional features with the detection network, making region proposals nearly cost-free. According to the Faster R-CNN paper, this two-stage approach achieved a frame rate of 5 frames per second on a GPU with the VGG-16 model while reaching high detection accuracy on PASCAL VOC and MS COCO datasets.
Mask R-CNN extends Faster R-CNN by adding a branch that predicts a segmentation mask for each detected object, in parallel with the bounding box branch. The Mask R-CNN paper on arXiv reports that it adds only a small overhead, running at 5 frames per second, and produces high-quality instance segmentation masks.
The COCO dataset and benchmarks
The Common Objects in Context (COCO) dataset is a standard benchmark for object detection and related tasks. The COCO dataset website lists challenges for detection, keypoints, panoptic segmentation, and DensePose across multiple years, providing a structured way to compare model performance.
COCO includes annotations for object instances, stuff categories, and keypoints, and its leaderboards allow researchers to evaluate detection accuracy. The dataset is widely used for training and testing modern object detection models, as seen in the YOLO and Faster R-CNN papers that report results on COCO.
| Model | Approach | Reported speed | Output |
|---|---|---|---|
| YOLO | Single-stage regression | 45 fps base, 155 fps Fast YOLO | Bounding boxes and class probabilities |
| Faster R-CNN | Two-stage with region proposal network | 5 fps on GPU with VGG-16 | Bounding boxes and class scores |
| Mask R-CNN | Extends Faster R-CNN with mask branch | 5 fps | Bounding boxes, class scores, and instance masks |
Frequently asked questions
What is Intersection over Union (IoU)?
IoU measures the overlap between a predicted bounding box and a ground truth box, calculated as the area of intersection divided by the area of union. It is used to determine whether a detection is correct.
What is Mean Average Precision (mAP)?
mAP is the average of Average Precision values across all object classes. AP is the area under the precision-recall curve for a class, and mAP summarizes detection accuracy across categories.
How does object detection differ from image segmentation?
Object detection outputs bounding boxes and labels for each object, while semantic segmentation assigns a category to every pixel. Instance segmentation, as in Mask R-CNN, combines object detection with per-instance masks.