Computer vision explained: how machines get meaning from images
Computer vision is a field of artificial intelligence that enables machines to process, analyze, and interpret visual inputs like images and videos. It typically involves data gathering, preprocessing, model selection, and model training, often using convolutional neural networks or vision transformers. Applications range from medical imaging to autonomous vehicles and security monitoring.
What is computer vision?
Computer vision is a subfield of artificial intelligence that gives machines the capability to handle and understand visual information from images and videos. It relies on machine learning to help computers extract useful meaning from visual data, as described in IBM's computer vision overview.
The field can be understood as the interaction of three broad processes: recognition, reconstruction, and reorganization. Recognition identifies actions, objects, people, places, and writing in digital images or videos. Reconstruction derives three-dimensional characteristics of those entities, while reorganization infers relationships between them.
Computer vision applications use AI and machine learning to process visual data for tasks such as object identification, optical character recognition, facial recognition, classification, recommendations, and monitoring, as noted by AWS's explanation of computer vision.
How computer vision works
Creating a computer vision system usually involves a sequence of steps: collecting data, cleaning it, choosing a model, and training that model. The first step is gathering a large dataset of visual information, often labeled by humans to indicate what is present in each image, according to AWS's description of the workflow.
Preprocessing involves cleaning and enhancing images to improve their quality, such as resizing or adjusting contrast. Data augmentation, which modifies existing images by flipping or rotating them, can expand the diversity of the dataset. Model selection involves choosing the best machine learning model for the task, with convolutional neural networks (CNNs) and vision transformers (ViTs) being common choices.
During model training, the algorithm performs feature extraction by analyzing labeled images to identify key attributes. Through repeated iterations, the model compares its predictions to correct labels and adjusts its internal parameters via backpropagation, as explained in AWS's computer vision guide.
Key models: CNNs and vision transformers
Convolutional neural networks (CNNs) are a class of artificial neural networks that use convolutional layers to filter inputs for useful information. The convolution operation combines input data with a convolution kernel to form a transformed feature map, and filters are adjusted based on learned parameters to extract the most useful information for a specific task, as described in NVIDIA's computer vision glossary.
CNNs can perform segmentation, classification, and detection. Segmentation classifies pixels to belong to a certain category, such as a car, road, or pedestrian. Classification determines what is in an image, and detection localizes where objects exist, often using rectangular bounding boxes.
Vision transformers (ViTs) divide an image into small patches and handle them as a sequence, much like words in a language model. They use a self-attention mechanism across these patches to build a representation of the whole image, and they often match or exceed the performance of CNNs on tasks like image classification, according to IBM's computer vision article.
Applications and use cases
Computer vision has numerous applications across industries. In medicine, it helps with rapid detection of tumors and hardening of the arteries, though it cannot by itself be trusted to provide diagnoses, as noted in NVIDIA's computer vision glossary.
In autonomous vehicles, computer vision can be used for early warning systems and can enhance or take over vehicle operations. In manufacturing, it supports quality control by automatically identifying product defects before products leave the factory, as described in AWS's computer vision use cases.
Security and safety applications include monitoring public spaces and industrial sites, detecting unauthorized individuals, and checking whether workers are wearing appropriate personal protective equipment. Computer vision also powers digital asset management, where it can group and identify images based on visible logos, faces, objects, and colors, and generate captions for images, as explained in Microsoft's Azure Vision overview.
Learning resources and standards
For those interested in studying computer vision, the Stanford CS231n course notes provide a comprehensive curriculum covering image classification, neural networks, convolutional neural networks, and more. The course includes assignments on topics like kNN, softmax, fully-connected neural networks, and convolutional nets.
The Hugging Face community computer vision course offers a structured path from fundamentals to advanced topics, including CNNs, vision transformers, multimodal models, and generative models. It includes theory, practical tutorials, and challenges, with prerequisites of Python programming and familiarity with transformers and neural networks.
When working with computer vision APIs, it is important to check the specific requirements and limitations of the service. For example, Microsoft's Azure Vision overview states that images must be in JPEG, PNG, GIF, or BMP format, less than 4 megabytes in file size, and greater than 50 x 50 pixels, with the Read API requiring dimensions between 50 x 50 and 10,000 x 10,000 pixels.
| Task | Description | Example use |
|---|---|---|
| Classification | Determines what is in an image | Identifying whether an image contains a cat or a dog |
| Detection | Localizes where objects exist in an image | Drawing bounding boxes around cars in a street scene |
| Segmentation | Classifies each pixel to a category | Separating road, cars, and pedestrians in a self-driving vehicle view |
| Optical character recognition (OCR) | Extracts text from images | Reading text from receipts or business cards |
| Facial recognition | Detects, recognizes, and analyzes human faces | Touchless access control or face blurring for privacy |
Frequently asked questions
What is the difference between computer vision and image processing?
Image processing focuses on transforming images, such as adjusting brightness or resizing, while computer vision uses AI and machine learning to derive meaningful information from visual data, such as identifying objects or extracting text.
What are the main types of models used in computer vision?
The most commonly used models are convolutional neural networks (CNNs) and vision transformers (ViTs). CNNs process images by applying convolutional filters that slide across the image, while ViTs process the entire image at once to capture global context.
What is data augmentation in computer vision?
Data augmentation involves modifying existing images, such as flipping or rotating them, to expand the diversity of the dataset. This helps make the model more adaptable and improves its ability to generalize to new images.
Can computer vision be used in medical diagnosis?
Yes, computer vision can help with rapid detection of tumors and other conditions in medical imaging. However, it cannot by itself be trusted to provide diagnoses and is used to support physicians by providing additional information.
What are the image requirements for Azure Vision?
Azure Vision can analyze images in JPEG, PNG, GIF, or BMP format, with a file size less than 4 megabytes and dimensions greater than 50 x 50 pixels. The Read API requires dimensions between 50 x 50 and 10,000 x 10,000 pixels.