Face recognition technology: detection, embeddings and matching
Face recognition systems detect faces, convert them into numerical embeddings, and compare these vectors to identify or verify a person. Accuracy depends on image quality and algorithm design, and NIST runs standardized evaluations to measure performance. Confidence scores indicate similarity between embeddings.
Face detection
Face detection is the initial step in a face recognition system. The Azure Face service provides a Detect API that locates human faces in an image and returns rectangle coordinates and a unique identifier for each detected face, according to Microsoft's Face service documentation. The same API can optionally return attributes such as head pose, age, emotion, and glasses.
Detection performance depends on the image quality. For reliable detection, faces should be clearly visible and unobstructed. The Azure documentation notes that some attributes are general predictions, not actual classifications, and advises that applications may guide users to improve image quality, such as removing sunglasses.
Embedding extraction
Once a face is detected, the system converts it into a compact numerical representation called an embedding. The FaceNet paper presents a deep convolutional network that learns a direct mapping from face images to a Euclidean space where distances correspond to face similarity, and it reports accuracy of 99.63 percent on the Labeled Faces in the Wild dataset and 95.12 percent on YouTube Faces, using a 128-byte vector per face, as described in the arXiv FaceNet paper.
Embeddings serve as feature vectors for recognition tasks. The same paper states that once the embedding space is produced, face recognition, verification, and clustering can be implemented using standard techniques with these embeddings as feature vectors. This compact representation allows efficient storage and comparison.
Matching and confidence scores
Matching compares two face embeddings to determine if they belong to the same individual. AWS describes confidence scores, also called similarity scores, which indicate how similar two images are; a higher score suggests a greater likelihood that the images show the same person, as explained in AWS's facial recognition overview.
The matching process can be used for 1:1 verification, confirming a person against a trusted image, or 1:N identification, searching a large gallery. The choice of threshold for the confidence score affects the trade-off between false matches and missed matches, and each implementation sets this based on its needs.
Evaluating accuracy
NIST operates ongoing evaluations of face recognition algorithms. According to NIST's FRVT page, the Face Recognition Vendor Test has measured performance on very large sets of facial imagery and quantified demographic effects using nearly 200 algorithms from almost 100 developers, with more than 18 million images of over 8 million people.
The evaluation program was rebranded and split into two tracks: Face Recognition Technology Evaluation (FRTE) for identity verification and Face Analysis Technology Evaluation (FATE) for image analysis, as recorded on the NIST Face Technology Evaluations page. FRTE includes 1:1 verification, 1:N identification, and other tracks, while FATE includes morph detection, quality assessment, and age estimation.
For 1:1 verification, FRTE measures the false non-match rate (FNMR) at a specified false match rate (FMR). The definition is given on the FRTE 1:1 verification page: FNMR is the proportion of mated comparisons below a threshold set to achieve the given FMR, and FMR is the proportion of impostor comparisons at or above that threshold.
| Use case | Description |
|---|---|
| Fraud detection | Identify users during account creation and verify identity for suspicious activity. |
| Cybersecurity | Unlock devices or add a second authentication factor instead of passwords. |
| Airport and border control | Use e-passports with biometric data to let travelers pass through automated terminals. |
| Touchless access control | Provide entry to buildings, stadiums, or theme parks without physical cards. |
| Face redaction | Blur or hide faces in video to protect privacy. |
| Healthcare | Restrict access to patient records and streamline registration. |
Frequently asked questions
What is facial recognition?
Facial recognition is a way to identify or confirm a person's identity using an image of their face. It compares facial features against a database or reference image.
How does face detection work?
The Azure Face service's Detect API finds faces in an image and returns coordinates and a unique ID, optionally extracting attributes like age and emotion.
What is a confidence score?
A confidence score, or similarity score, indicates how similar two face images are. A higher score means a higher likelihood they are the same person.
What does NIST evaluate?
NIST's FRTE and FATE programs measure the accuracy of face recognition algorithms for verification and analysis, using metrics like false non-match rate.
What factors affect accuracy?
Image quality, lighting, pose, and aging can affect performance. Ideal conditions yield higher accuracy than real-world settings.