How Facial Recognition Works: From Pixels to Face Embeddings

In July 2018, the American Civil Liberties Union (ACLU) ran photos of all 535 members of Congress through Amazon's Rekognition software, comparing them against a database of 25,000 publicly available arrest photos. The system falsely matched 28 members of Congress to mugshots. Nearly 40 percent of the false matches were people of color, who make up about 20 percent of Congress (ACLU, July 26, 2018). Amazon replied that the ACLU had used an 80 percent confidence setting, which it called far too low for law enforcement.

Both sides were describing real features of the technology. Facial recognition is a statistical comparison with an adjustable dial, not a yes-or-no oracle. Here is how it works, where it excels, and why its mistakes are not spread evenly.

What are the steps from a photo to a match?

Most systems run the same four-stage pipeline. Each stage hands a cleaner, smaller piece of data to the next.

1. Detection. The software finds where faces are in an image. Early detectors scanned for simple light-and-dark patterns. The best-known early method, the Viola-Jones detector published in 2001, used a cascade of increasingly demanding tests that threw out non-face regions quickly, which let it run at 15 frames per second, real time for its day (Viola and Jones, CVPR 2001). Modern detectors use neural networks, which cope better with odd angles, shadows, and partial coverings.
2. Alignment. The face is rotated, scaled, and cropped to a standard layout, often by locating the eyes and using them as fixed reference points. This removes head tilt and distance as sources of variation, so the next stage sees a consistent input.
3. Feature extraction. A neural network converts the aligned face into a compact list of numbers called an embedding (or feature vector). This is the heart of the system.
4. Matching. The software measures the distance between two embeddings. If the distance falls below a chosen threshold, the system declares a match.

Think of the embedding as a street address for a face. The address does not describe the face the way a portrait would. It places the face at a point in a space with many dimensions, so that two photos of the same person land near each other and photos of different people land far apart. Matching is then a question of how far apart two addresses are.

What is the difference between one-to-one and one-to-many?

The same technology serves two very different jobs, and their error profiles differ.

One-to-one verification asks, "Is this person who they claim to be?" Unlocking a phone or passing an automated passport gate are examples. The system compares one live face with one stored template, so there is a single chance to be wrong.

One-to-many identification asks, "Who is this person?" It searches a whole gallery of stored faces. A police search of a mugshot database is one example. Every extra face in the gallery is another chance for a coincidental look-alike to score well, so error behavior depends heavily on the gallery size.

The U.S. National Institute of Standards and Technology (NIST) tests both modes separately for this reason, and its findings on demographic differences, discussed below, differ between the two.

How does deep learning build a face embedding?

Before deep learning, engineers tried to specify the useful measurements by hand: the distance between the pupils, the width of the nose, the angle of the jaw. A landmark 1991 approach, called eigenfaces, took a different route and used a statistical technique (principal component analysis) to find the main ways that a set of face images varies, then described each face as a weighted mix of those patterns (Turk and Pentland, Journal of Cognitive Neuroscience, 1991). These methods worked under controlled conditions and struggled with changes in lighting, pose, and expression.

Modern systems use convolutional neural networks (CNNs), layered programs that learn their own image features from data. The basic ideas are explained in our guide to how neural networks work. Early layers pick up edges and textures, middle layers respond to parts such as eye shapes and skin patches, and later layers combine those into a representation tuned to tell people apart. Nobody tells the network which facial traits to use. It discovers whatever numbers best separate identities in the training data, and the result does not correspond neatly to human concepts like "wide jaw."

What is triplet loss?

A well-known training method, used in Google's FaceNet system, is the triplet loss. The network is shown three faces at once: an anchor, another photo of the same person, and a photo of a different person. It is penalized if the same-person pair is not closer together than the different-person pair. The FaceNet paper trained on triplets of roughly aligned matching and non-matching face patches and mined for informative triplets during training, and it produced a 128-byte embedding per face (Schroff et al., FaceNet, 2015).

An analogy helps. Imagine arranging thousands of index cards on a huge table so that cards for the same person sit in a tight clump and clumps for different people sit far apart. You cannot say in advance what rule places each card, but you can nudge cards closer or farther every time you see a mistake. After millions of nudges, the table becomes a map where distance means likeness. That map is the embedding space. The same trick underlies how search engines find similar text and images, explained in embeddings and vector search, and large face galleries are stored and searched with the tools described in vector databases.

A later method, ArcFace, changed the training objective to push the embeddings of different identities apart by an angular margin on a hypersphere, and reported gains on ten benchmarks (Deng et al., CVPR 2019). The details differ, but the principle is the same: learn a space where identity equals closeness.

How accurate is facial recognition?

It depends on what is being measured, and benchmarks flatter the technology.

FaceNet reported 99.63 percent accuracy on the Labeled Faces in the Wild (LFW) benchmark, a set of celebrity photos, and 95.12 percent on the YouTube Faces database (Schroff et al.). A 2015 paper that was the first to pass human-level performance on LFW put the human figure at 97.53 percent, on the same benchmark (Lu and Tang, AAAI 2015). On that narrow task, then, machines beat people.

But LFW pairs are relatively easy: mostly frontal, well-lit photos of public figures. Real deployments involve security cameras, bad angles, motion blur, and galleries with millions of faces. A system that scores 99.6 percent on a benchmark can still produce many errors in a large search. The pitfalls of leaning on a single headline figure are a general problem with AI evaluation, covered in why AI benchmarks can be misleading and why 99 percent accuracy can be useless.

What are false matches and false non-matches?

Every matching system makes two kinds of mistakes, and the threshold trades one for the other.

Error	What happens	Typical consequence
False match (false positive)	Two different people are declared the same	A stranger unlocks a phone; an innocent person is flagged
False non-match (false negative)	Two photos of the same person are declared different	A legitimate user is locked out; a wanted person is missed

Raise the confidence threshold and false matches drop, but false non-matches rise. Lower it and the reverse happens. The right setting depends on the cost of each error. A phone should lean strict, because the cost of letting a stranger in is high and the cost of retrying is trivial. A system that only generates leads for a human to check can tolerate a looser setting, though as the Amazon example shows, the setting someone uses in practice may not be the one the vendor recommends. Amazon said that for law enforcement matches it recommends no less than 99 percent confidence (AWS, "Thoughts on Machine Learning Accuracy").

Why do big galleries produce more errors?

Here is an illustration with made-up round numbers, to show the arithmetic rather than any real product. Suppose a system wrongly matches two different people one time in 100,000 comparisons. For a one-to-one phone unlock, that is a very low risk. But if you search a gallery of one million faces for a single probe photo, you make a million comparisons, and you would expect around 10 false candidates to appear even if the true person is not in the gallery at all. The same error rate that sounds tiny in one setting becomes a steady supply of look-alikes in another. This is why the GAO, reviewing the FBI's face recognition program in 2016, flagged that the bureau had tested detection rates only for candidate lists of 50 and had not assessed how often errors occur (GAO-16-267).

Why do error rates differ across demographic groups?

The most thorough public study came from NIST in December 2019. It tested 189 algorithms from 99 developers, using 18.27 million images of 8.49 million people drawn from State Department, Department of Homeland Security, and FBI databases. Key findings (NIST, December 19, 2019):

• In one-to-one matching, false positive rates for Asian and African American faces were higher than for Caucasian faces, by factors ranging from 10 to 100 depending on the algorithm.
• Among U.S.-developed algorithms, one-to-one false positive rates were similarly elevated for Asian, African American, and Native groups, with the American Indian demographic showing the highest rates.
• Algorithms developed in Asia did not show the same gap between Asian and Caucasian faces in one-to-one false positives.
• In one-to-many searches against an FBI mugshot database of 1.6 million images, African American females showed higher false positive rates.

NIST computer scientist Patrick Grother said the study found demographic differentials in the majority of algorithms examined, and noted that the results were "an encouraging sign that more diverse training data may produce more equitable outcomes."

The word "factor" matters here. A tenfold difference in a very small number is still small in absolute terms, but it is the kind of gap that adds up in a large search or when a system is applied to millions of people. The important point is that the differences are not fixed laws of nature. They varied widely between algorithms, and the best performers showed much smaller gaps.

What causes the gaps?

NIST's remark points to the first suspect, training data. A network learns what it is shown. If a group is thinly represented, the embeddings for that group are less well separated, and look-alike errors become more likely. The finding that algorithms made in different regions behaved differently is consistent with this, though NIST's summary stops short of proving the cause.

A second factor is how systems are tested. A 2018 study by Joy Buolamwini and Timnit Gebru, "Gender Shades," built a new benchmark balanced by gender and skin type. They tested three commercial systems that classify a face as male or female. That is a different task from identifying a person, but it exposed the same pattern: error rates reached up to 34.7 percent for darker-skinned women, against a maximum of 0.8 percent for lighter-skinned men (Buolamwini and Gebru, PMLR 2018). A single headline accuracy hides this unless the test set is broken out by group.

A third set of factors includes image quality and capture conditions, such as lighting and camera settings, which can vary across settings. How strongly each factor contributes in a given system is an active research question, and nobody should claim there is a single cause.

How do 3D and infrared systems differ from photo matching?

A phone's face unlock works differently from a security camera. Apple's Face ID uses a TrueDepth camera that projects thousands of invisible dots onto the face to build a depth map and also captures an infrared image. The data is encrypted and handled by a dedicated Secure Enclave chip. Apple says a random stranger has a less than one-in-a-million chance of unlocking a device with one enrolled appearance, that the odds are higher for twins, siblings who look alike, and children under 13, and that five failed attempts force a passcode (Apple Support).

Depth and infrared data help because a flat photograph or a screen does not have the 3D shape and heat signature of a real face. This is the job of liveness detection, the set of checks that tries to confirm that a live person is present rather than a photo, a replayed video, or a mask. Passive methods analyze textures and reflections in the image, while active ones ask the user to blink or turn. Attackers keep probing both, so liveness detection is an arms race. The same sensing principle, measuring depth by projecting light, is explained in how LiDAR works.

Does a face mask or sunglasses defeat it?

Not reliably, but it hurts. In a July 2020 NIST study of 89 commercial algorithms, error rates when matching digitally masked faces to unmasked photos ranged from about 5 percent to 50 percent. The most accurate algorithms failed about 0.3 percent of the time on unmasked images and about 5 percent with masks, while many others failed 20 to 50 percent of the time (NIST, July 27, 2020). The study had caveats: it tested one-to-one matching, applied masks digitally rather than on real faces, and used algorithms developed before COVID-19. Vendors have since retrained systems on masked faces. For sunglasses and other partial coverings, the general principle is that less of the face means less information, so accuracy falls, but the amount depends on the system and the image.

Where is facial recognition used?

Phones and personal devices. This is the most common use, a one-to-one match against a template stored on the device.

Law enforcement. Police departments run probe photos from cameras and phones against mugshot or license databases. A 2016 GAO report said the FBI's Next Generation Identification Interstate Photo System let law enforcement search a database of over 30 million photos, and found that the FBI had not updated a required privacy assessment in a timely way (GAO-16-267).

Scraped databases. The company Clearview AI built a searchable database of faces collected from public websites. In September 2024 the Dutch data protection authority fined it €30.5 million for building what it called an illegal database of billions of facial images. Clearview's chief legal officer called the decision unenforceable, saying the company has no EU place of business or customers (SecurityWeek, September 3, 2024). Separately, Clearview settled an Illinois lawsuit in June 2024 in a deal that attorneys estimated could be worth more than $50 million.

Retail. The U.S. Federal Trade Commission announced on December 19, 2023 that Rite Aid would be banned for five years from using facial recognition for surveillance. The FTC said the chain used the technology in hundreds of stores from 2012 to 2020, matching shoppers against a database of "persons of interest" built from tens of thousands of often low-quality images, producing thousands of false matches, and that the system disproportionately affected people of color (FTC).

Border control and public spaces. Many countries use automated gates and some deploy live cameras that compare passersby with watchlists. This is where the legal fights are fiercest, as described below.

What happens when the match is wrong?

The Rite Aid case shows the retail version: employees followed, searched, and ordered consumers to leave, and called police, based on false alerts. The law enforcement version can mean an arrest. Robert Williams was wrongfully arrested in Detroit in 2020 after facial recognition matched him to a suspect in a store theft. The city later settled his lawsuit. Under the terms, police may not arrest someone based solely on facial recognition results, or on a photo lineup that directly follows such a search. Officers must be trained on the technology's risks, including higher misidentification rates for people of color, and must notify a prosecutor if they make an arrest without independent supporting evidence. Police must also audit cases since 2017 in which facial recognition was used to obtain an arrest warrant, and a court can enforce the settlement for four years (report on the settlement).

The lesson is about process as much as math. A facial recognition result is a lead that needs independent evidence. Problems arise when a lead is treated as proof.

What do the laws say?

Rules vary widely and are changing quickly.

European Union. The EU AI Act's Article 5 prohibits law enforcement use of real-time remote biometric identification in publicly accessible spaces, except where strictly necessary for three purposes: targeted searches for victims of abduction, trafficking, or sexual exploitation and for missing persons; preventing a specific, imminent threat to life or a terrorist attack; and locating suspects of certain serious offenses carrying at least a four-year sentence. Use generally requires prior authorization from a judicial or independent administrative authority, with a narrow urgency exception. The Article also bans AI systems that build facial recognition databases through untargeted scraping of images from the internet or CCTV footage. These core prohibitions apply from February 2, 2025 (AI Act, Article 5). Member States may adopt stricter rules.

United Kingdom. In Bridges v South Wales Police, the Court of Appeal ruled on August 11, 2020 that the police force's use of automated facial recognition was not "in accordance with the law" under Article 8 of the European Convention on Human Rights, because the policies left too much discretion over who could be put on a watchlist and where the system could be used. The court rejected a separate argument that the use was disproportionate (Court of Appeal judgment).

United States. There is no comprehensive federal law. Illinois's Biometric Information Privacy Act (BIPA) requires consent before collecting biometric identifiers and has produced large class actions. A judge approved a $650 million settlement with Facebook, now Meta, in a BIPA-style case in 2021, and Texas announced a $1.4 billion settlement with Meta on July 30, 2024 over allegations that it used biometric data without permission (ABC13 report). In the Texas case, Meta said it was pleased to resolve the matter.

For individuals, the practical side of biometric data is covered in how to protect your online privacy. Biometric unlocking also sits alongside other login methods, such as the ones described in how passkeys replace passwords. Unlike a password, a face cannot be changed after a leak, which is why many privacy laws treat biometric templates as sensitive data and why it matters how they are stored and encrypted, a subject covered in how encryption works.

How does facial recognition relate to deepfakes and AI ethics?

Two neighboring topics often get mixed in with facial recognition. Deepfakes are synthetic images and video of faces. They are a separate technology, but they interact with recognition in two ways. Recognition systems can be fooled by high-quality fakes if liveness checks are weak, and the same embedding techniques are used to compare a fake with a real person's face. We cover the generative side in what are deepfakes.

The broader debate about who builds these systems, who is accountable, and when they should not be used is part of the wider discussion in AI ethics: the key debates. Facial recognition is often the test case because its harms are concrete: a wrongful arrest, a person ordered out of a shop, or the chilling effect of being identifiable at a protest.

What do people get wrong about facial recognition?

"It stores a photo of my face." Not necessarily. The matching step works on embeddings, and some systems keep only those. A list of numbers can still identify you, so it is personal data, but it is not a viewable picture. Whether a particular system also keeps source photos varies, so the answer is in its privacy policy, not the algorithm.

"It is either accurate or it is not." Accuracy depends on the threshold, image quality, gallery size, and which people are being matched. One number cannot describe all of that.

"The bias is just a training data problem, so it is fixed." Training data is a major factor, and NIST's remark about diverse data is encouraging, but the 2019 test found large differences between algorithms, and performance in lab tests does not guarantee performance in a store or on a street.

"A match means the person did it." A match means two images were close in embedding space. It says nothing about guilt, and the Detroit settlement exists because a lead was treated as proof.

"It only matters for governments." Retailers, landlords, employers, and social platforms have all used biometric matching, and the Rite Aid and Meta cases involved private companies.

Frequently asked questions

Q: How does facial recognition identify a person? A: It detects the face, aligns it to a standard layout, passes it through a neural network that produces an embedding (a list of numbers), and measures the distance between that embedding and stored ones. If the distance is below a threshold, it declares a match.

Q: How accurate is facial recognition? A: On easy benchmarks it is very high: FaceNet reported 99.63 percent on LFW, above the 97.53 percent human figure cited in a 2015 paper. In real-world use accuracy depends on image quality, the threshold chosen, and the size of the gallery being searched, and it can be much lower.

Q: Is facial recognition biased? A: NIST's 2019 study of 189 algorithms found that in one-to-one matching, many algorithms had false positive rates 10 to 100 times higher for Asian and African American faces than for Caucasian faces, though the size of the gap varied widely between algorithms. Algorithms developed in Asia did not show the same gap between Asian and Caucasian faces.

Q: Can masks or sunglasses fool facial recognition? A: They reduce accuracy. A 2020 NIST study found that the best algorithms went from about 0.3 percent failures on unmasked faces to about 5 percent with digitally added masks, and many others failed 20 to 50 percent of the time. Vendors have since retrained systems, so those numbers are a snapshot.

Q: How does Face ID differ from other face recognition? A: Face ID projects thousands of invisible dots to make a depth map and also uses an infrared image, matching against a template on the device. Apple says a random person has less than a one-in-a-million chance of unlocking it, with higher odds for twins, close relatives, and children under 13.

Q: Is facial recognition legal? A: It depends on where and who. The EU AI Act prohibits most real-time biometric identification by police in public spaces and bans untargeted scraping of faces, from February 2, 2025. In the United States there is no comprehensive federal law, and state laws such as Illinois's BIPA and Texas's biometric statute have been enforced against companies.

Q: What should happen after a facial recognition match? A: Under the Detroit settlement, police may not arrest a person based only on facial recognition output or on photo lineups built from it, and must have independent evidence. The wider principle is that a match is a lead for investigators, not proof.

The takeaway

Facial recognition works by turning a face into a set of coordinates and checking how close two sets are. That design explains both its strengths and its failures. A well-trained network can separate millions of identities under good conditions, which is why it unlocks phones reliably. But the threshold, the gallery size, the image quality, and the mix of faces used in training all change the error rates, and NIST's 2019 study showed that those errors are not evenly spread across demographic groups. The most useful question to ask about any deployment is not "how accurate is it?" but "accurate for whom, at what setting, and what happens when it is wrong?"

Sources

• NIST, "NIST Study Evaluates Effects of Race, Age, Sex on Face Recognition Software", December 19, 2019; "NIST Launches Studies into Masks' Effect on Face Recognition Software", July 27, 2020.
• Schroff, Kalenichenko, and Philbin, "FaceNet", 2015; Buolamwini and Gebru, "Gender Shades", 2018.
• ACLU, "Amazon's Face Recognition Falsely Matched 28 Members of Congress"; AWS response.
• GAO-16-267; FTC on Rite Aid; EU AI Act, Article 5; Bridges v South Wales Police.

---

Read next:

• How Cloud Computing Works — The Infrastructure Behind Everything
• How Ransomware Works — The Cybercrime That Shuts Down Hospitals
• How Social Engineering Hacking Works — Why Humans Are the Weakest Link