Neural Networks and Deep Learning: Mimicking the Brain

In the 2012 ImageNet contest, a network with 60 million adjustable numbers, trained for five to six days on two gaming graphics cards, missed the correct label among its top five guesses 15.3% of the time on a 1,000-category photo test. The next-best entry scored 26.2%. That gap shifted a long argument about whether machines should be programmed with rules or taught from examples. This article explains what a neural network computes, how it learns, why it failed for decades before 2012, and what the field still cannot do.

What a neural network computes, one neuron at a time

An artificial neuron is a tiny function. It takes several numbers as input, multiplies each by a weight (a number that says how much that input matters), adds the results, and produces an output based on whether the total is large enough. In 1943, Warren McCulloch and Walter Pitts proposed exactly this kind of unit as a simplified model of a nerve cell, in a paper titled "A Logical Calculus of Ideas Immanent in Nervous Activity" (Bulletin of Mathematical Biophysics, vol. 5, pp. 115 to 133). Their claim was that networks of such units can carry out logical operations.

You can check that with arithmetic. Let a neuron output 1 if the weighted sum of its inputs reaches a threshold, and 0 otherwise. Give it two inputs, each with weight 1, and a threshold of 2. It fires only when both inputs are 1, which is the logical AND. Lower the threshold to 1 and it fires when either input is 1, which is OR. Give a single input a weight of -1 and a threshold of 0 and it fires only when the input is off, which is NOT. AND, OR, and NOT are enough to build any logical circuit, which is why the 1943 result was taken seriously.

Biology gave the idea its vocabulary. A real neuron receives signals through dendrites, sums them in its cell body, and fires an electrical spike down its axon when the total passes a threshold, and the connections between neurons (synapses) strengthen or weaken with experience. But the resemblance is loose. Real neurons communicate in timed spikes, are wired in ways that change across a lifetime, and rely on a rich set of chemical signals. The artificial version keeps one idea from that picture, that knowledge lives in the strengths of connections, and discards almost everything else. "Mimicking the brain" is better read as inspiration than as a blueprint.

An analogy helps. Think of a neuron as a committee member who gets advice from several colleagues and trusts some more than others. The weights are the trust levels. The member announces a decision only if the weighted advice clears a bar. A network is a chain of committees, where the decisions of one layer become the advice for the next.

The perceptron, and the XOR problem that stalled the field

McCulloch and Pitts built their networks by hand. The missing piece was learning. In 1958, Frank Rosenblatt, then at the Cornell Aeronautical Laboratory, introduced the perceptron, a unit whose weights adjust themselves from examples. The learning rule is short. Show the perceptron an input. If its answer is wrong, nudge each weight in the direction that would have made the answer right, scaled by the input and by a small step size. Repeat across the training set. Rosenblatt proved that if a straight line (or flat plane, in more dimensions) can separate the two classes, this procedure eventually finds one.

The public reaction was loud. When the U.S. Office of Naval Research unveiled the perceptron in July 1958, running on a room-sized IBM 704 computer that learned to tell left-marked punch cards from right-marked ones, a New York Times headline announced a "NEW NAVY DEVICE LEARNS BY DOING," and The New Yorker called it "the first serious rival to the human brain ever devised," according to a Cornell retrospective.

The limit was built into the design. A single perceptron draws one straight line. Consider the XOR (exclusive OR) function, which outputs 1 when exactly one of its two inputs is 1:

Input A	Input B	XOR output
0	0	0
0	1	1
1	0	1
1	1	0

Plot the four input points on a square. The two points that should output 1 sit on one diagonal, and the two that should output 0 sit on the other. No single straight line separates them. In 1969, Marvin Minsky and Seymour Papert published Perceptrons, which laid out this and related limits for single-layer networks. The Cornell account says the book essentially sealed the perceptron's fate. Funding and attention moved toward symbolic AI, which encodes knowledge as explicit rules, and neural network research went quiet for years. For the wider timeline of boom and bust, see our history of artificial intelligence.

The critique was narrower than its reputation. It applied to single-layer networks. Adding a middle layer fixes XOR, and you can verify it by hand. Use two hidden neurons with the same inputs. The first computes OR (weights 1 and 1, threshold 1). The second computes AND (weights 1 and 1, threshold 2). An output neuron takes the first with weight +1 and the second with weight -1 and fires if its total reaches 1. For inputs (0,0) both hidden neurons are off, so the output is 0. For (0,1) and (1,0) only OR fires, so the output total is 1 and it fires. For (1,1) both hidden neurons fire, the total is 1 minus 1, which is 0, and the output stays off. That is XOR, solved with one hidden layer. The open question in 1969 was how to train such a network, not whether it could exist.

Backpropagation: how a network learns who to blame

The trouble with a multilayer network is the credit assignment problem. When the output is wrong, which of the thousands of weights, spread across several layers, caused the mistake? A perceptron can see its own error directly. A hidden neuron cannot.

The answer is backpropagation, popularized by David Rumelhart, Geoffrey Hinton, and Ronald Williams in a 1986 paper, "Learning representations by back-propagating errors," in Nature (vol. 323, pp. 533 to 536). It rests on the chain rule from calculus and runs in four steps:

1. Forward pass. Feed the input through the network to get a prediction.
2. Measure the loss. Compute a single number that says how wrong the prediction was.
3. Backward pass. Work from the output layer toward the input, using the chain rule to calculate how much the loss would change if each weight changed a little. That quantity is the weight's gradient.
4. Update. Move every weight a small step in the direction that reduces the loss.

Step 4 is gradient descent, and a hiker in fog is a fair picture of it. You cannot see the valley, but you can feel the slope under your boots. So you take a step downhill, feel again, and repeat. The loss is altitude, the weights are your coordinates on a map with millions of dimensions, and the gradient is the slope. (Our explainer on how gradient descent finds better answers walks through the mechanics.)

Why was backpropagation such a big deal? Because of cost. The brute-force way to learn a weight's gradient is to nudge it, run the network again, and see what happens, which needs one extra forward pass per weight. For a network with 60 million weights, that means 60 million passes for each update. Backpropagation reuses intermediate results so that one backward sweep, with a cost comparable to a forward pass, produces every gradient at once. Think of a restaurant kitchen where a bad review comes back and the owner traces blame step by step through plating, cooking, and prep, rather than interviewing every cook separately for every dish.

Why it took until 2012: algorithm, data, and hardware together

If backpropagation arrived in 1986, why did neural networks wait until 2012? Because the method needed more data and computing power than existed, and a few practical fixes for training deep stacks. Three ingredients converged.

Data. ImageNet is a dataset of over 15 million labeled, high-resolution images in roughly 22,000 categories, collected from the web and labeled by humans using Amazon Mechanical Turk. The annual competition built on it, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), used a subset of about 1.2 million training images, 50,000 validation images, and 150,000 test images across 1,000 categories (Krizhevsky, Sutskever, and Hinton, NeurIPS 2012).

Hardware. Training a large network is mostly multiplying big grids of numbers, which is exactly the work graphics processing units (GPUs) were built to do in parallel for video games. The AlexNet authors wrote their own optimized GPU code, and the network was split across two NVIDIA GTX 580 cards with 3 GB of memory each, taking five to six days to train. A single card could not hold the model. (For more on why this hardware suits the job, see how GPUs work and why AI needs them.)

Algorithmic tweaks. The AlexNet paper credits several. It used the ReLU (rectified linear unit), an activation that simply outputs max(0, x), instead of the smooth curves used before. On a four-layer test network, the authors found ReLU reached a given training error six times faster than the older tanh function. They also used dropout, which randomly switches off neurons during training so that the network cannot lean too heavily on any one of them, as a defense against overfitting, the habit of memorizing training examples instead of learning patterns that carry over to new ones. Our piece on overfitting and underfitting covers that trade-off.

Notice that none of the three was a new theory of learning. It was an old idea given enough data and horsepower.

ImageNet 2012: the result that changed the field

The ILSVRC 2012 task was to name the object in a photo. The score was top-5 error, the fraction of test images for which the correct label is not among a model's five most probable guesses. The official results list the top six entries:

Rank	Team	Top-5 error (ILSVRC 2012)
1	SuperVision (AlexNet)	15.3% (16.4% using only the supplied training data)
2	ISI	26.2%
3	OXFORD_VGG	27.0%
4	XRCE/INRIA	27.1%
5	University of Amsterdam	29.6%
6	LEAR-XRCE	34.5%

Source: ILSVRC 2012 official results. The winning run also used extra ImageNet data from the 2011 Fall release; the data-restricted run scored 16.4%.

The winner's margin over second place was 10.9 percentage points, or about 9.8 points even for the more restricted run. Contest scores usually moved by far smaller steps from year to year. The network behind it, later named AlexNet, had eight learned layers (five convolutional and three fully connected), roughly 60 million parameters, and 500,000 neurons. Its authors, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, were at the University of Toronto.

Newer networks pushed the numbers down quickly:

Year	Network	Top-5 error
2012	AlexNet	15.3%
2014	GoogLeNet (22 layers)	6.7%
2015	ResNet ensemble (up to 152 layers)	3.57%

Sources: ILSVRC 2012 results; GoogLeNet figure as reported in Karpathy's 2014 comparison; He et al., ResNet paper, 2015.

Did the machines beat people? Andrej Karpathy, then at Stanford, labeled 1,500 ILSVRC test images himself after training on 500 validation images, and measured his own top-5 error at 5.1%. GoogLeNet scored 6.7% on the full test set. The caveats matter, and Karpathy listed them: he was an unusually careful expert, less trained labelers reached roughly 12 to 15% error, and human accuracy "lives on a tradeoff curve" between effort and error rather than being one number. The fair summary is that by 2015 the best networks matched or edged out a dedicated human on this specific benchmark, not that they matched human vision in general.

How convolutional networks see: filters that slide

A photo is a grid of numbers. A 224 by 224 color image has 224 x 224 x 3 = 150,528 of them. If every pixel connected to every neuron in a first layer of 1,000 neurons, that single layer would need over 150 million weights, and it would treat a cat in the top-left corner as a different problem from a cat in the bottom-right.

Convolutional neural networks (CNNs) avoid this with a filter: a small grid of weights, say 3 by 3, that slides across the image and computes a weighted sum at every position. It works like a small magnifying glass dragged over a page, looking for one specific pattern such as a vertical edge. A 3 by 3 filter on a color image has only 3 x 3 x 3 = 27 weights, and the same 27 are reused at every position, which is called weight sharing. A first layer with 64 such filters has just 1,728 weights.

Stack the layers and the patterns compound. Early filters respond to edges and color blobs. Later layers combine those into textures and corners, then parts, then whole objects. Pooling layers shrink the grid between stages by keeping, for instance, the largest value in each 2 by 2 block, which makes the result less sensitive to small shifts in position. At the end, a few fully connected layers turn the final features into 1,000 class scores. The design was inspired by studies of the visual cortex, though the match is looser than the metaphor suggests.

Sequences, memory, and the vanishing gradient

Images have a fixed size. Language, speech, and sensor streams do not, and they depend on order. Recurrent neural networks (RNNs) handle this by passing a hidden state, a running summary, from one step to the next, using the same weights at every step.

Training them exposes a numerical trap. To learn from an error 100 steps back, the gradient is multiplied by roughly the same factor 100 times. If that factor is 0.9, the signal shrinks to about 0.9 raised to the 100th power, roughly 0.00003 of its original size. If it is 1.1, it grows to about 13,800 times. The first case is the vanishing gradient problem, in which early steps receive almost no learning signal. The second, an exploding gradient, makes training unstable. Picture a message passed down a line of 100 people, each of whom either whispers it a little quieter or shouts it a little louder.

Sepp Hochreiter and Jürgen Schmidhuber proposed a fix in 1997: the long short-term memory (LSTM) cell, which uses learned gates to control what to keep, what to write, and what to output, so that useful information can travel across many steps. LSTMs were widely used for speech and translation tasks until a different design took over.

Transformers: attention replaces recurrence

In June 2017, a team of eight researchers at Google posted "Attention Is All You Need." Their transformer dropped recurrence entirely. Instead of reading a sentence word by word, each word computes how much to "attend" to every other word, all at once, and those computations can run in parallel on GPUs. The authors reported 28.4 BLEU (a translation quality score) on English-to-German and 41.8 on English-to-French, with the French model trained in 3.5 days on eight GPUs, a small fraction of the cost of the best competing models (Vaswani et al., 2017).

The design spread fast. BERT (2018) read text in both directions at once and set new state-of-the-art scores on eleven language tasks (Devlin et al.). GPT-style models used the decoder half to generate text. In 2020, the Vision Transformer applied the same machinery to image patches and reported results comparable to the best convolutional networks while using less compute to train (Dosovitskiy et al.). For a deeper look, see our explainers on transformers and attention and how large language models work.

Networks that generate, and networks that play

Classification maps an input to a label. Generative models go the other way and produce new data. Two designs dominate.

A generative adversarial network (GAN), introduced by Ian Goodfellow and colleagues in 2014, trains two networks against each other in a "minimax two-player game." A generator produces samples, a discriminator estimates whether a sample came from real data or from the generator, and the generator is rewarded for fooling it (Goodfellow et al., 2014). Think of a forger and an art inspector improving each other.

Diffusion models train a network to remove noise from an image step by step, so that generation runs the process in reverse, starting from pure noise. The 2020 paper "Denoising Diffusion Probabilistic Models" by Jonathan Ho, Ajay Jain, and Pieter Abbeel reported a state-of-the-art FID score (an image quality metric where lower is better) of 3.17 on CIFAR-10 at the time (Ho et al.). Diffusion is the core of many current image tools; see our guide to how AI image generators work.

Neural networks also learn by acting. In deep reinforcement learning, a network picks actions and gets rewards. DeepMind's DQN agent, described in Nature in 2015, learned to play 49 Atari games from raw screen pixels and the score alone. It performed at a level comparable to a professional human games tester across the set, reaching more than 75% of the human score on more than half of the games (29 of 49), using the same network design and settings throughout (Mnih et al., 2015). In 2016, AlphaGo, which combined a policy network, a value network, and tree search, beat Lee Sedol 4 to 1 in Seoul (DeepMind). Our article on AlphaGo covers the match in detail. A separate method, Proximal Policy Optimization (PPO), published in 2017, offered a simpler way to train such agents and was evaluated on simulated robot locomotion and Atari (Schulman et al.).

Scaling laws: why bigger kept working

Around 2020, researchers noticed that language model performance improved smoothly and predictably with scale. Jared Kaplan and colleagues at OpenAI found that loss follows a power law in model size, dataset size, and training compute, over more than seven orders of magnitude, and that larger models are "significantly more sample-efficient" (Kaplan et al., 2020). A power law is a straight line on a log-log chart: each doubling of resources buys a steady fractional improvement, not a fixed amount.

In 2022, DeepMind's Jordan Hoffmann and colleagues revised the recipe. After training over 400 models from 70 million to over 16 billion parameters, they concluded that existing large models were "significantly undertrained" and that parameters and training tokens should scale together. Their 70-billion-parameter model, Chinchilla, used the same compute as the 280-billion-parameter Gopher but four times as much data, and outperformed Gopher, GPT-3 (175 billion parameters), and others on many tasks (Hoffmann et al., 2022). The lesson was that bigger is not the only lever. For a longer treatment, see scaling laws explained.

Scale has an energy price. Google researchers estimated that training GPT-3 used about 1,287 megawatt-hours of electricity and emitted about 552 metric tons of CO₂ equivalent, using US average grid emissions for the cloud region involved. For perspective, they list a passenger jet round trip between San Francisco and New York at about 180 metric tons of CO₂ equivalent, so one training run came to roughly three such flights. The same paper found that the choice of model, datacenter, and chip could reduce the footprint by 100 to 1,000 times, and that large, sparsely activated models can use less than a tenth of the energy of dense ones (Patterson et al., 2021). Those figures describe one 2020 model. Estimates for newer models vary and several developers have not published them.

What deep learning still cannot do well

It is hard to look inside. A trained network is millions or billions of numbers. You can see them, but they do not read like rules. Researchers use saliency maps (which input pixels mattered most), probing (training a small classifier on internal activations), and a newer approach, mechanistic interpretability, that tries to reverse-engineer the circuits inside a network. These tools give partial answers. In regulated areas such as lending and medicine, where a decision may have to be explained, that is a live problem.

It can be fooled in ways people are not. In 2013, Christian Szegedy and colleagues showed that small, nearly invisible perturbations, found by maximizing a network's prediction error, can make it misclassify an image, and that the same perturbation can often fool a different network trained on different data (Szegedy et al., 2013). A year later, Goodfellow's group argued the main cause is the models' largely linear behavior (Goodfellow, Shlens, and Szegedy). A photo of a panda with faint noise added can be labeled a gibbon with high confidence, the standard textbook example, while looking unchanged to a person.

It is often bigger than it needs to be. The lottery ticket hypothesis, from Jonathan Frankle and Michael Carbin and published at ICLR 2019, proposes that dense networks contain sparse subnetworks, "winning tickets," that can match the full network's accuracy when trained alone. In the tested cases they were under 10 to 20% of the original size (Frankle & Carbin). The practical implication is that efficiency research, such as pruning and distillation, may keep cutting the cost of running models.

Common misunderstandings

"Neural networks work like the brain." They borrow one idea from neuroscience, adjustable connection strengths, and differ in nearly everything else, including how they learn. Whether the brain does anything like backpropagation is still an open research question.

"Deep" means a lot of neurons. It means many layers. AlexNet had eight learned layers. The 2015 ResNet reached 152. Depth lets the network build features out of features, which is the point.

"The AI winters ended because of a theoretical breakthrough." The core learning algorithm was published in 1986. What changed by 2012 was data, GPUs, and a few training tricks such as ReLU and dropout. The lesson of the period runs against the usual story of a lone genius with a sudden insight.

"Scaling alone will keep working." Scaling laws describe trends over the ranges tested. They do not guarantee the same curve forever, and they say nothing about whether more of the same produces common sense, reliable reasoning, or understanding. That question is open, and researchers disagree about it.

Frequently Asked Questions

Q: What is the difference between a neural network and deep learning? A: A neural network is any system of connected units with adjustable weights. Deep learning refers to neural networks with many layers, trained end to end with backpropagation and gradient descent. AlexNet had eight learned layers and ResNet had up to 152, and depth lets later layers combine the features found by earlier ones.

Q: Why did neural networks fail in the 1960s and succeed in 2012? A: Single-layer perceptrons could not learn functions like XOR, and nobody had a practical way to train multilayer networks until backpropagation in 1986. Even then, data and computing power were too scarce. By 2012, ImageNet supplied 1.2 million labeled training images for the contest and GPUs made training feasible in days.

Q: What is backpropagation in simple terms? A: It is a way of working out how much each weight contributed to an error. The network makes a prediction, the error is measured at the output, and the chain rule carries that error backward through the layers, producing a gradient for every weight. Each weight is then nudged in the direction that lowers the error.

Q: Did AI beat humans at image recognition? A: On one benchmark, narrowly. In 2015 ResNet reached 3.57% top-5 error on ImageNet, while one expert who labeled 1,500 images scored 5.1%. The comparison has caveats: a single skilled person, a sample of the test set, and a specific task. It does not show that networks match human vision in general.

Q: Are neural networks just statistical pattern matching? A: In one sense yes: they fit functions to data. But the learned patterns can be layered and abstract, such as edges combining into parts and parts into objects. Whether that amounts to understanding is a philosophical dispute, and the evidence does not settle it. Failures like adversarial examples show the patterns can differ from human concepts.

Q: What comes after transformers? A: Nobody knows. Researchers are exploring alternatives and refinements, including sparse mixture-of-experts designs and other ways to reduce the cost of attention. Transformers have dominated since 2017 partly because of engineering maturity, and architectural shifts are hard to predict.

The takeaway

The history of neural networks is less a story of a single brilliant idea than of the same idea failing for decades and then working once the surrounding conditions caught up. The 1943 neuron, the 1958 perceptron, and the 1986 backpropagation algorithm were all in place long before 2012. What AlexNet showed was that, with 1.2 million labeled images, two GPUs, and a few practical fixes, a network could beat years of hand-built engineering by 10 percentage points.

That pattern continues. Transformers, diffusion models, and large language models mostly apply the same learning loop at greater scale and with better data and design. The same loop also carries the same limits: opaque internals, brittleness to adversarial input, and a large appetite for energy. Knowing which of these are properties of the method, and which are properties of today's scale, is the best guide to what the next ten years might and might not deliver.

Sources

• McCulloch, W. S. & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115 to 133.
• Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533 to 536.
• Hochreiter, S. & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735 to 1780.
• Krizhevsky, A., Sutskever, I. & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. NeurIPS.
• ImageNet, ILSVRC 2012 results and challenge description.
• Cornell Chronicle (2019), Professor's perceptron paved the way for AI, 60 years too soon.
• He, K. et al. (2015). Deep residual learning for image recognition. Szegedy, C. et al. (2014). Going deeper with convolutions.
• Karpathy, A. (2014). What I learned from competing against a ConvNet on ImageNet.
• Vaswani, A. et al. (2017). Attention is all you need. Devlin, J. et al. (2018). BERT. Dosovitskiy, A. et al. (2020). An image is worth 16x16 words.
• Goodfellow, I. et al. (2014). Generative adversarial networks. Ho, J. et al. (2020). Denoising diffusion probabilistic models.
• Mnih, V. et al. (2015). Human-level control through deep reinforcement learning. Nature, 518. DeepMind, AlphaGo. Schulman, J. et al. (2017). Proximal policy optimization.
• Kaplan, J. et al. (2020). Scaling laws for neural language models. Hoffmann, J. et al. (2022). Training compute-optimal large language models. Brown, T. et al. (2020). Language models are few-shot learners.
• Patterson, D. et al. (2021). Carbon emissions and large neural network training.
• Szegedy, C. et al. (2013). Intriguing properties of neural networks. Goodfellow, I., Shlens, J. & Szegedy, C. (2014). Explaining and harnessing adversarial examples.
• Frankle, J. & Carbin, M. (2019). The lottery ticket hypothesis. ICLR.

Read next:

• How Do Neural Networks Work? The Architecture Behind Modern AI
• AI Training vs. Inference: What Actually Happens
• How AI Models Are Trained: The Full Pipeline Explained