Chapter 1 · Lesson 1.4
Neural networks and deep learning
The neural network is the central model of modern ML.
From neuron to network
domino effect artificial neuron computes a weighted sum of its inputs, then applies an activation function. Learning consists in identifying the weights that weight the inputs. They are adjusted by the backpropagation algorithm (with gradient descent), which minimises the error between predictions and actual results.
An input layer feeds another layer, and so on. The Deep Learning uses networks with many multi-couches (sometimes hundreds): an input layer (raw data), hidden layers (increasingly abstract features), an output layer (result).
Deep learning works particularly well with large amounts of data and requires high computing power, often provided by GPUs or TPUs.
Going further: why a non-linear activation?
Without a non-linear activation function, stacking layers is pointless: two successive linear layers, W₂(W₁x), are equivalent to a single layer (W₂W₁)x. Non-linearity (ReLU: max(0, z) ; sigmoid: 1/(1+e⁻ᶻ) ; tanh) allows the network to represent curved boundaries such as the circle or the spiral of the chapter 1 demo.
Try it: backpropagation, step by step
Follow a complete learning cycle on a mini-network: the forward pass computes the error, the backward pass assigns each weight its share of responsibility, the update corrects the weights. All the calculations are displayed with their values.
Overview of architectures
| Architecture | Characteristics | Limits | Applications |
|---|---|---|---|
| Simple perceptron | 1 input layer → 1 output layer | Linear problems only | Historical entry point |
| MLP | Hidden layers + non-linear activations | Hard to interpret, overfitting | Classification, regression |
| CNN | Convolution + pooling | Lots of data and compute | Vision, images |
| RNN | Recurrent connections (memory) | Vanishing gradient | Time series, audio |
| LSTM / GRU | Gates (long-term memory) | Heavier | Translation, voice |
| Transformers | Based on attention, without recurrence | High memory cost | Chatbots (GPT), BERT |
| Autoencoders | Encoder → bottleneck → decoder | Do not generate realistic data (except VAEs) | Compression, anomalies |
| GAN | Generator versus discriminator | Unstable to train | Images, deepfakes |
Hands-on: four architectures
The perceptron and the MLP are handled in lesson 1-3 (“Hands-on: from neuron to network”), the Transformer in lesson 2-2. Here are the other four rows of the table, each in a collapsible block.
CNN: convolutional network
A convolutional network (Convolutional Neural Network) explicitly assumes that the inputs are images : it uses a small area of the previous layer (convolution filter) and thus encodes spatial properties in the architecture (e.g. LeNet-5).
Hands-on: CNN
A filter slides over the image and extracts a pattern from it (edges, diagonals); pooling reduces the size; a small classifier decides whether the resulting map looks like a T.
RNN and LSTM
domino effect RNN (recurrent network) processes a sequence element by element and feeds its state back in at the next step: this is a form of memory, useful for time series or audio. On long sequences, the gradient vanishes and the beginning is forgotten. The LSTM (and the GRU) adds portes that decide what to keep or forget, for longer memory, at the cost of a heavier model.
Hands-on: RNN and LSTM
Two ways of keeping the beginning of a sequence in memory.
Autoencoder
domino effect autoencodeur is made up of an encoder that compresses the input into a short representation (the “bottleneck”), then a decoder that tries to reconstruct it. Trained on normal data, it reconstructs poorly anything out of the ordinary: hence its uses in compression and anomaly detection.
Hands-on: autoencoder
Compress then reconstruct: the reconstruction error reveals what is out of the ordinary.
GAN
domino effect GAN (generative adversarial network) pits two networks against each other: a generator that makes fake data (images…) and a discriminateur that learns to tell real from fake. As they improve against each other, the generator produces very realistic content (hence deepfakes), but training is unstable.
Hands-on: GAN
A generator and a discriminator train against each other.
See the Graded exercise 1.1 et Graded exercise 1.2 to manipulate a neural network and a CNN in the browser. Course demos: gradient descent, backprop explainer, CNN demo.
A neural network learns its weights by backpropagation. Each architecture answers a type of data: CNNs for images, RNN/LSTM for sequences, autoencoders for compressing and spotting anomalies, GANs for generating, Transformers for language. More depth and data means more capacity, but also less interpretability.