Aurorisk FR

Chapter 1 · Lesson 1.4

Neural networks and deep learning

The neural network is the central model of modern ML.

An architecture for each type of dataNeural networks: CNN; RNN, LSTM; Autoencoder; GANNeural networksCNNimagesRNN, LSTMsequencesAutoencodercompression, anomaliesGANgeneration
A network learns its weights by backpropagation.

From neuron to network

domino effect artificial neuron computes a weighted sum of its inputs, then applies an activation function. Learning consists in identifying the weights that weight the inputs. They are adjusted by the backpropagation algorithm (with gradient descent), which minimises the error between predictions and actual results.

An input layer feeds another layer, and so on. The Deep Learning uses networks with many multi-couches (sometimes hundreds): an input layer (raw data), hidden layers (increasingly abstract features), an output layer (result).

Large scale

Deep learning works particularly well with large amounts of data and requires high computing power, often provided by GPUs or TPUs.

Going further: why a non-linear activation?

Without a non-linear activation function, stacking layers is pointless: two successive linear layers, W₂(W₁x), are equivalent to a single layer (W₂W₁)x. Non-linearity (ReLU: max(0, z) ; sigmoid: 1/(1+e⁻ᶻ) ; tanh) allows the network to represent curved boundaries such as the circle or the spiral of the chapter 1 demo.

Try it: backpropagation, step by step

Follow a complete learning cycle on a mini-network: the forward pass computes the error, the backward pass assigns each weight its share of responsibility, the update corrects the weights. All the calculations are displayed with their values.

Overview of architectures

ArchitectureCharacteristicsLimitsApplications
Simple perceptron1 input layer → 1 output layerLinear problems onlyHistorical entry point
MLPHidden layers + non-linear activationsHard to interpret, overfittingClassification, regression
CNNConvolution + poolingLots of data and computeVision, images
RNNRecurrent connections (memory)Vanishing gradientTime series, audio
LSTM / GRUGates (long-term memory)HeavierTranslation, voice
TransformersBased on attention, without recurrenceHigh memory costChatbots (GPT), BERT
AutoencodersEncoder → bottleneck → decoderDo not generate realistic data (except VAEs)Compression, anomalies
GANGenerator versus discriminatorUnstable to trainImages, deepfakes

Hands-on: four architectures

The perceptron and the MLP are handled in lesson 1-3 (“Hands-on: from neuron to network”), the Transformer in lesson 2-2. Here are the other four rows of the table, each in a collapsible block.

CNN: convolutional network

A convolutional network (Convolutional Neural Network) explicitly assumes that the inputs are images : it uses a small area of the previous layer (convolution filter) and thus encodes spatial properties in the architecture (e.g. LeNet-5).

Hands-on: CNN

A filter slides over the image and extracts a pattern from it (edges, diagonals); pooling reduces the size; a small classifier decides whether the resulting map looks like a T.

RNN and LSTM

domino effect RNN (recurrent network) processes a sequence element by element and feeds its state back in at the next step: this is a form of memory, useful for time series or audio. On long sequences, the gradient vanishes and the beginning is forgotten. The LSTM (and the GRU) adds portes that decide what to keep or forget, for longer memory, at the cost of a heavier model.

Hands-on: RNN and LSTM

Two ways of keeping the beginning of a sequence in memory.

Autoencoder

domino effect autoencodeur is made up of an encoder that compresses the input into a short representation (the “bottleneck”), then a decoder that tries to reconstruct it. Trained on normal data, it reconstructs poorly anything out of the ordinary: hence its uses in compression and anomaly detection.

Hands-on: autoencoder

Compress then reconstruct: the reconstruction error reveals what is out of the ordinary.

GAN

domino effect GAN (generative adversarial network) pits two networks against each other: a generator that makes fake data (images…) and a discriminateur that learns to tell real from fake. As they improve against each other, the generator produces very realistic content (hence deepfakes), but training is unstable.

Hands-on: GAN

A generator and a discriminator train against each other.

Try it out

See the Graded exercise 1.1 et Graded exercise 1.2 to manipulate a neural network and a CNN in the browser. Course demos: gradient descent, backprop explainer, CNN demo.

Key takeaway

A neural network learns its weights by backpropagation. Each architecture answers a type of data: CNNs for images, RNN/LSTM for sequences, autoencoders for compressing and spotting anomalies, GANs for generating, Transformers for language. More depth and data means more capacity, but also less interpretability.