Deep Learning Notes

Lecture 01

🧬 Neuron se Perceptron tak

Biology to Math — where does a brain cell become mathematics?

🧬 Biology ka Math — Neuron kya karta hai?

Sab kuch start hota hai ek sawaal se: kya ek single brain cell reasoning kar sakti hai?

The Biological Neuron — 3 Simple Jobs

Receive — dusre neurons se electrical signals lena (dendrites ke through)
Sum — sab signals ko add karna (cell body mein)
Fire — agar total ek threshold cross kare, signal aage bhej do (axon ke through)

Biological Neuron ka Mathematical Model: signals lao, multiply karo (weights), add karo (sum), fire karo agar enough!

🏛️ History — Santiago Ramón y Cajal (1888)

Spanish neurologist jisne prove kiya ki brain separate cells se bana hai jo signals ek direction mein pass karte hain. "Father of Modern Neuroscience."

McCulloch & Pitts (1943) — inhone neuron ko exact math mein convert kiya aur poochha: yeh kya compute kar sakta hai? Answer: koi bhi logic gate!

🔢 The Math — Weighted Sum + Threshold

McCulloch-Pitts Neuron (1943) Fire if: Σ w𝓎x𝓎 ≥ θ w𝓎 = weight (har input kitna important hai) x𝓎 = input (0 ya 1) θ (theta) = threshold (minimum sum to fire)

🎯 Analogy — Committee Vote

Neuron ek committee ki tarah hai. Har member (input) vote karta hai with different weight. Agar total votes threshold cross kare — fire ho jaata hai! Weights determine karte hain ki kiska vote zyada matter karta hai.

⚡ Logic Gates — AND, OR, NOT

Bohot exciting result tha: ek single neuron koi bhi logic gate ban sakta hai!

Gate	Weights	Threshold	Kab Fire karta hai
AND	w₁=1, w₂=1	≥ 2	Sirf jab DONO inputs = 1
OR	w₁=1, w₂=1	≥ 1	Jab koi bhi EK input = 1
NOT	w = -1	≥ 0	Jab input = 0 (negative weight!)

1943 ka Electrifying Result!

AND + OR + NOT se koi bhi logic circuit bana sakte hain. Matlab: neurons ka network = any computer!

But ek problem: weights kisi ne manually set kiye the. Network khud kuch nahi seekh raha tha!

📐 Perceptron — Pehli Learning Machine (Rosenblatt, 1958)

Perceptron = pehla model jisme weights automatically data se learn kiye ja sakte the!

Perceptron — 2 Steps Step 1 (Linear Combination): z = w₁x₁ + w₂x₂ + ... + b b = bias (threshold ko move karne ki trick) Step 2 (Activation/Decision): y = 1 if z ≥ 0 → Class 1 y = 0 if z < 0 → Class 0 Equivalently: y = step(w·x + b) Note: b = -θ, toh w·x + b ≥ 0 same as w·x ≥ θ

Perceptron ka Decision Boundary: ek straight line jo space ko 2 classes mein baant ti hai. Example: 2x₁+3x₂-12=0

⚠️ Perceptron ki ONLY Straight Lines Limitation!

x₁ aur x₂ sirf first power pe hain — koi x², koi x₁x₂ nahi. Isliye boundary hamesha ek straight line rahegi. Activation function boundary change nahi karta — sirf sign check karta hai.

Minsky & Papert (1969) ne prove kiya: single perceptron XOR solve NAHI kar sakta. Isse First AI Winter aaya!

🚨 XOR — The Problem that Broke AI

XOR = "exactly one switch ON." Jaise ghar ki staircase light jo do switches se control hoti hai — ek on to light on, dono on to light off!

x₁	x₂	XOR	Meaning
0	0	0	Dono off
0	1	1	Ek on
1	0	1	Ek on
1	1	0	Dono on?! off!

Key Insight: Purple circles (XOR=1) diagonal corners pe hain — (0,1) aur (1,0). Red crosses (XOR=0) dusre diagonal pe — (0,0) aur (1,1). Koi bhi ek straight line dono groups separate NAHI kar sakti. Yahi tha 1969 ka bombshell jo AI field ko sona dene par majboor kar diya!

Lecture 02

📘 XOR ka Fix — Hidden Layers & AI History

"What I cannot create, I do not understand." — Feynman

🕒 AI ka Itihas — 3 Golden Ages, 2 Dark Ages

1943

McCulloch & Pitts artificial neuron — mathematical brain cell model

1950–1956 ✨ First Golden Age Starts

Turing Test (1950) → Dartmouth Conference 1956 → "AI" naam rakha gaya

1957–1959 💡

Rosenblatt ka Perceptron (1957), Widrow-Hoff ka ADALINE (1959) — pehli learning machines!

1969 💥 First AI Winter

Minsky & Papert prove kiya: perceptron XOR solve nahi kar sakta. Funding band, research collapse!

1986 🚀 Second Golden Age

Rumelhart, Hinton & Williams ne Backpropagation popularise kiya — hidden layers train ho sakti hain! XOR solved!

1989 — UAT

Universal Approximation Theorem — ek hidden layer koi bhi function approximate kar sakti hai

1995 ❄️ Second AI Winter

SVMs neural nets se better the. Neural nets ka dusra winter!

1998 — LeNet

LeCun ka CNN — digit recognition ke liye (quietly alive!)

2006 🚀 Third Golden Age

Hinton ka RBM pretraining breakthrough — deep networks train ho sakti hain!

2012 🏆 AlexNet

ImageNet competition mein massive win — deep learning renaissance confirmed!

2017 ⚡ Transformer

"Attention is All You Need" — GPT, ChatGPT, GPT-4 sab isi se bane

2025 — DeepSeek-R1

Hum abhi yahan hain — aur yeh sirf continues kar raha hai!

Pattern Notice Karo: Har AI Winter kisi technical limitation se aya. Har Golden Age kisi breakthrough solution se start hua. Backpropagation ne XOR fix kiya, AlexNet ne scale fix kiya, Transformer ne sequence fix kiya. AI ek smooth climb nahi — spurts aur crashes!

💡 XOR ka Solution — Hidden Layer Magic!

Problem: ek neuron = ek straight line. Agar do neurons ko different kaam dein, phir combine karein?

🎯 Staircase Light Analogy

Ghar ki staircase light do switches se control hoti hai. Ek switch on = light on. Dono on = light off (wiring cancel). Yahi XOR hai!

Electrician ek switch se nahi, two-switch combination se solve karta hai. Neural network same — hidden layer!

The Trick: OR + NAND = XOR

Do hidden neurons ko alag kaam dein:

h₁ = x₁ OR x₂ (fires when at least one input is 1)
h₂ = x₁ NAND x₂ (fires unless both inputs are 1)
y = h₁ AND h₂ (fires when OR is on but NAND is also on)

x₁	x₂	h₁ (OR)	h₂ (NAND)	y = h₁ AND h₂	XOR?
0	0	0	1	0 AND 1 = 0	✅
0	1	1	1	1 AND 1 = 1	✅
1	0	1	1	1 AND 1 = 1	✅
1	1	1	0	1 AND 0 = 0	✅

Key Insight — Feature Space Transform!

Hidden layer ek naya representation space create karta hai. (x₁,x₂) space mein XOR non-linearly separable tha. But (h₁,h₂) space mein yeh linearly separable ho jaata hai!

Yahi hai hidden layers ki asli superpower: inputs ko aisa transform karo ki output layer ki job easy ho jaaye.

🏗️ Shallow Neural Network Architecture

Shallow Network: 2 inputs → 3 hidden neurons → 1 output. Har arrow = ek weight.

📥 Input Layer

Raw data. Koi computation nahi!

Neurons = features in data

Example: 10 features → 10 neurons

🔮 Hidden Layer(s)

Non-linear features seekhta hai.

a = φ(Wx + b)

Yahi XOR solve karta hai!

📤 Output Layer

Final prediction. Size = problem type.

Binary: 1 neuron (sigmoid)

k classes: k neurons (softmax)

Regression: 1 (linear)

🏷️ Weight Notation — Postal Address System

Think like a postal address: Country → City → Street → House. Weight notation bhi aisa hai!

Weight Notation: w[l]_ij w[l]_ij means: [l] = layer number (kaun si layer) i = destination neuron (is layer mein kahan) j = source neuron (pichli layer se kahan) Example: w[1]_21 Layer 1 weight, hidden neuron 2 mein, input neuron 1 se Matrix Form: z[l] = W[l] · a[l-1] + b[l] Then: a[l] = φ(z[l]) W[l] shape = (n_out × n_in) — destination rows, source columns

Symbol	Matlab kya hai	Example
x𝓎	i-th input feature	x₁ = first column value
W[l]	Weight matrix for layer l	W[1] = input→hidden weights
b[l]	Bias vector for layer l	b[1] = hidden biases
z[l]	Pre-activation (Wx+b, before φ)	z[1] = W[1]x + b[1]
a[l]	Post-activation (after φ)	a[1] = φ(z[1])
φ or σ	Activation function	sigmoid, ReLU, etc.

Lecture 03

📏 Linear Boundary, Forward Pass & Parameter Counting

Kyon boundary hamesha straight line hoti hai — prove karo!

📏 Straight Line — First Principles se Proof

1
Linear Combination: w₁x₁ + w₂x₂ + b — har input ko weight se multiply karo, sab add karo, bias add karo
2
Decision: Agar result ≥ 0 → class 1, nahi toh class 0
3
Boundary: Exactly wahan jahan w₁x₁ + w₂x₂ + b = 0 — yeh ek straight line hai!

Why Always Straight? School Algebra!

Boundary equation: w₁x₁ + w₂x₂ + b = 0

x₁ aur x₂ dono first power pe hain. x₁² nahi, x₁x₂ nahi, koi bhi curve nahi.

School algebra: ax + by + c = 0 hamesha ek straight line hoti hai. W linear combination → boundary straight.

Activation function (step ya sigmoid) sirf sign check karta hai — boundary nahi badalta!

Concrete Example: w₁=2, w₂=3, b=-12

Boundary: 2x₁ + 3x₂ - 12 = 0

At x₁=0: x₂ = 4 | At x₂=0: x₁ = 6

Above line (2x₁+3x₂-12 > 0): Class 1

Below line (2x₁+3x₂-12 < 0): Class 0

🔄 Forward Propagation — Prediction kaise aati hai

Forward propagation = input leke prediction nikalna. Data layer by layer aage badhta hai.

One Hidden Layer — Full Forward Pass Layer 1 (Hidden): z[1] = W[1] · x + b[1] ← linear step (matrix multiply) a[1] = φ(z[1]) ← non-linear activation Layer 2 (Output): z[2] = W[2] · a[1] + b[2] ← use previous layer's output ŷ = φ(z[2]) ← final prediction

Forward propagation: input → z[1] → a[1] → z[2] → ŷ. Har step: linear combination phir activation.

🔢 Parameter Counting

Network mein kitne learnable parameters hain? Weights + Biases = Total Parameters.

General Formula — One Layer Parameters = (n_in × n_out) + n_out weights biases Note: Input layer has NO bias — it just holds values!

Example: 2-3-1 Network

Layer	Weights	Biases	Total
Input→Hidden (2→3)	2×3 = 6	3	9
Hidden→Output (3→1)	3×1 = 3	1	4
TOTAL	9	4	13

Practice: 3-3-1 Network

Hidden: 3×3 = 9 weights + 3 biases = 12

Output: 3×1 = 3 weights + 1 bias = 4

Total = 16 parameters

Lecture 04

⚡ Activation Functions — ReLU, Leaky ReLU, ELU & Softmax

Sigmoid fail kyun karti hai — aur kya fix karta hai?

📉 Vanishing Gradient — Sigmoid ka Asli Problem

Sigmoid aur Tanh — dono S-shaped curves hain. Jab z bahut bada ho jaata hai, curve flat ho jaati hai. Flat curve = near-zero derivative = almost no learning!

⚠️ Sigmoid Saturates at z=6 — Numerical Proof

Given: x=1, w=6, b=0, y=0, α=1

z = 6·1 + 0 = 6
σ(6) = 0.9975 (prediction: almost 1)
Error = 0.9975 - 0 = 0.9975 (maximum possible ≈ 1)
σ'(6) = σ(6)(1-σ(6)) = 0.00247 (TINY!)
Gradient = error × σ' × x = 0.00246 (step taken)
Almost 100% error but only 0.0025 correction!

This is Vanishing Gradient: |z| bada hone pe sigmoid flat → derivative ≈ 0 → no learning!

Tanh bhi fix nahi karta!

Tanh ka max derivative 1.0 hai (sigmoid ka 0.25). Near z=0 tanh better hai. But z=1.663 pe dono cross karte hain, aur uske baad tanh aur tezi se flat ho jaati hai!

S-shape hi problem hai — koi bhi rescaling fix nahi karega.

⚡ ReLU — Rectified Linear Unit

S-curve band karo. Ek function use karo jo positive side pe kabhi flat nahi hoti!

ReLU Definition f(z) = max(0, z) = z if z > 0 (positive: identity, passes through) = 0 if z ≤ 0 (negative: zero, blocks signal) f'(z) = 1 if z > 0 ← ALWAYS 1, never shrinks! f'(z) = 0 if z < 0 ← zero (dead neuron problem) At z=0: corner exists, libraries take f'(0)=0

ReLU vs Sigmoid — Same Problem, Massive Difference!

Activation	f'(z=6)	Step Taken	Ratio
Sigmoid	0.00247	0.00246	1×
ReLU	1	6	≈2439× bigger!

ReLU same problem 1 step mein zero error karta hai jahan sigmoid thousands of steps leta!

⚠️ Dying ReLU Problem

Agar z negative ho gaya (e.g., w=-3, x=1 → z=-3):

f(-3) = 0, f'(-3) = 0 → gradient = 0 → weight update = 0 → neuron PERMANENTLY DEAD!

No learning rate can rescue a zero gradient. Ek baar dead = hamesha dead.

🔧 Leaky ReLU — Dead Neurons ka Fix

ReLU negative side mein f'=0. Fix: negative side ko tiny slope do — 0 nahi, 0.01!

Leaky ReLU f(z) = z if z > 0 ← same as ReLU f(z) = 0.01z if z ≤ 0 ← tiny slope, never zero! f'(z) = 1 if z > 0 f'(z) = 0.01 if z < 0 ← 0.01 ≠ 0 → never dies!

z=-3 pe comparison

Activation	ŷ	f'(-3)	Gradient	Dead?
ReLU	0	0	0	☠️ YES
Leaky ReLU	-0.03	0.01	-0.0103	✅ No, but crawling

Leaky ReLU ki Weaknesses

Negative output bahut chota (-0.03 vs -0.95 ELU) — barely meaningful signal
z=0 pe derivative jump: 0.01 → 1 (discontinuous!)
0.01 ek handpicked arbitrary constant — no theoretical basis

🌊 ELU — Exponential Linear Unit

Leaky ReLU: negative side straight line (slope 0.01). ELU: negative side ko smooth exponential curve se replace karo!

ELU Definition f(z) = z if z > 0 ← same as ReLU/Leaky f(z) = e𝑧 - 1 if z ≤ 0 ← exponential curve, floor at -1 f'(z) = 1 if z > 0 f'(z) = e𝑧 if z < 0 ← derivative of (e𝑧-1) Key insight: at z=0, f'(0-)=e⁰=1=f'(0+) → SMOOTH! No jump! As z→-∞: f(z)→-1 (bounded below, never linear)

Full Comparison at z=-3 (e⁻³=0.0498)

Activation	ŷ	f'(-3)	Gradient	Verdict
ReLU	0	0	0	☠️ Dead
Leaky ReLU	-0.03	0.01	-0.0103	🐌 Crawling
ELU	-0.950	0.0498	-0.097	✅ Moving properly!

ELU step ≈9.4× Leaky ReLU se bada! More meaningful signal on negative side.

🎲 Softmax — Multi-Class Classification ke liye

Sigmoid = ek class (yes/no). Agar 3 classes: cat, dog, bird? Teen separate sigmoids ka sum 1 nahi hoga! Softmax = ek function jo sabko simultaneously handle kare.

Softmax Formula softmax(z𝓪) = e𝑧𝓪 / Σⱼ e𝑧ⱼ Step 1: Har score ko exponentiate karo (positive banao) Step 2: Sab exponentials ko add karo Step 3: Har exponential ko total se divide karo (sum to 1!)

Worked: z₁=2.0 (cat), z₂=1.0 (dog), z₃=0.1 (bird)

1
Exponentiate: e²⋅⁰ = 7.389, e¹⋅⁰ = 2.718, e⁰⋅¹ = 1.105
2
Sum = 7.389 + 2.718 + 1.105 = 11.212
3
P(cat) = 7.389/11.212 = 65.9% | P(dog) = 24.2% | P(bird) = 9.9%
4
Check: 0.659 + 0.242 + 0.099 = 1.000 ✅
5
Prediction: CAT (65.9% confidence)

Bonus: Softmax with 2 classes = Sigmoid!

n=2 leke karo: e𝑧/(e𝑧+e⁰) = e𝑧/(e𝑧+1) = 1/(1+e⁻𝑧) = sigmoid!

Binary classification ek special case hai — separate idea nahi!

📊 Complete Comparison

Function	Formula	Range	Max f'	Use Case	Problem
Sigmoid	1/(1+e⁻𝑧)	(0,1)	0.25	Output binary, LSTM gates	Vanishing gradient
Tanh	(e𝑧-e⁻𝑧)/(e𝑧+e⁻𝑧)	(-1,1)	1	RNNs, old shallow nets	Saturates faster
ReLU	max(0,z)	[0,∞)	1	Default hidden (CNNs)	Dying neurons (z<0)
Leaky ReLU	z or 0.01z	(-∞,∞)	1	GANs (dying destabilizes)	Tiny gradient, hand-tuned
ELU	z or e𝑧-1	(-1,∞)	1	When ReLU units dying	Slower (exponential)
Softmax	e𝑧𝓪/Σe𝑧ⱼ	(0,1), sum=1	-	Output multi-class	Layer-wise only

Quick Decision Rule:
Hidden layers → ReLU (default), Leaky ReLU / ELU (if neurons dying)
Output (binary) → Sigmoid (1 neuron)
Output (k classes) → Softmax (k neurons)
Output (regression) → Linear / nothing

Lecture 05

🧠 MLP & Backpropagation

XOR solved by hand, then gradients flow backward — this is how learning happens!

🧩 MLP se XOR — Actually Build it!

Step activation use karo: output 1 if z≥0, else 0.

XOR MLP — Specific Weights OR Neuron (h₁): a₁[1] = step(x₁ + x₂ - 0.5) Fires when at least one input is 1 (sum ≥ 0.5) AND Neuron (h₂): a₂[1] = step(x₁ + x₂ - 1.5) Fires only when both inputs are 1 (sum ≥ 1.5) XOR Output: ŷ = step(a₁[1] - a₂[1] - 0.5) Fires when OR is on but AND is off

x₁	x₂	h₁ (OR)	h₂ (AND)	h₁-h₂-0.5	ŷ	XOR?
0	0	0	0	-0.5	0	✅
0	1	1	0	0.5	1	✅
1	0	1	0	0.5	1	✅
1	1	1	1	-0.5	0	✅

Geometrically Why it Works

Har hidden neuron ek line draw karta hai. OR line aur AND line parallel hain. XOR = strip between them.

Ek line kabhi XOR separate nahi kar sakti. Do lines mili ke — kar sakti hain!

↩️ Backpropagation — Network Kaise Seekhta Hai

Forward pass = prediction nikalna (left se right). Backward pass = mistakes se seekhna (right se left, chain rule use karke).

1-1-1 Network: x → h → ŷ, Sigmoid + Squared Error Loss Loss: L = ½(ŷ - y)² Forward Pass: z[1] = w₁·x + b₁, a[1] = sigmoid(z[1]) z[2] = w₂·a[1] + b₂, ŷ = sigmoid(z[2]) Backward Pass (Chain Rule): ∂L/∂ŷ = (ŷ - y) ← loss w.r.t. prediction ∂ŷ/∂z[2] = ŷ(1 - ŷ) ← sigmoid derivative ∂z[2]/∂w₂ = a[1] ← how z[2] changes with w₂ ∂L/∂w₂ = (ŷ-y) · ŷ(1-ŷ) · a[1] ← chain rule! Then similarly propagate to w₁ (one more chain step back)

Chain Rule — Yahi Hai Backprop ka Core

Composite functions: L depends on ŷ, ŷ depends on z[2], z[2] depends on w₂.

∂L/∂w₂ = (∂L/∂ŷ) × (∂ŷ/∂z[2]) × (∂z[2]/∂w₂)

Har weight ke liye chain continue hoti hai jab tak us weight tak nahi pahuncho. Deep networks mein chain bahut lambi hoti hai!

Backprop: Forward pass mein prediction, backward pass mein chain rule se har weight ka gradient.

Deep Network mein Vanishing Gradient aur Bura Kyun?

Sigmoid ka max derivative 0.25. 10 layers mein: 0.25¹⁰ ≈ 0.000001! First layers ko practically zero gradient milta hai — they learn nothing.

Isliye deep networks mein ReLU zaroori hai — f'=1 hamesha (positive side pe), multiplication se shrink nahi hota!

Lecture 06

⛰️ Gradient Descent — Model ka Seekhna

Foggy pahari mein neeche jaana — blindfolded!

⛰️ Core Idea

🎯 Foggy Hill Analogy

Pahari pe blindfold lagake khadhe ho. Minimum nahi dekh sakte. But local slope feel kar sakte ho.

Strategy: slope ke opposite direction mein ek chota step lo. Baar baar karo — eventually bottom!

Yahi hai gradient descent. Loss function = pahari. Parameters = tumhari position.

The Update Rule θₜ₊₁ = θₜ - α · ∇L(θₜ) θ (theta) = parameters (weights, biases) ∇L(θₜ) = gradient (uphill direction) α (alpha) = learning rate (step size) Negative sign: gradient OPPOSITE direction mein jaao (downhill!)

Worked Example: L(w) = (w-3)², w₀=0, α=0.25

1
g₀ = 2(0-3) = -6, w₁ = 0 - 0.25(-6) = 1.5, L(1.5) = 2.25
2
g₁ = 2(1.5-3) = -3, w₂ = 1.5 - 0.25(-3) = 2.25, L(2.25) = 0.5625
3
g₂ = 2(2.25-3) = -1.5, w₃ = 2.25 - 0.25(-1.5) = 2.625
4
Har step: loss aadha ho jaata hai! Eventually w=3 (minimum) pe pahunchenge.

Gradient negative hai (w < 3), toh subtraction w ko badhata hai → minimum ki taraf!

🎯 Teen Flavors — Kitna Data Ek Baar Mein?

Vanilla GD (Full-Batch)

Sab N examples ek saath use karo

✅ Accurate gradient

✅ Smooth convergence

❌ N=1M → huge memory, slow

1 update per epoch

True SGD (1 Example)

Ek random example, turant update

✅ Super fast

✅ Noise helps escape local minima

❌ Very noisy zigzag path

N updates per epoch

Mini-batch GD ⭐

Small group: 32, 64, 128 examples

✅ GPU efficiently use karta

✅ Balanced noise vs speed

✅ Modern DL ka default!

N/B updates per epoch

Real World Mein: "SGD" bolne se usually mini-batch SGD matlab hota hai. True SGD (1 example) almost kabhi use nahi hota. Mini-batch (32-256) = modern deep learning ka standard!

🎚️ Learning Rate — Step Size ka Magic

Observation	Diagnosis	α kaisa hai?
Loss bahut slowly gir rahi hai	Stagnation	Too LOW ⬆️
Loss gir rahi but bounce karti hai	Oscillation	Slightly HIGH
Loss tezi se badh rahi (NaN!)	Divergence	Much too HIGH 💥

📅 Learning Rate Scheduling

Fixed α = ek hi speed everywhere — highway aur parking dono pe! Better: schedule karo.

Step Decay

Milestones pe suddenly reduce

Graph: staircase

Simple

Abrupt drops

Exponential Decay

Smoothly reduce — same % every step

αₜ = α₀ × e⁻𝛋ₜ

Smooth

Cosine Decay ⭐

Cosine curve follow karo

Gentle "soft landing"

Modern default!

Warm-up

Pehle chota α, phir badhao!

Early instability prevent

Transformers mein essential

Reduce on Plateau

Val loss improve nahi → reduce

Reactive strategy

No total epoch needed

Lecture 07

🧠 Optimisers with Memory

EWMA, Momentum, NAG, AdaGrad, RMSProp, Adam, AdamW — the full journey!

🗺️ The Optimiser Journey

SGD
Reacts

→

EWMA
Remembers

→

Momentum
Direction

→

NAG
Look-ahead

→

AdaGrad
Scale

→

RMSProp
Fading

→

Adam
Both

→

AdamW
+Decay

🎯 Core Analogy — SGD vs Smart Optimisers

SGD = boat steering by only the newest ripple. One bad ripple → wrong direction!

Memory optimisers = boat keeping a compass heading (direction history) and a sea chart (scale history). One noisy ripple doesn't derail them.

📡 Part 1: EWMA — Fading Memory

EWMA Rule sₜ = β · sₜ₋₁ + (1-β) · xₜ β = decay factor (0 ≤ β < 1) sₜ₋₁ = previous state (memory) xₜ = new observation Large β → longer memory, slower reaction Small β → shorter memory, faster reaction Unrolled: sₜ = (1-β)xₜ + β(1-β)xₜ₋₁ + β²(1-β)xₜ₋₂ + ... Older observations fade geometrically! Coefficient of xₜ₋₂ = β²(1-β)

🎯 Coach Analogy

Cricket coach player ki form judge karta hai. Kal ki performance sabse zyada important, pichle week ki thodi kam, pichle mahine ki bahut kam. Coach ek "running impression" rakhta hai — ek hi number (sₜ)! EWMA exactly yahi hai.

Worked: s₀=0, β=0.8, x₁=5, x₂=1

1
s₁ = 0.8(0) + 0.2(5) = 1.0
2
s₂ = 0.8(1.0) + 0.2(1) = 1.0
3
Newer x₂ ka coefficient: 0.2 | Older x₁ ka coefficient: 0.16. Newer gets more weight ✓

EWMA sirf ek state sₜ store karta hai — puri history nahi!

🚀 Part 2: Momentum — Direction Yaad Rakho

EWMA ko gradients pe apply karo → velocity banao jo direction yaad rakhti hai.

SGD with Momentum vₜ = β · vₜ₋₁ + (1-β) · gₜ ← velocity update (EWMA of gradients) θₜ₊₁ = θₜ - α · vₜ ← parameter update (use velocity, not raw g!) State kept: vₜ (velocity = direction history)

🎯 Rolling Bicycle Analogy

Bicycle ek nayi direction mein ek chhoti wobble se immediately face nahi karta. Usme inertia hai! Momentum gradient zigzag ko smooth karta hai — overall downhill direction yaad rehti hai.

Worked: v₀=0, θ₀=2, β=0.9, α=0.1, g₁=4, g₂=-2

1
v₁ = 0.9(0) + 0.1(4) = 0.4
2
θ₁ = 2.0 - 0.1(0.4) = 1.96
3
v₂ = 0.9(0.4) + 0.1(-2) = 0.36 - 0.2 = 0.16 (still positive!)
4
θ₂ = 1.96 - 0.1(0.16) = 1.944 (θ still decreasing!)
5
g₂ negative tha lekin velocity positive rahi! Ek opposing gradient direction reverse nahi karta.

🔭 Part 3: NAG — Look Before You Leap

Momentum: gradient at current point measure karo. NAG: pehle predict karo ki momentum tum kahan le jaayega, WAHAN gradient measure karo!

Nesterov Accelerated Gradient (NAG) Step 1: θ_look = θₜ - α·β·vₜ ← look-ahead point Step 2: Measure gradient at θ_look (not θₜ!) Step 3: vₜ₊₁ = β·vₜ + (1-β)·∇L(θ_look) ← corrected velocity Step 4: θₜ₊₁ = θₜ - α·vₜ₊₁ ← update real parameter

🎯 Cyclist at Sharp Bend

Cyclist sharp bend approach karte waqt sirf current speed nahi dekhta. Bend par look karta hai — agar road upar hai, pehle se brake lagao! NAG: anticipate karo gradient kahan hoga, wahan se correct karo.

Worked: θₜ=2.0, vₜ=0.5, α=0.1, β=0.9, ∇L(θ_look)=3.0

1
θ_look = 2.0 - 0.1(0.9)(0.5) = 2.0 - 0.045 = 1.955
2
βvₜ = 0.45, (1-β)∇L = 0.30
3
vₜ₊₁ = 0.45 + 0.30 = 0.75
4
θₜ₊₁ = 2.0 - 0.1(0.75) = 1.925

Momentum vs NAG

Momentum: "Yahan slope kya hai?" → gradient at current θₜ

NAG: "Momentum mujhe kahan le jaayega? Wahan slope kya hai?" → gradient at θ_look

⚖️ Part 4: AdaGrad — Coordinate-wise Scale

Problem: different features ke gradients bahut different sizes ke ho sakte hain. Ek hi α sabke liye? Inefficient!

AdaGrad rₜ = rₜ₋₁ + gₜ² ← accumulate squared gradients θₜ₊₁ = θₜ - α · gₜ / (√rₜ + ε) ← normalize by scale Large past gradients → large rₜ → smaller effective step (cautious) Small past gradients → small rₜ → larger effective step (bolder)

Worked: rₜ₋₁=(0,0), gₜ=(4,1)

1
gₜ² = (16,1)
2
rₜ = (0,0) + (16,1) = (16,1)
3
√rₜ = (4,1)
4
gₜ/√rₜ = (4/4, 1/1) = (1,1) — both equalized!

⚠️ AdaGrad ka Fatal Flaw

rₜ = sum of ALL past g² — never resets! Deep training mein rₜ → ∞ → step → 0. Learning stops!

Old gradients never forget — 1000 steps purana gradient aaj bhi utna hi influential hai.

AdaGrad ka Fix: RMSProp

Replace cumulative sum with EWMA — taaki puraane gradients fade ho jaayein!

RMSProp sₜ = ρ·sₜ₋₁ + (1-ρ)·gₜ² ← EWMA of squared gradients θₜ₊₁ = θₜ - α · gₜ / (√sₜ + ε) ← normalize by fading scale ρ = decay (usually 0.9). Old scale fades → learning can recover!

Property	AdaGrad	RMSProp
State	Cumulative Σg²	Fading EWMA of g²
Old scale	Stays forever	Repeatedly discounted
Late training	Increasingly timid	Can recover!

⚡ Part 5: Adam — Best of Both Worlds

Adam = Momentum + RMSProp. Direction memory AND scale memory dono ek saath!

Adam: Adaptive Moment Estimation mₜ = β₁·mₜ₋₁ + (1-β₁)·gₜ ← 1st moment: direction (like Momentum) vₜ = β₂·vₜ₋₁ + (1-β₂)·gₜ² ← 2nd moment: scale (like RMSProp) Bias Correction (hat notation): m̂ₜ = mₜ/(1-β₁𝕼) ← correct early zero bias v̂ₜ = vₜ/(1-β₂𝕼) ← correct early zero bias Update: θₜ₊₁ = θₜ - α·m̂ₜ/(√v̂ₜ + ε) ← normalized update Defaults: β₁=0.9, β₂=0.999, ε=10⁻⁸, α=0.001

⚠️ Bias Correction Kyun? Zero Start Problem

m₀ = v₀ = 0 se start karte hain. Pehle steps mein moving averages zero ke paas hain (too much zero in them)!

Analogy: Class test average mein imaginary zero marks add karo — first real mark bahut chota lagta hai. Bias correction woh imaginary zeros remove karta hai!

t=1, β₁=0.9: m₁=0.4 (raw) → m̂₁ = 0.4/0.1 = 4. 10× correction! Very first step mein zaroori.

Worked: m₀=0, v₀=0, g₁=4, β₁=0.9, β₂=0.999, α=0.01

1
m₁ = 0.9(0) + 0.1(4) = 0.4 (direction memory)
2
v₁ = 0.999(0) + 0.001(16) = 0.016 (scale memory)
3
m̂₁ = 0.4/(1-0.9) = 0.4/0.1 = 4 (bias corrected!)
4
v̂₁ = 0.016/(1-0.999) = 0.016/0.001 = 16 (bias corrected!)
5
m̂₁/√v̂₁ = 4/√16 = 4/4 = 1 (normalized direction)
6
Δθ₁ = 0.01(1) = 0.01, θ₁ = θ₀ - 0.01

🎯 Part 6: AdamW — Decay ko Separate Karo

Regular L2 regularization Adam ke adaptive scaling ke saath interfere karta hai. Fix: decay aur adaptive step alag rakho!

AdamW θₜ₊₁ = (1-αλ)·θₜ - α·m̂ₜ/(√v̂ₜ + ε) (1-αλ)θₜ = decay term: parameter shrink toward 0 α·m̂ₜ/√v̂ₜ = Adam move: gradient-based update λ = weight decay coefficient (e.g., 0.01) TWO SEPARATE operations, not one mixed update!

🎯 Bicycle Controls Analogy

Steering = destination ki taraf jaana (Adam adaptive step)

Lane keeping = road centre ke paas rehna (weight decay)

AdamW in dono mix nahi karta — alag rakhta hai! Regular Adam mein adaptive scaling se decay ka actual effect change ho jaata tha.

Worked: θₜ=1.0, α=0.01, λ=0.1, Adam direction=0.5

1
Decay: (1-0.01×0.1)(1.0) = (1-0.001)(1.0) = 0.999
2
Adaptive move: 0.01×0.5 = 0.005
3
θₜ₊₁ = 0.999 - 0.005 = 0.994

🗺️ Complete Summary Table

Optimiser	Memory State	Solves	Limitation
SGD	None	Baseline	Noisy, zigzag
EWMA	sₜ (fading avg)	Smooth signals	Building block only
Momentum	vₜ (velocity)	Noisy/zigzag gradients	Can overshoot
NAG	vₜ + look-ahead	Overshoot prevention	More complex
AdaGrad	rₜ (Σg²)	Uneven coordinate scales	Denominator grows forever
RMSProp	sₜ (EWMA of g²)	Uneven scales + recovery	No direction memory
Adam	mₜ + vₜ	Direction + scale both	L2 reg interfered
AdamW	mₜ + vₜ + decay	Adam + proper weight decay	Extra λ hyperparameter

Exam Ke Liye Yaad Karo:
Adam = Momentum + RMSProp (direction memory + scale memory)
AdamW = Adam + decoupled weight decay (λ directly on θ, not through scale)
NAG = Momentum + look-ahead gradient measurement
RMSProp = AdaGrad + EWMA (scale can recover unlike AdaGrad)
Modern default: Adam or AdamW (almost everything!)