Training is often
maximum likelihood.
For independent observations, likelihoods multiply. Logs turn the product into a sum; negation turns maximization into minimization. Most familiar losses fall out of this one move.
Model an observation
→yᵢ ~ p(y | model(xᵢ; θ))Write joint likelihood
→L(θ) = ∏ᵢ p(yᵢ | xᵢ, θ)Take negative log
→ℒ(θ) = −Σᵢ log p(yᵢ | xᵢ, θ)Differentiate
θ ← θ − η∇θℒObservation assumptionNegative log-likelihoodFamiliar objective
y ~ Normal(ŷ, σ²)constant + (y−ŷ)² / 2σ²Mean squared error
y ~ Laplace(ŷ, b)constant + |y−ŷ| / bMean absolute error
y ~ Bernoulli(p)−y log p − (1−y)log(1−p)Binary cross-entropy
y ~ Categorical(p)−Σ yₖ log pₖCross-entropy
y ~ Poisson(λ)λ − y log λ + constantPoisson NLL
Next lesson
Objective functions
Probe each loss curve to see how its assumptions change penalties and gradients.
Compare the losses