Modeling
AtlasBrowse all 76
Menu
LearnProbabilityMechanisms, distributions, transformations, and limits.Bayesian learningPosterior updates, Thompson sampling, and Bayesian optimization.Deep learningActivations, likelihood-derived objectives, and the architectures built from them.01Activations02Likelihood to loss03ObjectivesSemi-supervisedUse a few labels and many unlabeled examples without hiding the assumptions.ReferenceDistributionsRelationship mapSource & method

Nonlinearity gives depth


something to learn.

Without nonlinear activations, stacked linear layers collapse into one linear map. The function’s shape controls sparsity, saturation, smoothness, and gradient flow.

Fundamentals

Pre-activationweighted input z = Wx + b

Activationnonlinear output h = f(z)

Gradientlocal sensitivity used by backpropagation

h = activation(Wx + b)
Activation

ReLU

f(x) = max(0, x)
−5+5
output f(x)0.500
gradient f′(x)1.000
Use it when

Strong default for hidden layers, especially CNNs and MLPs.

Watch for

Zero gradient for negative inputs can create dead units.

History & theory

Rectifying units appeared in early neural models and were popularized for deep networks around 2010–2011 because they trained more reliably than saturating sigmoids.

Hidden MLP / CNN

Start with ReLU. Try GELU or SiLU when smooth gating improves optimization.

Transformer block

GELU and SiLU are common defaults; architecture conventions matter.

Output probability

Sigmoid for independent binary outcomes; softmax for one categorical outcome.

Positive parameter

Softplus for variance, scale, concentration, or rate without a hard zero kink.

Next lesson

From likelihood to loss

Derive MSE, MAE, and cross-entropy as negative log-likelihoods of observation models.

Follow the derivation