Since the random-rescaling technique described above doesn't add any explicit penalty term to the loss function, it doesn't actually count as a regularization technique.

False. Regularization's shared goal is constraining capacity or injecting noise to reduce overfitting — it isn't limited to methods that add a penalty term to the loss. A technique that injects noise during training (like this rescaling, or dropout) still serves that same purpose even though nothing is added to the loss function.

Regularization techniques Interview Q&A

Regularization

Regularization is any training-time method that constrains a model’s capacity or injects noise, trading a little fit on the training data for better generalization to new data — the shared goal behind every regularization technique, whatever its mechanism.

Example

Imagine a model memorizing training examples perfectly but failing on new ones; adding a penalty that discourages overly large weights forces it to rely on simpler, more general patterns instead of memorized noise.

Every regularization technique follows the same schema: a specific training-time mechanism paired with an intended effect on a particular overfitting symptom. Weight decay's mechanism (penalizing large weights) targets the symptom of weights growing unboundedly to fit noise. Dropout's mechanism (randomly zeroing activations) targets the symptom of units co-adapting to memorize specific training patterns. When you meet an unfamiliar technique, ask: what does it change during training, and what overfitting behavior does that change suppress?

Weight decay

Adds an L2 penalty on weight magnitude to the loss
Targets weights growing large to fit training noise
Shrinks weights uniformly toward zero each update

Dropout

Randomly zeroes activations during each forward pass
Targets units co-adapting to memorize specific patterns
Forces redundant, more general representations

Myth

Regularization just means adding an L2 penalty on the weights, like weight decay does.

Reality

L2 penalty is only one mechanism among many. Regularization is any method — capacity constraint or noise injection — that reduces overfitting; dropout’s random noise and label smoothing’s softened targets count too, even though neither adds a penalty term to the loss.

Regularization techniques

Regularization

Sources

A focused session on your AI Research Scientist interview

More Deep Network Training Basics questions