Because a network's training sits in the NTK linearization regime, that fact alone lets you compute the exact NTK formula for that specific architecture.

False. The regime explains WHY wide-network training behaves like linear kernel regression and generalizes predictably — it does not hand you the specific kernel formula for an architecture. Deriving that formula, and finite-width correction theory, are separate topics beyond what this regime claims.

Neural tangent kernel linearization regime Interview Q&A

NTK linearization regime

As width grows to infinity, a network’s output is well-approximated by its first-order Taylor expansion around initialization, so training reduces to linear regression in a fixed feature space defined by the NTK.

Example

A very wide two-layer network trained by gradient descent behaves like linear regression over features fixed at initialization, not a model that reshapes its own features while training.

Because training is just linear regression over a fixed feature space, gradient descent converges to one well-defined, predictable solution — the same guarantees classical kernel regression enjoys. That's the whole payoff: wide networks generalize predictably not because their optimization is mysterious, but because it secretly isn't nonlinear optimization at all.

Setup

A width-10,000 network starts at

f_0(x)

. After many gradient-descent steps its output is

f_t(x)

1
Write $f_t(x) \approx f_0(x) + \nabla_\theta f_0(x)\cdot(\theta_t-\theta_0)$ .
WhyAt large width, weights barely move from initialization, so this first-order term already captures almost all the change.
2
Treat $\nabla_\theta f_0(x)$ as a fixed feature map instead of something that evolves.
WhySince the gradient stays nearly constant during training, it acts like a static kernel feature vector, not a learned representation.
3
Fit $(\theta_t-\theta_0)$ by gradient descent on these fixed features.
WhyThis is exactly linear regression with the NTK as kernel, so the dynamics are convex and predictable.

Takeaway

Wide-network training tracks its own linearization so closely that the whole process is just kernel linear regression in disguise.

Setup

You know a specific wide CNN sits in the NTK regime and want its exact predictions.

Expected

You might expect the regime itself to hand you a formula for that architecture’s exact kernel, adjusted for its real, finite width.

What breaks

It doesn’t: deriving the exact kernel for a given architecture, and correcting for finite-width effects, are separate, more advanced theories beyond this regime’s claim.

Lesson

This regime only explains why wide-network training behaves linearly and predictably — not how to compute the specific kernel or finite-width corrections.

Neural tangent kernel linearization regime

NTK linearization regime

Sources

A focused session on your AI Research Scientist interview

More Generalization Theory & Phenomena questions