What is a derivative? Slope, limit, chain rule, interactively

The derivative of a function $f$ at a point $x$ is the number

In words: how fast the output changes per unit change of the input, measured so close to $x$ that nothing but the local behaviour of $f$ matters.

The same number has a second reading that is more useful than the first. For small $h$,

Near $x$, the function behaves like a straight line with slope $f'(x)$ — the tangent. Everything on this page is that one line read in different directions: the tangent is its picture, finite differences estimate it, the chain rule composes it, backpropagation computes it for millions of inputs at once, and gradient descent uses it to walk downhill.

The route follows the first lecture of Andrej Karpathy’s Neural Networks: Zero to Hero, where the derivative is where building a neural network from scratch starts. Every panel below is live: change something and the numbers are recomputed, never typed in.

The lecture works with the parabola $3x^2 - 4x + 5$. This page uses a deliberately uneven curve instead, so its slope changes size and sign from place to place and there is more than one valley to fall into:

One point has no slope

Through a single point of a curve pass infinitely many straight lines. Which of them “has the slope of the curve”?

Drag the point along the curve and turn the line by its round handle. From a distance, plenty of lines look about right. The inset magnifies a small window around the point, and there the difference shows: a wrong line keeps crossing the curve at an angle however far you zoom, while one line merges with the curve. That line is the tangent, and its slope is $f'(x)$.

The legend measures what the lens shows: the gap between curve and line divided by the distance from the point, at $\Delta x = 0.1$, $0.01$ and $0.001$. At $x = 3$ a flat line gives $2.64,\ 3.42,\ 3.49$ — the ratio closes in on $3.50$ and stays there, and every other wrong slope settles the same way on its own error. Turn the line to slope $3.5$ and the same three numbers become $-0.86,\ -0.08,\ -0.01$: ten times smaller for every tenfold zoom. Move the point, and the slope that worked a moment ago no longer does. A derivative is not one number for the whole curve but a new number at every point.

Why exactly one line? Expand the curve around $x$. The gap between it and a line of slope $m$, per unit of distance $u$, is

As $u$ shrinks, everything but the first term disappears. Any error in the slope survives the zoom; only $m = f'(x)$ lets the ratio go to zero. This, and not “the line that touches the curve at one point”, is what a tangent is: a line can touch a curve once and still cross it, and a tangent can cross the curve (at an inflection point it does).

From secant to tangent: the limit

The point alone gives no slope, so take a second one. The points $x$ and $x+h$ fix a line, the secant, and its slope is the ratio of the legs of the orange triangle:

It is the tangent of the secant’s angle to the horizontal, $\tan\alpha$ — rise over run.

Now drag the second point towards the first. The $h$ slider does the same job: its right end is always the edge of the plot and its left end a single pixel, whatever the zoom. As $h$ shrinks, the secant turns onto the tangent.

At $x = 3$ the true slope is $f'(3) = 3.4998$. The secant gets there step by step:

At $h = 1$ the secant even has the wrong sign: it jumps over the crest just past $x = 3$. Every further tenfold decrease of $h$ makes the error ten times smaller.

Now zoom in — with the zoom slider, or a pinch on a trackpad. The second point keeps its place on screen, so $h$ shrinks with the view, and the curve straightens under it: the secant turns onto the tangent without $h$ being touched. At a thousandth of the original width the curve is indistinguishable from its tangent. This is why the limit exists at all: a smooth function is, locally, linear. The derivative is the slope of that local line.

The derivative as a function

Since every point has its own slope, the derivative is itself a function, $x \mapsto f'(x)$. It is rarely computed from the limit directly. A handful of rules, each proven once from the definition, is enough:

• a sum differentiates term by term: $(u + v)' = u' + v'$;
• a constant factor stays: $(c\,u)' = c\,u'$;
• powers: $(x^n)' = n\,x^{n-1}$;
• a product: $(u\,v)' = u'\,v + u\,v'$;
• $(\sin x)' = \cos x$;
• $(e^x)' = e^x$ — a function that is its own slope (it and its multiples $c\,e^x$ are the only ones);
• the chain rule for a composition: $\bigl(g(u(x))\bigr)' = g'(u(x)) \cdot u'(x)$.

Together they reach $\tanh x = \dfrac{e^{2x} - 1}{e^{2x} + 1}$, the activation of the first neuron in the lecture: the rules give $(\tanh x)' = 1 - \tanh^2 x$. That slope is $1$ at $0$ and fades towards $0$ as $|x|$ grows, which is why a neuron driven deep into saturation barely learns — whatever gradient arrives at it is multiplied by almost nothing.

For our curve, the chain rule turns $\sin(2.5x)$ into $\cos(2.5x) \cdot 2.5$, and the rest is term by term:

That formula is what the dashed tangents on this page are drawn from.

Where $f'(x) = 0$, the tangent is horizontal. On the plotted range that happens 7 times; where $f'$ changes from negative to positive the curve has a local minimum — the deepest at $x \approx 1.885$, where $f \approx -4.406$ — and where it changes from positive to negative, a local maximum.

Not every function has a derivative everywhere. $|x|$ has slope $-1$ to the left of $0$ and $+1$ to the right; the secant has no single limit at $0$, so $|x|$ is not differentiable there. The activation function ReLU, $\max(0, x)$, has the same corner, and neural-network libraries simply agree to use $0$ there.

Zoom into the corner and see for yourself. The panel draws two secants from the point, one to the left and one to the right, with $h$ tied to the zoom. On a smooth curve, zooming in straightens the picture and the two secants fold into one line. At the corner of $|x|$ the picture never changes: at every zoom it is the same V, the left secant stays at $-1$ and the right at $+1$. There is no single number for the derivative to be.

How small should h be on a computer?

In exact arithmetic, a smaller $h$ is always better. On a computer it is not.

The secant slope $\bigl(f(x+h) - f(x)\bigr)/h$, the forward difference, subtracts two nearly equal numbers. A 64-bit float carries about 16 significant digits, so the difference loses as many of them as $f(x+h)$ and $f(x)$ share. The rounding error that is left is then divided by the tiny $h$.

Two errors pull in opposite directions:

• truncation error — the curve is not a line — is about $\tfrac12 |f''(x)|\,h$ and shrinks with $h$;
• rounding error is about $\varepsilon\,|f(x)|/h$, where $\varepsilon = 2^{-52} \approx 2.2 \cdot 10^{-16}$ is the float precision, and grows as $h$ shrinks.

Their sum is smallest near $h \approx \sqrt{\varepsilon} \approx 1.5 \cdot 10^{-8}$.

The central difference $\bigl(f(x+h) - f(x-h)\bigr)/2h$ uses points on both sides. The $f''$ terms cancel, its truncation error is of order $h^2$, and its best $h$ moves up to about $\sqrt[3]{\varepsilon} \approx 6.1 \cdot 10^{-6}$.

The panel computes both formulas in ordinary 64-bit arithmetic and plots their error against $h$; drag across it to move $h$.

Read the plot from right to left, the way $h$ shrinks. At first both lines fall with slopes 1 and 2: every tenfold decrease of $h$ divides the forward error by 10 and the central error by 100. Then rounding takes over and both lines turn up into noise.

At $x = 3$ the forward difference is most accurate at $h \approx 4 \cdot 10^{-9}$, with error $1.3 \cdot 10^{-8}$. The central difference reaches $3 \cdot 10^{-11}$ at $h \approx 4 \cdot 10^{-6}$ — about 450 times more accurate, with $h$ 1000 times larger.

This is why numerical derivatives are used to check analytic ones — a “gradient check” with the central difference and $h$ around $10^{-5}$ — and not to train networks: each estimate costs extra function evaluations per input, and the answer is good to only about ten digits.

The chain rule on a computation graph

Real formulas are compositions of simple operations. Break

into steps: $e = a \cdot b$, $d = e + c$, $L = d \cdot s$. This is a computation graph.

The forward pass computes values left to right. With $a = 2,\ b = -3,\ c = 10,\ s = -2$: $e = -6,\ d = 4,\ L = -8$.

The backward pass answers a different question: for every node, how much does $L$ change when that node is nudged? That is the derivative $\partial L / \partial \text{node}$, called its gradient. A node’s value and its gradient are different numbers.

Each operation only needs to know its own local derivative:

• for $L = d \cdot s$: $\partial L / \partial d = s$ and $\partial L / \partial s = d$;
• for $d = e + c$: $\partial d / \partial e = \partial d / \partial c = 1$ — addition passes the gradient through unchanged;
• for $e = a \cdot b$: $\partial e / \partial a = b$ and $\partial e / \partial b = a$ — multiplication hands each input the value of the other.

Rates along a chain multiply. The lecture borrows George F. Simmons’s example: if a car travels twice as fast as a bicycle, and the bicycle four times as fast as a walking man, the car travels $2 \cdot 4 = 8$ times as fast as the man. The chain rule multiplies local derivatives along the path in exactly that way:

Move the backward-pass slider: every notch processes one operation, from right to left, and labels each edge it crosses with its local derivative. Then drag any input sideways.

It starts from $\partial L / \partial L = 1$: nudging the output by $\varepsilon$ changes it by $\varepsilon$. At the end $\partial L/\partial a = 6,\ \partial L/\partial b = -4,\ \partial L/\partial c = -2,\ \partial L/\partial s = 4$, and dragging checks them the old way. Drag $a$ from $2$ to $2.5$ and $L$ moves by $3$ — exactly $\partial L/\partial a \cdot \Delta a = 6 \cdot 0.5$. The gradient is that prediction: how much the output moves per unit nudge of the input. Here it is exact, because $L$ is linear in each input on its own; for a curved function it holds only for small nudges, which is the whole content of $f(x+h) \approx f(x) + f'(x)\,h$.

Now move all four inputs at once. “Step along the gradient” adds to every input $0.01$ times its own gradient, and $L$ rises from $-8.00$ to $-7.29$. The gradient predicted a rise of $0.01 \cdot (6^2 + 4^2 + 2^2 + 4^2) = 0.72$; the true one is $0.71$. The gap is the products of inputs — $a \cdot b$, $e \cdot s$ — that change together and that a first-order prediction cannot see: $L$ is linear in each input alone, not in all of them at once. Stepping against the gradient would lower $L$ instead, and that is the whole of training, as the last section shows.

A ”—” before a node is reached means “not computed yet”, which is not the same as zero. Drag $s$ to $0$ and every gradient to its left becomes zero, although none of those nodes has value zero: when the last factor is zero, nothing upstream can move $L$.

This is backpropagation. One backward sweep yields the gradient of the single output with respect to every node, at a cost of a small constant times the forward pass. A network has one loss and millions of parameters, and that ratio is why it trains backwards and not forwards.

One variable, several paths: gradients add

What if a value is used more than once? Take

The variable $x$ enters the graph three times: twice as an input of $x \cdot x$ and once in $k \cdot x$. Each use is a separate path to $L$, and each path sends back its own contribution: $x$ from each side of the product, $k$ from the second term. The multivariable chain rule says the derivative is their sum:

That is why a backpropagation engine writes grad += ... and not grad = .... The second button replays the bug: each contribution overwrites the one before, only the last survives, and the tangent drawn from that gradient misses the curve.

Drag the point to $x = -1.5$ with $k = 3$. The sum is zero and the tangent is horizontal, yet no path has disappeared: $-1.5 - 1.5 + 3 = 0$. Contributions can cancel, and a zero gradient does not mean nothing depends on $x$.

Two different things are called “accumulating” gradients, and it pays not to mix them up. Within one backward pass, contributions must add. Between training steps, the stored gradients must be reset to zero first — otherwise the previous step’s gradient is added to the new one. The first is the chain rule; forgetting the second is a bug.

Order matters too. A node may pass its gradient on only after every use of it has reported back, which is why the backward pass visits nodes in reverse topological order.

What the derivative is for: gradient descent

The sign of $f'(x)$ says which way $f$ goes up. To go down, step the other way:

The step size $\eta$ is the learning rate. Where the slope is steep, the step is long; near a minimum, where $f' \approx 0$, it is short, and the walk stops by itself.

From $x = 3$ with $\eta \approx 0.03$, the walk reaches the deepest valley, $x \approx 1.885$, in 20 steps. Drag the start (or move its slider) to $x = 3.3$ instead — just past the peak at $3.198$ — and the dashed preview shows the same rule taking it the other way, into the shallower valley at $4.294$. The derivative is local information: it knows which way is down here, not where the best valley is.

Now raise $\eta$. Near the minimum at $1.885$ the curvature is $f''(1.885) \approx 19.5$, and the walk settles only if $\eta < 2/f'' \approx 0.10$. Above that, every step overshoots by more than it corrects, and the point bounces between the walls of the valley instead of settling.

Training a neural network is this same loop. $x$ becomes millions of weights, $f$ becomes the loss on the training data, $f'(x)$ becomes the gradient vector, and backpropagation computes it in one sweep. The rest of the lecture builds exactly that on top of the tiny engine in the panels above.

Short answers

What is a derivative in one sentence?

The derivative f′(x) is the rate at which f changes per unit change of its input near x — the slope of the straight line that best approximates the graph at that point.

Why is the tangent defined by a limit?

A single point does not determine a line's slope: infinitely many lines pass through it. A second point at distance h does determine one, the secant, and the tangent is what the secant turns into as h goes to zero.

Does every function have a derivative?

No. |x| has a corner at 0: the slope is −1 on the left and +1 on the right, so the secant has no single limit. ReLU, max(0, x), has the same corner; neural-network libraries simply pick a value there, usually 0.

What is the difference between a derivative and a gradient?

A derivative is the rate of change of a function of one variable. A gradient collects the partial derivatives of a function of many variables, one per input, into a vector; it points in the direction of steepest increase.

Numerical, symbolic or automatic differentiation — which is which?

Numerical differentiation estimates the slope from function values at nearby points and trades truncation error for rounding error. Symbolic differentiation manipulates formulas. Automatic differentiation, which backpropagation is an instance of, applies the chain rule to the actual sequence of operations a program executed and gives exact derivatives up to floating-point rounding.

Why does backpropagation run backwards?

A network has one loss and millions of parameters. Going backwards from the loss, one sweep over the graph yields the derivative of that single output with respect to every input, at a cost of a small constant times the forward pass. Going forwards would take one sweep per parameter.

Why are gradients accumulated with += in backpropagation?

Because a value that is used in several places influences the output along several paths, and by the multivariable chain rule its derivative is the sum of the contributions of all those paths. Assigning instead of adding keeps only the last path.

Further reading

• A. Karpathy, “The spelled-out intro to neural networks and backpropagation: building micrograd”, Neural Networks: Zero to Hero, lecture 1, 2022. The lecture this page follows.
• 3Blue1Brown, Essence of Calculus, video series, 2017. The geometric intuition behind derivatives and the chain rule.
• M. Spivak, Calculus, 4th ed., Publish or Perish, 2008. Chapters 9–10: the definition of the derivative and the rules, proven properly.
• J. Nocedal, S. J. Wright, Numerical Optimization, 2nd ed., Springer, 2006. Chapter 8: finite differences, their errors, and automatic differentiation.
• A. G. Baydin, B. A. Pearlmutter, A. A. Radul, J. M. Siskind, “Automatic differentiation in machine learning: a survey”, Journal of Machine Learning Research 18 (2018), 1–43.
• A. Griewank, A. Walther, Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, 2nd ed., SIAM, 2008. The reference on reverse-mode differentiation.

Nikolay Eremeev, Senior Data Infrastructure Engineer. Writes about what he builds in production. CV ·hi@nikolay-eremeev.com