Differentiation

For any function, what we fundamentally want to understand is: if the input changes by an infinitesimally small amount, how does the output of the function respond?

Before we dive into the idea of differentiation, it’s helpful to first understand two key concepts: what a function is, and what we mean by the slope of a line.

1. A function is a rule that takes an input and gives exactly one output. For example, if we have a function $f(x) = x^2$, then inputting $x = 2$ gives us the output $f(2) = 4$. Every valid input $x$ is mapped to a corresponding output $f(x)$.
2. The slope measures how steep a line is — it tells us how much the value of $y$ changes for a given change in $x$. For two points on a line, say $(x_1, y_1)$ and $(x_2, y_2)$, the slope is given by: $\text{slope} = \frac{y_2 - y_1}{x_2 - x_1} = \frac{\text{rise}}{\text{run}}$

Now let’s start to look into the definition of a derivative through an illustration:

Using the illustration, we can understand how the function changes when its input changes by a small amount $\Delta x$. To do this, we calculate the slope of the line passing through two points on the function: $P = (x, f(x))$ and $Q = (x + \Delta x, f(x + \Delta x))$. This line is called the secant line, and its slope is given by:

This slope represents the average rate of change of the function between the two points.

To find the instantaneous rate of change at the point $x$, we bring point $Q$ closer and closer to point $P$ by letting $\Delta x$ approach zero. As this happens, the secant line approaches the tangent line at $x$, and its slope becomes the derivative of the function at that point:

This limit, if it exists, tells us how the function responds to an infinitesimally small change in the input. A function $f(x)$ is said to be differentiable at a point $x$ if this limit exists and yields a finite value. In other words, the function must be smooth (no sharp corners or breaks) in an open interval around $x$.

Now, given we know that one fundamental formulae, yopu are supposed to like whiz through the sections below - you can occasionally take a pen and paper to solve it for yourself - but otherwise just accept it and move on.

Constant Rule

If $f(x) = c$, where $c$ is a constant, then the output of the function is the same for every input $x$. That is, no matter how $x$ changes, $f(x)$ remains unchanged. This means the rate of change, or the slope of the function, is zero at all points:

Using the definition of the derivative:

Since the function always outputs the same value $c$, we have:

So the difference in output is:

Therefore:

A constant function has no change, and hence its derivative is zero everywhere.

Sum Rule

then using,

we have:

Product Rule

If $f(x)$ and $g(x)$ are both differentiable functions, then the derivative of their product is:

We start with the definition of the derivative:

We add and subtract a middle term $f(x + \Delta x)g(x)$ to split the expression:

Group the terms:

Take the limit of each part:

Which simplifies to:

So,

Chain Rule

If $y = f(g(x))$, where both $f$ and $g$ are differentiable functions, then the derivative of the composition is:

Let’s consider the function $y = f(g(x))$. To find $\frac{dy}{dx}$, we start with the definition of the derivative:

Let $\Delta g = g(x + \Delta x) - g(x)$. Since $g$ is differentiable at $x$, as $\Delta x \to 0$, we also have $\Delta g \to 0$.

We now rewrite the difference quotient as:

Take the limit of both parts:

As $\Delta g \to 0$, the first term becomes $f’(g(x))$, and the second becomes $g’(x)$. So:

Summary

1. Constant Rule: \(\frac{d}{dx}[c] = 0\)

2. Sum Rule: \(\frac{d}{dx}[a \cdot f(x) + b \cdot g(x)] = a \cdot f'(x) + b \cdot g'(x)\)

3. Product Rule: \(\frac{d}{dx}[f(x) \cdot g(x)] = f(x) \cdot g'(x) + f'(x) \cdot g(x)\)

4. Quotient Rule: \(\frac{d}{dx} \left[\frac{f(x)}{g(x)}\right] = \frac{f'(x) \cdot g(x) - f(x) \cdot g'(x)}{[g(x)]^2}\)

5. Chain Rule: \(\frac{d}{dx}[f(g(x))] = f'(g(x)) \cdot g'(x)\)

Let the complication begin!

Okay, so now that we’ve formed a basis for like the normal-ass differentiation we cna start to tackle the really complicated. If you’ve like taken an introductory differentiation course or something, depending on the country of your orgin - you might have been given a list to memorize - mostly I think the concern behind that was that you not in a million years would be able to derive eveyrthing from scratch in the 3 hours stipulated for the exam that decides your life so -

Derivative of $\cos(x)$

We want to compute the derivative of $f(x) = \cos(x)$. By definition, the derivative is:

Now apply the trigonometric identity for $\cos(A + B)$:

Substitute this into the limit:

Group terms:

Now use the standard limits:

(the quick explanation for $\lim_{\Delta x \to 0} \frac{\sin(\Delta x)}{\Delta x} = 1$ is that $\sin(0) = 0$, so $\sin(\Delta x)$ is not exactly $0$ but like somewhere near it, and so we say $\sin(\Delta x) = \Delta x$, and $ \lim_{\Delta x \to 0} \frac{\Delta x}{\Delta x} = 1$)

So we get:

Therefore:

References

1. Limit Definition and Illustration
2. Backpropagation Through Time
3. Unrelated read on ODEs and euler
4. Unrelated read on Topological sorting - this is used in both micrograd and DNAs