\documentclass{article}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{enumerate}
\usepackage{ifthen}
\usepackage{ulem} % for strikethrough \sout{...} command
\usepackage{color}

\usepackage{tikz}
\tikzset{node distance=1.5cm, auto}

\setlength{\oddsidemargin}{-0.50in}
\setlength{\evensidemargin}{-0.50in}
\setlength{\topmargin}{-0.5in}
\setlength{\headheight}{0.0in}
\setlength{\headsep}{0.0in}
\setlength{\topskip}{0.0in}
\setlength{\textheight}{9.6in}
\setlength{\textwidth}{7.5in}


\newcommand{\myspace}{\mbox{\parbox[c][0.35in]{0.35in}{\hfill}}}
\newcommand{\varspace}[2]{\mbox{\parbox[c][#1]{#2}{\hfill}}}

\newcommand{\bdot}{\, {\scriptstyle{\bullet}} \,}
\newcommand{\B}[1]{{\bf {#1}}}
\newcommand{\la}{\langle}
\newcommand{\ra}{\rangle}

\newcommand{\force}{{$\mbox{\;}$}}

\begin{document}

\noindent
\parbox{1.5in}{\bf Math 2130}
\hfill {\Large \bf Differentiability} \hfill
\parbox{1.5in}{\bf \hfill Supplement}

\vspace{0.3in}

For a function of one variable, differentiability is synonymous with the existence of the derivative. However, the notion of differentiability is much more subtle for functions of more than one variable.

\vspace{0.05in}

Recall that a function $f(x)$ is {\it differentiable} at $x=a$ if $\displaystyle f'(a) = \lim\limits_{h \to 0} \dfrac{f(a+h)-f(a)}{h}$ exists. Let's recast this definition is little. First, set $x=a+h$, then $h=x-a$. Now $h \to 0$ becomes $x \to a$. This means that being differentiable at $x=a$ is equivalent to the existence of the limit $\displaystyle \lim\limits_{x \to a} \dfrac{f(x)-f(a)}{x-a}$.

\vspace{-0.05in}

Let's manipulate our current definition a little more. We have $f'(a) = \displaystyle \lim\limits_{x \to a} \dfrac{f(x)-f(a)}{x-a}$ so\dots
$$0=\displaystyle \lim\limits_{x \to a} \dfrac{f(x)-f(a)}{x-a}-f'(a) = \lim\limits_{x \to a} \dfrac{f(x)-f(a)-f'(a)(x-a)}{x-a} = \lim\limits_{x \to a} \dfrac{f(x)-[f(a)+f'(a)(x-a)]}{x-a}$$

This means that $f(x)$ is differentiable at $x=a$ if $\dfrac{f(x)-[\mbox{linearization of $f$ at $x=a$}]}{x-a} \to 0$ as $x \to a$. We have recast differentiability into a statement about a comparison between $f$ and its linearization. We have arrived at a working definition of differentiability in general.

\vspace{0.05in}

\noindent {\bf Definition:} A function is {\bf differentiable} at a point if it can be {\it well-approximated} by a linearization at that point.

\vspace{0.05in}

Let's make the above definition more concrete. Consider a function of $n$ variables, $f(\B x) = f(x_1,\dots,x_n)$, and fix a point $\B a = \la a_1,\dots,a_n \ra$. We will use ${\bf a}$ as our base point. Let $\B v = \la v_1,\dots,v_n \ra$ be some fixed vector. We can create a linear function based at ${\bf x}={\bf a}$ as follows: $L(\B x) = f(\B a)+\B v \bdot \B (\B x - \B a)$. In fact, this would be the linearization of $f({\bf x})$ at $\B x = \B a$ if $\B v = \nabla f(\B a)$. However, for now let's leave $\B v$ ambiguous.

\vspace{0.05in}

Consider $f(\B x) - L(\B x)$. We cannot ``divide by $\B x - \B a$'' since in the same way we divided by $x-a$ when we had a function of a single variable since now our objects are vectors. However, dividing magnitudes {\it does} make sense. Now we have a more concrete definition of differentiability\dots

\vspace{0.05in}

\noindent {\bf Definition:} $f(\B x)$ is {\bf differentiable} at $\B x = \B a$ if there is some vector $\B v$ such that\dots

\vspace{-0.1in}

$$\lim\limits_{\B x \to \B a} \dfrac{|f(\B x) - L(\B x) |}{\|\B x - \B a\|} = \lim\limits_{\B x \to \B a} \dfrac{\Big|f(\B x) - [f(\B a) + \B v \bdot (\B x - \B a)]\Big|}{\|\B x - \B a\|} = 0$$
When this is the case, we {\it define} $\nabla f(\B a) = \B v$.

\vspace{0.05in}

Notice that in the special case $n=1$ (i.e. a single variable function), we have\dots 
$$\lim\limits_{x \to a} \dfrac{|f(x)-[f(a)+v(x-a)]|}{|x-a|}=0  \quad \Longleftrightarrow \quad \lim\limits_{x \to a} \dfrac{f(x)-[f(a)+v(x-a)]}{x-a}=0
  \quad \Longleftrightarrow \quad \lim\limits_{x \to a} \dfrac{f(x)-f(a)}{x-a}=v$$
In other words, for single variable functions, differentiability is the same as the derivative existing (in this case $f'(a)=v$).

\vspace{0.05in}

Next, recall that if a limit exists, it exists and matches if we approach along any continuous curve. Let's see what happens when we approach along coordinate curves (approach parallel to coordinate axes), say $\B r(t) = \la a_1,\dots,a_{i-1},t,a_{i+1},\dots,a_n \ra$ (so as $t \to a_i$ we get $\B r(t) \to \B a$). We must have that\dots

\vspace{-0.1in}


$$0=\lim\limits_{t \to a_i} \dfrac{| f(\B r(t)) - f(\B a) - \B v \bdot (\B r(t) - \B a) |}{\| \B r(t) - \B a\|}$$ $$= \lim\limits_{t \to a_i} \dfrac{| f(a_1,\dots,a_{i-1},t,a_{i+1},\dots,a_n)-f(a_1,\dots,a_n)-\B v \bdot \la 0,\dots,0,t-a_i,0,\dots,0 \ra|}{\| \la 0,\dots,0,t-a_i,0,\dots 0\ra\|}$$
$$=\lim\limits_{t \to a_i} \dfrac{| f(a_1,\dots,a_{i-1},t,a_{i+1},\dots,a_n)-f(a_1,\dots,a_n)-v_i(t-a_i)|}{|t-a_i|}$$
 
\noindent This is equivalent to saying that\dots

\vspace{-0.15in}

$$ \lim\limits_{t \to a_i} \dfrac{f(a_1,\dots,a_{i-1},t,a_{i+1},\dots,a_n)-f(a_1,\dots,a_n)}{t-a_i} = v_i$$
which is precisely the same as stating that $f_{x_i}(\B a) = v_i$. In other words, {\it if} our function is differentiable at $\B x = \B a$, then $\nabla f(\B a) = \la f_{x_1}(\B a),\dots,f_{x_n}(\B a) \ra$. This means that our ``new'' definition of the gradient matches our old definition (as it should). Also, we just learned that\dots

\vspace{0.05in}

\noindent {\bf Theorem:} Differentiability implies the existence of partials.\\
\force \hspace{0.65in} In particular, if $f(\B x)$ is differentiable at $\B x = \B a$, then $f_{x_i}(\B a)$ exists for $i=1,\dots,n$.

\vspace{0.05in}

The converse of this theorem does not hold! This should make pretty good sense at this point. We know that just because a limit exists along several lines, does not mean that the full multivariate limit exists. So it should not be surprising to learn that: {\it existence of partials does {\bf not} imply differentiability!} I will forgo giving an actual counterexample. We will soon see why we never run into this problem in practice.

\vspace{0.05in}

We learn in Calculus I that differentiable functions are always continuous functions. This is still true for functions of more than one variable.

\vspace{0.05in}

\noindent {\bf Theorem:} Differentiability implies continuity.

\vspace{0.05in}

{\bf Proof:} Suppose that $f({\bf x})$ is differentiable at ${\bf x}={\bf a}$. Then 
$\displaystyle \lim\limits_{\B x \to \B a} \dfrac{|f(\B x) - L(\B x) |}{\|\B x - \B a\|}=0$. For this limit to be 0, the numerator must limit to 0. This means that $\lim\limits_{\B x \to \B a} \Big|f(\B x) - [f(\B a) + \B v \bdot (\B x - \B a)]\Big| = 0$ so that $\lim\limits_{\B x \to \B a} f(\B x) - f(\B a) - \B v \bdot (\B x - \B a) = 0$. Now $\B v \bdot (\B x - \B a) \to 0$ as $\B x \to \B a$. Thus $\lim\limits_{\B x \to \B a} f(\B x) - f(\B a)=0$ and so $\lim\limits_{\B x \to \B a} f(\B x) = f(\B a)$ which is means $f(\B x)$ is continuous at $\B x = \B a$.

\vspace{0.05in}

It should come as no surprise that there are non-differentiable continuous functions (i.e. the converse of this theorem does not hold). In fact, we knew this in Calculus I. It is easy to come up with continuous functions which have ``sharp corners'' where they cannot be differentiated. A two variable example would be something like $f(x,y)=|x-y|$. This function isn't differentiable at any point where $x=y$ (the graph of this function looks like a creased piece of paper with the fold along the line $y=x$).

Now for a final theorem which lays all concerns to rest.

\vspace{0.05in}

\noindent {\bf Theorem:} Continuous partials implies differentiability.

\vspace{0.05in}

I will not provide a proof of this theorem. Its proof is more technical than the last two results. Also, just as with the other theorems, the converse of this theorem does not hold. There are differentiable functions which have discontinuous partials. Again, I will forgo giving a concrete counterexample -- such an example is tricky to cook up. Every function we run into in this class will have continuous partials (where they are defined). This means that for us, computing partials (and calling on this theorem) will {\it prove} differentiability. 

\vspace{0.05in}

\noindent {\bf Example:} $f(x,y)=\sin(xy^2)$ has partial derivatives $f_x=\cos(xy^2) \cdot y^2$ and $f_y = \cos(xy^2) \cdot 2xy$. Since $f_x$ and $f_y$ are continuous everywhere, we have (by this last theorem) that $f(x,y)=\sin(xy^2)$ is differentiable everywhere. 

\begin{center}
\begin{tikzpicture}
   \node (CP) {Continuous Partials};   

   \node (D) [below of=CP] {Differentiable};

   \node(GL) [left of=CP,xshift=0.4cm,yshift=-0.1cm] {\,};        % Ghost node to the left of CP.
   \node(GR) [right of=CP,xshift=-0.4cm,yshift=-0.1cm] {\,};    % Ghost node to the right of CP.

   \node (G) [below of=D] {\,};  % Ghost below D.

   \node (PE) [left of=G] {Partials Exist};

   \node (C) [right of=G] {Continuous};

   \draw[-latex] (CP)--(D);
   \draw[-latex] (D)--(PE);
   \draw[-latex] (D)--(C);
   \draw[-latex] (GR) to[bend left=30] (C);
   \draw[-latex] (GL) to[bend right=30] (PE);
 \end{tikzpicture}
\end{center}

The figure above summarizes our ``big'' theorems. Keep in mind that none of the arrows go backwards in general. Well, unless we have single variable functions, then ``partials exist'' (meaning the derivative exists) is the {\it same} as ``differentiable''. But again, that's only for functions of one variable.

\vspace{0.25in}

\noindent {\bf Note:}

Our textbook (like many Calculus III texts) defines differentiability as follows: $f(x,y)$ is differentiable at $(x,y)=(a,b)$ if the partial derivatives $f_x(a,b)$ and $f_y(a,b)$ exist as well as locally defined functions $\epsilon_1(x,y)$ and $\epsilon_2(x,y)$ such that $$f(x,y)=\underbrace{f(a,b)+f_x(a,b)(x-a)+f_y(a,b)(y-b)}_{\mathrm{linearization}}+\underbrace{\epsilon_1(x,y)(x-a)+\epsilon_2(x,y)(y-b)}_{\mathrm{error\  terms}}$$ 

\vspace{-0.05in}

\noindent and in addition \quad $\lim\limits_{(x,y) \to (a,b)} \epsilon_1(x,y)=0$ \quad and \quad $\lim\limits_{(x,y)\to (a,b)} \epsilon_2(x,y)=0$. 

It turns out that this definition and our definition (in the case of two variables) are equivalent. You can see that the book's definition says that $f(x,y)$ is differentiable at $(x,y)=(a,b)$ if $f(x,y)$ is equal to its linearization at $(x,y)=(a,b)$ plus some suitably structured error terms.  If the term $f(a,b)$ is brought over to the other side of the defining equation, we get something like $dz=f_x\,dx+f_y\,dy+\epsilon_1\,dx+\epsilon_2\,dy$ so $dz = (f_x+\epsilon_1)\,dx+(f_y+\epsilon_2)\,dy$. In other words, $f_x$ and $f_y$ don't perfectly capture the change in $f$, but come close (up to some error terms).

It is my opinion that the definition presented in this handout is more conceptually clear. It also has the advantage of being immediately generalizable to functions from $\mathbb{R}^n$ to $\mathbb{R}^m$ and even to functions on arbitrary Banach spaces (whatever those are). Adapting our definition to functions from $\mathbb{R}^n$ to $\mathbb{R}^m$ leads to the definition of the {\it Jacobian matrix}. More on that later.

\end{document}