\documentclass{article}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{enumerate}
\usepackage{ifthen}
\usepackage{ulem} % for strikethrough \sout{...} command
\usepackage{color}

\usepackage{hyperref}

\usepackage{amssymb}
\usepackage{wasysym}


\usepackage{tikz}
\tikzset{node distance=1.5cm, auto}

\setlength{\oddsidemargin}{-0.50in}
\setlength{\evensidemargin}{-0.50in}
\setlength{\topmargin}{-0.5in}
\setlength{\headheight}{0.0in}
\setlength{\headsep}{0.0in}
\setlength{\topskip}{0.0in}
\setlength{\textheight}{9.6in}
\setlength{\textwidth}{7.5in}


\newcommand{\myspace}{\mbox{\parbox[c][0.35in]{0.35in}{\hfill}}}
\newcommand{\varspace}[2]{\mbox{\parbox[c][#1]{#2}{\hfill}}}

\newcommand{\bdot}{\, {\scriptstyle{\bullet}} \,}
\newcommand{\B}[1]{{\bf {#1}}}
\newcommand{\la}{\langle}
\newcommand{\ra}{\rangle}

\newcommand{\diagquotient}[2]{\displaystyle{{#1 \atop \;}\hspace*{-0.1in} \mbox{\put(0,0){\line(2,1){2}}} \hspace*{0.1in} {\; \atop #2}}}


\newcommand{\F}{\mathbb{F}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\K}{\mathbb{K}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\C}{\mathbb{C}}

\newcommand{\LEMMA}{\vspace{0.05in} \noindent \fbox{\large\sc Lemma:}  }
\newcommand{\COR}{\vspace{0.05in} \noindent \fbox{\large\sc Corollary:}  }
\newcommand{\DEF}{\vspace{0.05in} \noindent \fbox{\large\sc Definition:}  }
\newcommand{\THM}{\vspace{0.05in} \noindent \fbox{\large\sc Theorem:}  }
\newcommand{\EX}{\vspace{0.05in} \noindent \fbox{\large\sc Example:}  }
\newcommand{\bproof}{\vspace{0.1in} \noindent {\bf Proof:} }
\newcommand{\eproof}{$\blacksquare$ \vspace{0.05in}}

\newcommand{\force}{{$\mbox{\;}$}}

\begin{document}

\noindent
\parbox{1.5in}{\bf Math 3510}
\hfill {\Large \bf Differentiability} \hfill
\parbox{1.5in}{\bf \hfill Supplement}

\vspace{0.3in}

For a function of one variable, differentiability is synonymous with the existence of the derivative. However, the notion of differentiability is much more subtle for functions of more than one variable.

\vspace{0.05in}

Recall that a function $f(x)$ is {\it differentiable} at $x=a$ if $\displaystyle f'(a) = \lim\limits_{h \to 0} \dfrac{f(a+h)-f(a)}{h}$ exists. Let's recast this definition is little. First, set $x=a+h$, then $h=x-a$. Now $h \to 0$ becomes $x \to a$. This means that being differentiable at $x=a$ is equivalent to the existence of the limit $\displaystyle \lim\limits_{x \to a} \dfrac{f(x)-f(a)}{x-a}$.

\vspace{-0.05in}

Let's manipulate our current definition a little more. We have $f'(a) = \displaystyle \lim\limits_{x \to a} \dfrac{f(x)-f(a)}{x-a}$ so\dots
$$0=\displaystyle \lim\limits_{x \to a} \dfrac{f(x)-f(a)}{x-a}-f'(a) = \lim\limits_{x \to a} \dfrac{f(x)-f(a)-f'(a)(x-a)}{x-a} = \lim\limits_{x \to a} \dfrac{f(x)-[f(a)+f'(a)(x-a)]}{x-a}$$

This means that $f(x)$ is differentiable at $x=a$ if $\dfrac{f(x)-[\mbox{linearization of $f$ at $x=a$}]}{x-a} \to 0$ as $x \to a$. We have recast differentiability into a statement about a comparison between $f$ and its linearization. We have arrived at a working definition of differentiability in general.

\vspace{0.05in}

\noindent {\bf Definition:} A function is {\bf differentiable} at a point if it can be {\it well-approximated} by a linearization at that point.

\vspace{0.05in}

Let's make the above definition more concrete. Consider a function $f:\mathbb{R}^n \to \mathbb{R}^m$ and a fixed point $\B a = (a_1,\dots,a_n)$ [actually, we only need $f$ to be defined on an open ball containing $\B a$]. Fix notation $$f(\B x) = f(x_1,\dots,x_n) = (f^1(x_1,\dots,x_n),\dots,f^m(x_1,\dots,x_n))$$ so that $f^j(\B x)$ is a scalar valued function on $\mathbb{R}^n$ (i.e. $f^j:\mathbb{R}^n \to \mathbb{R}$).

\vspace{0.05in}

\noindent {\bf Definition:} $f(\B x)$ is {\bf differentiable} at $\B x = \B a$ if there is some linear map $J_{\bf a}:\mathbb{R}^n \to \mathbb{R}^m$ such that\dots

\vspace{-0.1in}

$$\lim\limits_{\B x \to \B a} \dfrac{\|f(\B x) - L(\B x) \|}{\|\B x - \B a\|} = \lim\limits_{\B x \to \B a} \dfrac{\Big\|f(\B x) - [f(\B a) + J_{\bf a}(\B x - \B a)]\Big\|}{\|\B x - \B a\|} = 0$$
Here we have $L(\B x) = f(\B a) + J_{\B a}(\B x - \B a)$ is the {\it linearization} of $f(\B x)$ at $\B x = \B a$ and $J_{\bf a}$ is the {\it Jacobian} of $f(\B x)$ at $\B x = \B a$.

\vspace{0.15in}

\noindent {\bf Theorem:} If $f(\B x)$ is differentiable at $\B x = \B a$, then the Jacobian at $\B x = \B a$ is unique.

\vspace{0.05in}

\bproof {\it (Sketch)} Let $J$ and $K$ both be Jacobians at $\B x = \B a$. Then $\|J(\B x - \B a) - K(\B x - \B a)\| = \|f(\B x)-[f(\B a)+K(\B x - \B a)] - (f(\B x)-[f(\B a)+J(\B x-\B a)])\| \leq \|f(\B x)-[f(\B a)+K(\B x - \B a)]\| +  \| f(\B x)-[f(\B a)+J(\B x-\B a)] \|$. Thus 

$$\lim\limits_{\B x \to \B a} \dfrac{\|J(\B x - \B a) - K(\B x - \B a)\|}{\|\B x - \B a\|} \leq \lim\limits_{\B x \to \B a} \dfrac{\|f(\B x)-[f(\B a)+K(\B x - \B a)]\|}{\|\B x -\B a\|} + \dfrac{\|f(\B x)-[f(\B a)+J(\B x - \B a)]\|}{\|\B x -\B a\|} = 0+0 = 0$$
{\it Sketchy part:} This means that the norm of the operator $J-K$ is $\|J-K\|=0$ and thus $J-K=0$ so that $J=K$. \eproof

Often we prefer to represent linear operators as (coordinate) matrices. First, consider projecting onto the $i$-th output coordinate: $f$ projects to $f^i$. Say $J_{\B a}$ projects to $J_{\bf a}^i$ (in fact, let's identify this linear map its coordinate matrix/vector). Thus $f^i(\B x) \approx L^i(\B x) = f^i(\B a)+J_{\B a}^i \bdot (\B x -\B a)$. In particular, we have that $\lim\limits_{\B x \to \B a} \dfrac{f^i(\B x) - [f^i(\B a) + J^i_{\bf a} \bdot (\B x - \B a)]}{\|\B x - \B a\|} = 0$. Since this limit exists along any continuous curve through $\B x=\B a$, we have (approaching along $x_k=a_k$, $k\not=j$ and $x_j=t$),  $\lim\limits_{t \to a_j} \dfrac{f^i(a_1,\dots,a_{j-1},t,a_{j+1},\dots,a_n) - [f^i(\B a) + J^i_{\bf a} \bdot (0,\dots,0,t-a_j,0,\dots,0)]}{|t-a_j|} = 0$. Denote the $j$-th entry of the coordinate matrix/vector $J^i_{\B a}$ by $J_{ij}(\B a)$. Then we have $\lim\limits_{t \to a_j} \dfrac{f^i(a_1,\dots,a_{j-1},t,a_{j+1},\dots,a_n) - [f^i(\B a) + J_{ij}({\bf a}) (t-a_j)]}{t-a_j} = 0$. After some algebraic manipulations, we get $\lim\limits_{t \to a_j} \dfrac{f^i(a_1,\dots,a_{j-1},t,a_{j+1},\dots,a_n) - f^i(a_1,\dots,a_n)}{t-a_j} = J_{ij}({\bf a})$. We call the $m \times n$ matrix with $(i,j)$-entries $J_{ij}({\bf a})$ the {\it Jacobian matrix} of $f(\B x)$ based at $\B x = \B a$ (this is the coordinate matrix for the operator $J_{\bf a}$). 

\vspace{0.05in}

\noindent {\bf Theorem:} If $f(\B x)$ is differentiable at $\B x=\B a$, then its component functions' partial derivatives exist (at $\B x =\B a$). Moreover, the $(i,j)$-entry of the Jacobian matrix of $f(\B x)$ is $J_{ij}(\B a) = (f^i)_{x_j}(\B a)$ (i.e. the partial derivative of the $i$-th component function with respect to the $j$-th input variable). In particular, the partials of the component functions of $f$ exist (at $\B x =\B a$).

\newpage

The converse of this theorem does not hold! From Calculus III, we know that just because a limit exists along several lines, does not mean that the full multivariate limit exists. So it should not be surprising to learn that: {\it existence of partials does {\bf not} imply differentiability!} I will forgo giving an actual counterexample. We will soon see why we don't tend to run into this problem in practice.

\vspace{0.05in}

We learn in Calculus I that differentiable functions are always continuous functions. This is still true.

\vspace{0.05in}

\noindent {\bf Theorem:} [Differentiability implies continuity] Let $f(\B x)$ be differentiable at $\B x=\B a$. Then $f(\B x)$ is continuous at $\B x=\B a$.

\vspace{0.05in}

\bproof Suppose that $f({\bf x})$ is differentiable at ${\bf x}={\bf a}$. Then 
$\displaystyle \lim\limits_{\B x \to \B a} \dfrac{\|f(\B x) - L(\B x) \|}{\|\B x - \B a\|}=0$. For this limit to be 0, the numerator must limit to 0. This means that $\lim\limits_{\B x \to \B a} \Big|f(\B x) - [f(\B a) + J_{\B a}(\B x - \B a)]\Big| = 0$ so that $\lim\limits_{\B x \to \B a} f(\B x) - f(\B a) - J_{\B a}(\B x - \B a) = \B 0$. Now $J_{\B a}(\B x - \B a) \to 0$ as $\B x \to \B a$ because $J_{\B a}(\B 0)=\B 0$ and linear operators are continuous (everywhere). Thus $\lim\limits_{\B x \to \B a} f(\B x) - f(\B a)= \B 0$ and so $\lim\limits_{\B x \to \B a} f(\B x) = f(\B a)$ which is means $f(\B x)$ is continuous at $\B x = \B a$. \eproof

\vspace{0.05in}

It should come as no surprise that there are non-differentiable continuous functions (i.e. the converse of this theorem does not hold). In fact, we knew this in Calculus I. It is easy to come up with continuous functions which have ``sharp corners'' where they cannot be differentiated. A two variable example would be something like $f(x,y)=|x-y|$. This function isn't differentiable at any point where $x=y$ (the graph of this function looks like a creased piece of paper with the fold along the line $y=x$).

Now for a final theorem which lays differentiability concerns to rest.

\vspace{0.05in}

\noindent {\bf Theorem:} [Continuous partials implies differentiability] Let the component functions of $f(\B x)$ have continuous (first) partials at $\B x =\B a$. Then $f(\B x)$ is differentiable at $\B x =\B a$.

\vspace{0.05in}

I will not provide a proof of this theorem. Its proof is more technical than the last two results. Also, just as with the other theorems, the converse of this theorem does not hold. There are differentiable functions which have discontinuous partials. Again, I will forgo giving a concrete counterexample -- such an example is tricky to cook up. Every function we typically run into has continuous partials (where they are defined). This means that for us, computing partials (and calling on this theorem) will {\it prove} differentiability. 

\vspace{0.05in}

\begin{center}
\begin{tikzpicture}
   \node (CP) {Continuous Partials};   

   \node (D) [below of=CP] {Differentiable};

   \node(GL) [left of=CP,xshift=0.4cm,yshift=-0.1cm] {\,};        % Ghost node to the left of CP.
   \node(GR) [right of=CP,xshift=-0.4cm,yshift=-0.1cm] {\,};    % Ghost node to the right of CP.

   \node (G) [below of=D] {\,};  % Ghost below D.

   \node (PE) [left of=G] {Partials Exist};

   \node (C) [right of=G] {Continuous};

   \draw[-latex] (CP)--(D);
   \draw[-latex] (D)--(PE);
   \draw[-latex] (D)--(C);
   \draw[-latex] (GR) to[bend left=30] (C);
   \draw[-latex] (GL) to[bend right=30] (PE);
 \end{tikzpicture}
\end{center}

The figure above summarizes our main differentiability theorems. Keep in mind that none of the arrows go backwards in general. Well, unless we have single variable functions, then ``partials exist'' (meaning the derivative exists) is the {\it same} as ``differentiable''. But again, that's only for functions of one variable.

\vspace{0.1in}

\noindent {\bf Note:} Most Calculus III texts define differentiability as follows: $f(x,y)$ is differentiable at $(x,y)=(a,b)$ if the partial derivatives $f_x(a,b)$ and $f_y(a,b)$ exist as well as locally defined functions $\epsilon_1(x,y)$ and $\epsilon_2(x,y)$ such that $$f(x,y)=\underbrace{f(a,b)+f_x(a,b)(x-a)+f_y(a,b)(y-b)}_{\mathrm{linearization}}+\underbrace{\epsilon_1(x,y)(x-a)+\epsilon_2(x,y)(y-b)}_{\mathrm{error\  terms}}$$ 

\vspace{-0.05in}

\noindent and in addition \quad $\lim\limits_{(x,y) \to (a,b)} \epsilon_1(x,y)=0$ \quad and \quad $\lim\limits_{(x,y)\to (a,b)} \epsilon_2(x,y)=0$. 

It turns out that this definition and our definition (in the case of scalar valued functions of two variables) are equivalent. You can see that this definition says that $f(x,y)$ is differentiable at $(x,y)=(a,b)$ if $f(x,y)$ is equal to its linearization at $(x,y)=(a,b)$ plus some suitably structured error terms.  If the term $f(a,b)$ is brought over to the other side of the defining equation, we get something like $dz=f_x\,dx+f_y\,dy+\epsilon_1\,dx+\epsilon_2\,dy$ so $dz = (f_x+\epsilon_1)\,dx+(f_y+\epsilon_2)\,dy$. In other words, $f_x$ and $f_y$ don't perfectly capture the change in $f$, but come close (up to some error terms).

It is my opinion that the definition presented in this handout is more conceptually clear. It also has the advantage of being immediately generalizable to functions from $\mathbb{R}^n$ to $\mathbb{R}^m$ and even to functions on arbitrary Banach spaces (whatever those are). 

\end{document}