Let $V$ be an inner product space and let $\Omega \subseteq V$ be a subset. The set is called:

1. Orthogonal: if $0 \not \in \Omega$ and any distinct pair of vectors $v\neq u$ in $\Omega$ are perpendicular, namely $\angles {u,v} =0$ .

2. Orthonormal: if in addition $\norm v = 1$ for all $v\in \Omega.$

Suppose that $\{v_1,...,v_n\}\subseteq V$ is an orthogonal set. Then:

1. Pythagoras: $\norm {\sum_1^n v_i}^2 = \sum_1^n \norm {v_i}^2$ .

2. Coefficients: If $v\in span \{ v_1, ..., v_n \}$ , namely $v=\sum_1^n \alpha_i v_i$ , then $\alpha_i = \frac{\angles {v,v_i}}{\norm {v_i}^2}$ .

   In particular for an orthonormal set we get

   $$v = \sum_1^n \angles {v,v_i} v_i \; , \text{ and } \; \norm{v}^2 = \sum_1^n |\angles {v,v_i}|^2.$$

1. Under the assumption that $\angles {v_i,v_j} = 0$ for $i\neq j$ , we get that

   $$\norm{\sum_1^n v_i}^2 = \angles {\sum_{i=1}^n v_i, \sum_{j=1}^n v_j}=\sum_{i,j=1}^n \angles{v_i,v_j}=\sum_{i=1}^n \angles{v_i,v_i}=\sum_1^n \norm {v_i}^2.$$

2. This is a straight forward computation:

$$\angles {v,v_j}= \angles{\sum_{i=1}^n \alpha_i v_i,v_j} = \sum_{i=1}^n \alpha_i \angles{v_i,v_j} = \alpha_i \norm {v_i}^2$$

Any orthogonal set is linearly independent.

In particular, in an inner product space of dimension $n$ , an orthogonal set of size $n$ is a basis.

Let $\Omega$ be an orthogonal set, and suppose that we can write zero as linear combination of its elements (which by definition is a finite combination). This means that

$$ 0 = \sum_1^n \alpha_i \omega_i $$

for some elements $\omega_i \in \Omega$ . However, since the set is orthogonal, we know that $\alpha_i=\frac{\angles {0,\omega_i}}{\norm {\omega_i}^2}=0$ , so that this combination is trivial, and we conclude that $\Omega$ is linearly independent.

Consider the following set of functions which contains the constant 1 function, and for each integer $0\leq p$ , and integer $0\leq n \leq 2^p - 1$ the function

$$s_{p,i}(x) = \begin{cases}-1 & \frac{n}{2^p}< x< \frac{n+0.5}{2^p} \\ 1 & \frac{n+0.5}{2^p} < x < \frac{n+1}{2^p} \\ 0 & else \end{cases}.$$

In words: each such function chooses a section of length $1/2^p$ and has $-1$ on half of it and $+1$ on the second half. It should not be very hard to convince yourselves that these functions form an orthogonal set (prove it!).

What would you change to make it an orthonormal set?

Consider the inner product on functions on $[-\pi,\pi]$ defined by

$$\frac {1}{\pi}\int_{-\pi}^\pi f(x)\overline{g(x)}\dx .$$

The set of functions

$$\Omega=\{\cos(kx) : k\geq 0\} \cup \{\sin(kx) : k\geq 1\}$$

is (almost) orthonormal - the only exception is that $cos(0x)\equiv 1$ has norm $2$ instead of $1$ .

If you know some complex integration, this is a very simple exercise using exponentials. For example, writing $\cos(nx)=\frac{e^{inx}+e^{-inx}}{2}$ , we get that

$$\align{\angles {\cos(mx),\cos(nx)} & =\angles {\frac{e^{imx}+e^{-imx}}{2},\frac{e^{inx}+e^{-inx}}{2}} \\ & =\frac {1}{4}(\angles {e^{imx},e^{inx}} + \angles {e^{imx},e^{-inx}} + \angles {e^{-imx},e^{inx}} + \angles {e^{-imx},e^{-inx}})}.$$

These expressions are easy to compute:

$$\angles {e^{inx},e^{imx}} = \frac{1}{\pi} \int_{-\pi}^\pi e^{i(m-n)x}dx.$$

If $m=n$ , then the integrand is simply $1$ , so the integral is $\frac{2\pi}{\pi} = 2$ . Otherwise, when $m-n\neq 0$ we get that

$$\angles {e^{inx},e^{imx}} = \frac{1}{\pi} \frac{1}{i(m-n)}e^{i(m-n)x} \mid_{-\pi}^\pi = 0.$$

Ignoring the complex integration, we can use directly the trigonometric equalities. Recall that the product of two cosine satisfies:

$$\cos(\theta)\cos(\varphi) = \frac {\cos(\theta-\varphi)+\cos(\theta+\varphi)}{2}.$$

Using this equality, we obtain that for $m,n\geq 0$ we have that

$$\begin{align} \angles {\cos(mx),\cos(nx)} & =\frac {1}{\pi}\int_{-\pi}^\pi \cos(mx)\overline{\cos(nx)} \dx \\ & =\frac {1}{\pi}\int_{-\pi}^\pi \frac{\cos((m-n)x)+\overline{\cos((m+n)x)}}{2} \dx = \begin{cases} 2&m=n=0\\ 1&m=n\geq 1 \\ 0 & m\neq n \end{cases}. \end{align} $$

A similar computation can be done for the rest of the inner products.

Finally, one more way is by using some basic integral operations in an interesting way. The integral above for $m=n=0$ is trivial, and in general it is symmetric, so let’s assume that $m\neq 0$ . Using integration by parts, with the idea that integrating\differentiating $\cos(x)$ twice we return to the same place, we obtain the following:

$$\begin{align}\frac {1}{\pi}\int_{-\pi}^\pi \cos(mx)\overline{\cos(nx)} \dx & = \overbrace{\frac{1}{m\pi}\sin(mx)\cos(nx)\mid _{-\pi}^\pi}^{=0} + \frac {n}{m\pi}\int_{-\pi}^\pi \sin(mx)\sin(nx) \dx\\& = -\overbrace{\frac {n}{m^2\pi} \cos(mx)\sin(nx)\mid_{-\pi}^\pi} +\frac {n^2}{m^2\pi} \int_{-\pi}^\pi \cos(mx)\cos(nx) \dx\end{align}.$$

It then follows that

$$\left(1-\frac{n^2}{m^2}\right)\frac {1}{\pi}\int_{-\pi}^\pi \cos(mx)\overline{\cos(nx)} \dx = 0,$$

so unless $m=n$ , the integral must be $0$ . As for the $m=n\geq 1$ case, namely integrating $\cos^2(nx)$ , you should first convince yourself that $\int_{-\pi}^\pi \cos^2(nx) \dx = \int_{-\pi}^\pi \sin^2(nx) \dx$ (why?). Given that, we obtain that

$$2\int_{-\pi}^\pi \cos^2(nx) \dx = \int_{-\pi}^\pi \cos^2(nx) \dx+\int_{-\pi}^\pi \sin^2(nx) \dx =\int_{-\pi}^\pi 1 \dx = 2\pi,$$

so we conclude that for $n\geq 1$ we have

$$ \angles {\cos(nx),\overline{\cos(nx)}} = 1.$$

Given a point $v\in V$ and a set $W\subseteq V$ , find the distance from $v$ to $W$ , and if possible find the closest point to $v$ .

Let $W\leq V$ be a finite dimensional subspace, and let $\{w_1,...,w_k\}$ be an orthonormal basis for $W$ . The orthogonal projection to $W$ is defined by

$$P_W(v):=\sum_1^k \angles {v,w_i}w_i.$$

This orthogonal projection of $v$ is the closest point to $v$ in $W$ .

Let’s try to approximate the function $f(x)=x$ using the orthogonal step functions we defined previously, and we shall use only the first four such functions.

(soon to be pictures)

These four function form an orthogonal set, and in order to use the theorem we need to normalize them, namely $\hat{f}_i := \frac{f_i}{\norm {f_i}}$ , so that

$$ \hat{f}_1 = f_1, \;\; \hat{f}_2 = f_2, \;\; \hat{f}_3 = \sqrt{2}f_3, \;\; \hat{f}_4 = \sqrt{2}f_4.$$

The coefficients are then going to be

$$\angles{x,\hat{f}_1}=\frac{1}{2}, \;\; \angles{x, \hat{f}_2}=\frac{1}{4}, \;\; \angles{x, \hat{f}_3} = \angles{x, \hat{f}_4} =\frac{1}{16}. $$

This means that we get the approximation:

$$ x \sim \frac{1}{2}\cdot f_1 + \frac{1}{4}\cdot f_2 + \frac{1}{16} \cdot 2 \cdot (f_3+f_4).$$

And finally, unsurprisingly, we get the approximation:

Compute $\angles{x, f_i}$ for any of the step function in the orthogonal basis. Do it one time directly using the integral, and one time “geometrically”. Try to conjecture how the approximation will look like, when taking all the functions with $p\leq P$ for some positive integer $P$ , and then prove it. Try to do the same for other function (e.g. $f(x)=x^2$ ) - can you say in general how the approximation will look like?

Note that in the example above we had $P=2$ since the corresponding segments of our functions were of length $1=1/2^0$ , $1/2$ and $1/2^2$ .

Similar to the previous example, we can try to approximate $f(x)=x$ using sine waves (explain why we don’t need cosines).

Add some explanations…

When does the norm convergence implies also pointwise convergence? And if there isn’t pointwise convergence, then what can we say?

To get some intuition about the last question, try to approximate some other “nice” function and try to guess the answer for this question for those functions. Try it on a range of families of functions like: constant, linear, polynomial, exponential, unbounded function, noncontinuous, etc.