[Axler] Inner Product and Norm

Inner Products and Norms (Part 1)

Based on Linear Algebra Done Right by Sheldon Axler

Hello, I'm Sheldon Axler, the author of Linear Algebra Done Right. This video discusses part one of the section titled "Inner Products and Norms." In this video, we will focus on inner products.

1. Standard Notation & Motivation

Let's quickly recall our standard notation. $F$ denotes either the scalar field $\mathbf{R}$ of real numbers or the scalar field $\mathbf{C}$ of complex numbers. We also let $V$ denote a vector space over $F$.

To motivate the definition of the inner product, we'll start by considering the dot product on $\mathbf{R}^n$. Specifically, we'll begin by examining the case in $\mathbf{R}^2$. We have a vector with coordinates $(x_1, x_2)$, and the length of this vector is the square root of the sum of the squares of the coordinates: $\sqrt{x_1^2 + x_2^2}$.

If we move from $\mathbf{R}^2$ to $\mathbf{R}^n$, looking at a vector $\mathbf{x} = (x_1, \dots, x_n)$, we define the norm of $\mathbf{x}$, denoted $\|\mathbf{x}\|$, to be the square root of the sum of the squares of its components:

$$ \|\mathbf{x}\| = \sqrt{\sum_{k=1}^{n} x_k^2} $$

However, linearity is not immediately apparent in this formula. To introduce linearity, we define the dot product.

Dot Product in $\mathbf{R}^n$:
For vectors $\mathbf{x}$ and $\mathbf{y}$ in $\mathbf{R}^n$, the dot product is defined as: $$ \mathbf{x} \cdot \mathbf{y} = \sum_{k=1}^{n} x_k y_k $$

We see immediately that $\mathbf{x} \cdot \mathbf{x} = \|\mathbf{x}\|^2$. Now let's look at some key properties of the dot product:

1. For any vector $\mathbf{x} \in \mathbf{R}^n$, $\mathbf{x} \cdot \mathbf{x} \ge 0$.
2. $\mathbf{x} \cdot \mathbf{x} = 0$ if and only if $\mathbf{x} = \mathbf{0}$.
3. If we fix a vector $\mathbf{y} \in \mathbf{R}^n$, then the map $\mathbf{x} \mapsto \mathbf{x} \cdot \mathbf{y}$ is linear.
4. For any vectors $\mathbf{x}, \mathbf{y} \in \mathbf{R}^n$, $\mathbf{x} \cdot \mathbf{y} = \mathbf{y} \cdot \mathbf{x}$ (Commutativity).

---

2. The Complex Case

The properties above form the basis for the abstract definition of an inner product. This works perfectly for real vector spaces. For complex vector spaces, however, we need to consider an additional factor.

For a vector $\mathbf{z} = (z_1, \dots, z_n)$ in $\mathbf{C}^n$, we define its norm $\|\mathbf{z}\|$ using magnitudes:

$$ \|\mathbf{z}\| = \sqrt{\sum_{k=1}^{n} |z_k|^2} $$

To recover the relationship $\|\mathbf{z}\|^2 = \langle \mathbf{z}, \mathbf{z} \rangle$, the inner product for vectors $\mathbf{w}$ and $\mathbf{z}$ in $\mathbf{C}^n$ must be defined using the complex conjugate:

$$ \langle \mathbf{w}, \mathbf{z} \rangle = \sum_{k=1}^{n} w_k \overline{z_k} $$

---

3. Definition of Inner Product

We are now ready to define an inner product generally for vector spaces over either $\mathbf{R}$ or $\mathbf{C}$.

Definition: Inner Product

An inner product on a vector space $V$ is a function that takes each ordered pair of vectors $(\mathbf{u}, \mathbf{v})$ in $V$ to a scalar, denoted $\langle \mathbf{u}, \mathbf{v} \rangle$, with the following properties:

1. Non-negativity: $\langle \mathbf{v}, \mathbf{v} \rangle \ge 0$ for all $\mathbf{v} \in V$.
2. Definiteness: $\langle \mathbf{v}, \mathbf{v} \rangle = 0$ if and only if $\mathbf{v} = \mathbf{0}$.
3. Additivity in the first argument: $\langle \mathbf{u}_1 + \mathbf{u}_2, \mathbf{v} \rangle = \langle \mathbf{u}_1, \mathbf{v} \rangle + \langle \mathbf{u}_2, \mathbf{v} \rangle$.
4. Homogeneity in the first argument: $\langle \lambda \mathbf{u}, \mathbf{v} \rangle = \lambda \langle \mathbf{u}, \mathbf{v} \rangle$ for all $\lambda \in F$.
5. Conjugate Symmetry: $\langle \mathbf{u}, \mathbf{v} \rangle = \overline{\langle \mathbf{v}, \mathbf{u} \rangle}$ for all $\mathbf{u}, \mathbf{v} \in V$.

* Note: An inner product space is a vector space $V$ together with a specific inner product defined on it.

4. Examples

1. Euclidean Inner Product on $F^n$:
   $\langle \mathbf{w}, \mathbf{z} \rangle = \sum_{k=1}^{n} w_k \overline{z_k}$.
2. Weighted Euclidean Inner Product:
   Choose positive constants $c_1, \dots, c_n$. Define $\langle \mathbf{w}, \mathbf{z} \rangle = \sum_{k=1}^{n} c_k w_k \overline{z_k}$.
3. Inner Product on $C([-1, 1], \mathbf{R})$:
   $\langle f, g \rangle = \int_{-1}^{1} f(x)g(x) \, dx$.
4. Inner Product on $P(\mathbf{R})$:
   $\langle p, q \rangle = \int_{0}^{\infty} p(x)q(x) e^{-x} \, dx$.

5. Basic Properties

Based on the definition, we can derive the following properties:

• Linearity in the first argument: For fixed $\mathbf{u}$, the map $\mathbf{v} \mapsto \langle \mathbf{v}, \mathbf{u} \rangle$ is linear.
• Interaction with zero: $\langle \mathbf{0}, \mathbf{v} \rangle = 0$ and $\langle \mathbf{v}, \mathbf{0} \rangle = 0$.
• Additivity in the second argument: $\langle \mathbf{u}, \mathbf{v}_1 + \mathbf{v}_2 \rangle = \langle \mathbf{u}, \mathbf{v}_1 \rangle + \langle \mathbf{u}, \mathbf{v}_2 \rangle$.
• Conjugate Homogeneity in the second argument: $\langle \mathbf{u}, \lambda \mathbf{v} \rangle = \overline{\lambda} \langle \mathbf{u}, \mathbf{v} \rangle$.

(Note: In the real case where $F = \mathbf{R}$, the inner product is linear in the second argument as well.)

---

This concludes part one of the video on inner products and norms.

Inner Products and Norms (Part 2)

Based on Linear Algebra Done Right by Sheldon Axler

This discussion covers part two of the section on Inner Products and Norms. Throughout this post, we assume $V$ denotes an inner product space.

1. Definition of Norm

Definition

For a vector $v \in V$, the norm of $v$, denoted $\|v\|$, is defined as:

$$ \|v\| = \sqrt{\langle v, v \rangle} $$

Since $\langle v, v \rangle \ge 0$ by definition of the inner product, this square root is always valid.

Examples

1. Euclidean Norm on $\mathbf{F}^n$:
   For a vector $z = (z_1, \ldots, z_n) \in \mathbf{F}^n$, using the standard inner product: $$ \|z\| = \sqrt{|z_1|^2 + \cdots + |z_n|^2} $$ (Unless specified otherwise, $\mathbf{F}^n$ is assumed to have this Euclidean norm.)
2. Norm on Function Space:
   For continuous real-valued functions on $[-1, 1]$ with $\langle f, g \rangle = \int_{-1}^{1} f(x)g(x) \, dx$: $$ \|f\| = \sqrt{\int_{-1}^{1} (f(x))^2 \, dx} $$

---

2. Basic Properties of Norms

• Positivity: $\|v\| = 0$ if and only if $v = 0$. (This follows directly from the definiteness of the inner product.)
• Homogeneity: For any scalar $\lambda \in \mathbf{F}$, $\|\lambda v\| = |\lambda| \, \|v\|$.

Proof of Homogeneity:
$$ \|\lambda v\|^2 = \langle \lambda v, \lambda v \rangle = \lambda \langle v, \lambda v \rangle = \lambda \overline{\lambda} \langle v, v \rangle = |\lambda|^2 \|v\|^2 $$ Taking the square root of both sides gives $\|\lambda v\| = |\lambda| \, \|v\|$.

---

3. Orthogonality & The Pythagorean Theorem

Two vectors $u, v \in V$ are called orthogonal if $\langle u, v \rangle = 0$. This generalizes the geometric concept of perpendicularity.

• The zero vector is orthogonal to every vector.
• $\mathbf{0}$ is the only vector orthogonal to itself.

Pythagorean Theorem

If $u$ and $v$ are orthogonal vectors in $V$, then:

$$ \|u + v\|^2 = \|u\|^2 + \|v\|^2 $$

Proof:
$\|u + v\|^2 = \langle u + v, u + v \rangle = \langle u, u \rangle + \langle u, v \rangle + \langle v, u \rangle + \langle v, v \rangle$
Since $u, v$ are orthogonal, $\langle u, v \rangle = 0$ (and thus $\langle v, u \rangle = 0$).
Therefore, $\|u + v\|^2 = \|u\|^2 + 0 + 0 + \|v\|^2$.

---

4. Key Inequalities

Cauchy-Schwarz Inequality

For any vectors $u, v \in V$:

$$ |\langle u, v \rangle| \le \|u\| \, \|v\| $$

Equality holds if and only if one vector is a scalar multiple of the other.

Applications:

• In $\mathbf{R}^n$: $( \sum u_k v_k )^2 \le ( \sum u_k^2 ) ( \sum v_k^2 )$
• In Integrals: $( \int fg )^2 \le ( \int f^2 ) ( \int g^2 )$

Triangle Inequality

For any vectors $u, v \in V$:

$$ \|u + v\| \le \|u\| + \|v\| $$

Equality holds if and only if one vector is a non-negative real multiple of the other.

Proof (using Cauchy-Schwarz):
$$ \begin{aligned} \|u + v\|^2 &= \langle u + v, u + v \rangle \\ &= \|u\|^2 + \langle u, v \rangle + \langle v, u \rangle + \|v\|^2 \\ &= \|u\|^2 + 2\operatorname{Re}\langle u, v \rangle + \|v\|^2 \end{aligned} $$ Since $\operatorname{Re}\langle u, v \rangle \le |\langle u, v \rangle| \le \|u\|\|v\|$ (by Cauchy-Schwarz): $$ \|u + v\|^2 \le \|u\|^2 + 2\|u\|\|v\| + \|v\|^2 = (\|u\| + \|v\|)^2 $$ Taking square roots yields the result.

---

5. Parallelogram Identity

For any vectors $u, v \in V$:

$$ \|u + v\|^2 + \|u - v\|^2 = 2\|u\|^2 + 2\|v\|^2 $$

This identity reflects the geometry of a parallelogram: the sum of the squares of the lengths of the diagonals ($u+v$ and $u-v$) equals the sum of the squares of the lengths of the four sides.

Proof:
Expand $\|u + v\|^2 + \|u - v\|^2$: $$ (\|u\|^2 + \langle u, v \rangle + \langle v, u \rangle + \|v\|^2) + (\|u\|^2 - \langle u, v \rangle - \langle v, u \rangle + \|v\|^2) $$ The cross terms cancel out, leaving $2\|u\|^2 + 2\|v\|^2$.

---

This concludes part two on inner products.