3

I wanted to clear my understanding of adjoints by relating it to the row and column space of a matrix. I recently spent time studying about Strang's 4 fundamantal subspaces(http://web.mit.edu/18.06/www/Essays/newpaper_ver3.pdf) which relate column space and null space.

I will refer to the adjoint map of a transformation(or matrix) $T$ as $T^T$ as we are only dealing with real spaces here. Let $v$ be from $R^N$ and $w$ from $R^M$

The definition of adjoint says $\langle T v, w\rangle=\langle v, T^T w\rangle$

Here is the derivation $\langle T v, w\rangle=(Tv)^T \cdot w=v^TT^T \cdot w=v^T(T^Tw)=\langle v,T^T w\rangle$

$Tv$ is a linear combination of the column space of $T$. We dot this with some $w$ already in that dimensional space. While $T^Tw$ is the linear combination of the row space which is dotted with our original $v$. Why do these yield the same result? I understand adjoints take a vector of one space to the appropriate space for the inner product but the operations are quite different. One is concerning column space and another row. I would like some intuition why they are equivalent.

  • " Why do these yield the same result?" I'm not sure what you're asking. It sounds as though you're asking why $Tv=T^Tw$, but that''s not so. What "two" yield the same result? – David C. Ullrich Jul 28 '19 at 11:55
  • I meant the two sides of the adjoint definition, $\langle T v, w\rangle$ and $\langle v, T^T w\rangle$. – Rahul Deora Jul 28 '19 at 13:03

2 Answers2

4

Coming from a pure, functional analysis background, I always preferred to step away from matrices and dot products, and into linear maps and dual spaces (and inner products where applicable). The dot product can be thought of as a special type of inner product, but there's a further generalisation which I find helpful in terms of intuition. Bear with me, as there's a bit to brush over.

Consider a (real) normed linear space $V$. There's the notion of the (topological) dual $V^*$, which is the set of continuous linear functionals on $V$. What is a linear functional? It's a linear map from $V$ to the scalar field $\Bbb{R}$. That is, it is a function. The set of such functions forms a vector space (a vector subspace of all real-valued functions on $V$), and indeed comes with its own norm: $$\|f\|_{V^*} = \sup_{\|x\| \le 1} |f(x)|.$$ The continuity of $f \in V^*$ guarantees that this supremum is always finite.

Now, the dual $V^*$ interacts with $V$ in a very natural way. We can, in a sense, "multiply" a vector in $V$ with a vector in $V^*$ in the following way: $$\langle v, f\rangle : = f(v).$$ Remember, $f \in V^*$ is a function of $V$. So, in this way, we obtain a scalar-valued product of $V$ and $V^*$. It's not difficult to see linearity here too: \begin{align*} \langle av_1 + v_2, f \rangle &= a\langle v_1, f \rangle + \langle v_2, f \rangle \\ \langle v , af_1 + f_2 \rangle &= a\langle v, f_1 \rangle + \langle v, f_2 \rangle. \end{align*} What else? Well, the definition of the norm of $f$ also yields the following inequality: $$|\langle v, f \rangle| \le \|v\|_V \|f\|_{V^*}.$$ Look familiar? This looks a lot like Cauchy-Schwarz inequality!

This isn't an exhaustive list either. This scalar-valued between $V$ and $V^*$ shares a lot of properties with inner products. Indeed, in (complete) inner product spaces (e.g. $\Bbb{R}^n$), there is a very natural way of identifying $V^*$ with $V$ (the Riesz Representation Theorem), and in applying this identification to the $V^*$ argument, this scalar product literally becomes the inner product.

In many ways, I think it's a far more simple, intuitive beast than the inner product anyway. All it is is simple function evaluation, whereas the dot product is not quite so immediate.


So, what about $\Bbb{R}^n$? We can freely construct elements in the dual of $\Bbb{R}^n$ by deciding what happens to its standard basis. Let $E = (e_1, \ldots, e_n)$ be the standard basis, and for each $i$, consider the unique linear function $f_i$ that sends $e_i$ to $1$, and $e_j$ to $0$ for $j \neq i$. Then, $F = (f_1, \ldots, f_n)$ forms a basis (indeed the dual basis to $E$) of $(\Bbb{R}^n)^*$. Note that, $$\langle e_i, f_j \rangle = \delta_{ij} = e_i \cdot e_j.$$ This suggests that we can identify $(\Bbb{R}^n)^*$ with $\Bbb{R}^n$, by turning a functional $f$ to its coordinate vector with respect to $F$, and this will turn the product between $\Bbb{R}^n$ and $(\Bbb{R}^n)^*$ into the standard dot product.

The convention is to denote vectors in $\Bbb{R}^n$ by column vectors (really, these are matrices in $\Bbb{R}^{n \times 1}$), and elements of the dual, $(\Bbb{R}^n)^*$, by row vectors in $\Bbb{R}^{1 \times n}$. Then, the scalar product (which, again, is more or less the dot product) can be represented by matrix multiplication.

For example, $\begin{pmatrix} 1 & 2 \end{pmatrix} \in \Bbb{R}^{1 \times 2}$ represents the linear functional $f(x, y) = x + 2y$. Evaluating it at $(1, -1) \in \Bbb{R}^2$ corresponds to the multiplication $$\begin{pmatrix} 1 & 2 \end{pmatrix}\begin{pmatrix} 1 \\ -1 \end{pmatrix} = -1 = f(1, -1),$$ which is, of course, the dot product between the vectors $(1, 2)$ and $(1, -1)$ in $\Bbb{R}^2$.

This is how I think about these things: column vectors are vectors in $\Bbb{R}^n$, and row vectors are functionals on $\Bbb{R}^n$! Matrices in $\Bbb{R}^{m \times n}$, are really just a collection of $m$ row vectors (i.e. functionals), ready to accept vectors in $\Bbb{R}^n$, and turn their scalar output into a (column) vector in $\Bbb{R}^m$.


So, with that in mind, let's tackle the adjoint. The adjoint is defined on a normed linear space the same way as it is on an inner product space, just with our alternate scalar product. Given $T : V \to W$, we define $T^*$ (leaving the domain and codomain blank for now) by: $$\langle Tv, f \rangle = \langle v, T^*f\rangle.$$ What are the natural choices for domain and codomain here? Well, $Tv$ is a vector in $W$, making the left product the one derived from $W$. Thus, it looks like $f$ is going to have to be an element of $W^*$. On the right, $v \in V$ (necessary for $Tv$ to make sense), so the product must be derived from $V$. Thus, we can expect $T^*f \in V^*$. So, $T^* : W^* \to V^*$.

What does it do functionals in $W^*$? Well, if we write these products in terms of their definitions, $$(T^*f)(v) = f(Tv) = (f \circ T)(v) \implies T^*f = f \circ T.$$ That is, the adjoint does something very simple and natural: it takes a functional on $W$, and turns it into a functional on $V$, by composing the map $T$ on the right! Again, these are natural operations given the objects at play here.


So, the hope is that this big info dump might help make it easier to see the answer to your question. An adjoint naturally operates on the dual space, i.e. a space of row vectors. Applying the adjoint to a dual/row vector $w$ "works" because $w$ is secretly a functional, and applying $T^*$ to it secretly composes $T$ with this functional. In that way, we get a functional/row vector which takes a vector $v$, applies $T$ to it, then applies $w$ to it. The row vector representing this total transformation is $T^*w$.

I hope that helps in some way. :-)

Theo Bendit
  • 50,900
0

$\newcommand{\ip}[2]{\langle #1,#2\rangle}$

Theo Bendit's answer is much more interesting than this one in terms of explaining wha's really going on mathematically. But if all you want to know is why $\ip{Tv}w=\ip v{T^Tw}$ in terms of matrices the answer is you can just work out both sides in terms of coordinates.

Say $T=(t_{j,k})$. Then the coordinates of $Tv$ are $(Tv)_j=\sum_kv_kt_{j,k}$, so $$\ip{Tv}w=\sum_{j,k}v_kw_jt_{j,k}.$$Similarly, since $(T^T)_{j,k}=t_{k,j}$, $$\ip v{T^T w}=\sum_{k,j}w_jv_kt_{j,k}.$$

Theo Bendit
  • 50,900
  • Yes it will take time for me to go through his reasoning. I know how it works out but I wanted to know what happens in terms of column and row spaces if you could answer from that standpoint – Rahul Deora Jul 28 '19 at 13:45
  • Sorry, I don't see an explanation for $<Tv,w>-<v,T^Tw>$ in terms of column spaces and row spaces. – David C. Ullrich Jul 28 '19 at 13:56