In my lecture notes, we have a linear map $\mathcal{A}(X)=X_{11}$ that maps $X\in S^2$ to its first element. It then claims that $\mathcal{A}^*(X_{11})= \begin{bmatrix} X_{11}&0\\ 0&0 \end{bmatrix}$ is the adjoint.
To me I understand it by $\mathcal{A}(X)= \begin{bmatrix}1&0\end{bmatrix} X \begin{bmatrix}1\\0\end{bmatrix} =X_{11}$ , so we solve for $X$: $$ \begin{align} X&=\begin{bmatrix}1\\0\end{bmatrix} \Bigg(\begin{bmatrix}1&0\end{bmatrix} X \begin{bmatrix}1\\0\end{bmatrix}\Bigg) \begin{bmatrix}1&0\end{bmatrix}\\ &= \begin{bmatrix}1\\0\end{bmatrix} X_{11} \begin{bmatrix}1&0\end{bmatrix}\\ &=\mathcal{A}^*(X_{11}). \end{align}$$
However, (coming from a physics background) this seems like we found the inverse $\mathcal{A}^{-1}$, not $\mathcal{A}^*$ the adjoint. From what I know (from physics), the adjoint of an operator is its (conjugate) transpose, which we would have derived by doing: $$\mathcal{A}^*(X_{11})= \begin{bmatrix}1&0\end{bmatrix}^T X_{11} \begin{bmatrix}1\\0\end{bmatrix}^T = \begin{bmatrix}1\\0\end{bmatrix} X_{11} \begin{bmatrix}1&0\end{bmatrix}. $$
Is either of these the correct way? I'm guessing it's probably the latter, and that here we happen to have $\mathcal{A}^*=\mathcal{A}^{-1}$, but I'm confused because I've never seen this done before, and I've also never seen a map/operator not represented just by a single matrix.