Unbounded operators in Hilbert space.

Question

If $A\subset B $, $A$ not equals to $B$, are unbounded operators, I need to prove that:

$(a)$ If $B$ is selfadjoint, then $A$ is symmetric, but not selfadjoint.

$(b)$ If $A$ is selfadjoint, then $B$ is not symmetric.

Answer 1

It follows from the fact that (unbounded) self-adjoint operators on a Hilbert space are maximally symmetric, that is if $S$ is self-adjoint and $S\subset T$ with $T$ symmetric, then $T=S$: indeed, we have (see Rudin, functional analysis, theorem $13.15$) $$T\subset T^*\subset S^*=S\subset T.$$

$(a)$ $A$ is symmetric because $B$ is symmetric and $B=A$ on the domain of $A$. Now if $A$ is self-adjoint, then $A$ is maximally symmetric, and since $A\subset B$ and $B$ is symmetric, we must have $A=B$, a contradiction.

$(b)$ Similarly, if $A$ is self-adjoint, then $A$ is maximally symmetric, and then, if $B$ is symmetric, we must have $B=A$ which is impossible by assumption.

Answer 2

If $A$ and $B$ are densely-defined linear operators with $A\preceq B$ then $B^*\preceq A^*$, where $\prec$, $\preceq$ indicate graph inclusions. The graph of $A$ is included in the graph of $B$ iff $A$ is a restriction of $B$. The adjoint of an operator $A$ is not defined if $A$ is not densely-defined, because it wouldn't be unique.

You were given the strict inclusion $A\prec B$. Because $A$ is assumed to be densely-defined, then the adjoints of $A$ and $B$ exist, and $$ A\prec B \implies B^*\preceq A^*. $$ An operator $A$ is symmetric iff $A \preceq A^*$.

In part (a) you were given $B^*=B$. Hence, $$ A \prec B = B^*\preceq A^* $$ So $A$ is symmetric, but it cannot be selfadjoint.

In part (b) you were given $A^*=A$. Hence, $$ B^* \preceq A^* = A \prec B. $$ So $B$ is not symmetric because $B\preceq B^*$ cannot hold.