Since the statement is symmetric in $x$ and $y$, it suffices to show: for fixed $y>0$, the quantity $x^x+y^y-x^y-y^x$ is nonnegative for $x\ge y$. And since this quantity vanishes at $x=y$, it suffices to show that it is increasing for $x\ge y$. But its derivative (with respect to $x$) is
$$
x^x \log x + x^x - yx^{y-1} - y^x\log y = (x^x \log x - y^x \log y) + (x^x - yx^{y-1});
$$
it now suffices to show that both quantities in parentheses are positive for $x\ge y>0$.
But for fixed $x>0$, the functions $t^x\log t$ and $tx^{t-1}$ are both increasing functions of the positive variable $t$: the derivative of the first function is $t^{x-1}+t^x\log^2t$ which is clearly positive; while the derivative of the second function is $x^{t-1}+tx^{t-1}\log t$, which is positive because $t\log t \ge -e^{-1} > -1$ for all $t$. In particular, their values at $t=x$ exceed their values at $t=y$, which confirms that both quantities are positive.
EDIT: Somehow I got both these last derivatives wrong, as LeGrandDODOM pointed out. The actual derivatives of $t^x\log t$ and $tx^{t-1}$ are, respectively, $t^{x-1}(x\log t+1)$ and $x^{t-1}(t\log x+1)$, neither of which is forced to be positive. I'm making this answer community wiki in case others can see how to rescue the idea.