Any odd number squared is of the form $4k+1$, so that $p^2 - 1 = 4k$ for some integer $k$. If $p^2 - 1 = 2m$ then of course $m$ would be even (because $2m$ is divisible by 4).
Take $p = 5, k = 3, n = 1$. Then $16 < 25$ and $16$ is not prime.
EDIT: $p = 17, n=144, k=1$. So $161 < 289$ and $161 = 7\cdot23$ is not prime. I suspect that the counterexamples come very easily if we consider Fermat primes, though I can't prove this right now.
EDIT 2: If we consider $p = 2^k + 1$, then $p^2 = 2^{2k} + 2\cdot 2^k + 1$ so $n = 2^{2k-1} + 2^k$. Then, letting $k = 1$, we have $n + p = 2^k(2^{k-1} + 1) + 2^k + 1 = (2^k + 2)(2^{k-1} + 1) - 1 = 2(2^{k-1} + 1)^2 - 1$. My guess is that many numbers of this form are composite. Maybe if we set up some sort of Pell equation $2a^2 - 1 = b^2$ some good things will happen?
Lastly, if $p = 11$, then $m = 61$, and if $k = 4$ we have $105$ divisible by $5$ but $105 < 121$