The answer is no, you cannot treat it as a constant.
The tricky part in your problem is finding $\frac{dC'}{dy}(x)$. You can find it using the Chain rule.
$$\frac{dC'}{dy}=\frac{dC'}{dx} \frac{dx}{dy} = \frac{C''}{2x}$$ since $\frac{dy}{dx} = 2x \implies \frac{dx}{dy}=\frac{1}{2x}$. So the final answer of your task would be
$$\frac{dp}{dy}=\frac{C''(x)y}{2x(y-1)}-\frac{C'(x)}{(y-1)^2}$$
BONUS: Consider the following simpler example to make things clearer why you cannot treat it as a constant.
Suppose we have a variable $y=x^2$ and the expression $p = a(x) + y $ where $a(x)=x^2$.
If we treat $a(x)$ as constant we would get $\frac{dp}{dy} = 1$.
But clearly, our expression can be rewritten as follows $p = a(x) + y = x^2 + y = 2y$. So the derivative is now $\frac{dp}{dy} = 2$. The second approach is certainly correct because we calculated the derivative directly, without any assumptions such as "$a(x)$ is constant".
Conclusion: You cannot treat a function as a constant as long as your function is dependent on a variable over which you are derivating, even if that dependency is given implicitly (in both examples by $y=x^2$).