I only have problems determining the values of $\alpha$ and $\beta$, so I will only show the solution used to derive their values:
So, I know how to get the inequalities at ★ and † but I don't know how use them to deduce $\alpha$ and $\beta$.
All I know is $\alpha \geq -1$ and I got this from ★ because if $\alpha=-1$ then ★ is 0 and if $\alpha > -1$, ★ is greater than 0.
Please show me the next steps (I don't understand the reasoning the solution provided).
The reasoning of the solution is the following.
Firstly, the inequalities $(\star)$ and $(\dagger)$ represent the two principle minors of the Hessian matrix of $f$. It is known that $f$ is convex iff it's Hessian matrix is positive semidefinite and the Hessian matrix is positive semidefinite if it's principle minors are positive. So, that explains the derivation of the two inequalities (which you said in your post that you have already understood).
Secondly, the two inequalities are solved simultaneously, by discriminating cases for $α$. We note that the roots of the polynomial in $(\star)$ $$α(α+1)=0$$ are $α=-1$ and $α=0$. Thus the term $α(α+1)$ is $\ge0$ if $α \notin (-1,0)$. So there are two basic cases (1.) $α\ge 0$ and (2.) $α\le -1$ in which the $(\star)$ inequality is satisfied.
First case $α \ge 0$. Then $(\star)$ is satisfied and for $(\dagger)$ we have that $$\underbrace{(α+1)}_{\text{positive}}(β-1)(1-α-β)$$ Therefore in order for the whole term to be positive (so that $(\dagger)$ is satisfied) we need that $$(β-1)(1-α-β)\ge 0$$ The roots of this polynomial are $β_1=1$ and $β_2=1-α$, where the second, due to the selection of $α$, is less or equal than $1$ ($α\ge 0 \implies 1-a \le 1$). Obviously this polynomial (in $β$) is nonegative when $$β \in [1, 1-α]$$ (just let $β \to \pm \infty$ to see what happens and then take in account that in the roots the sign of the terms changes) therefore there is one admissible region for $β$ which is $$ 1 \le β \le 1-α$$ Combining the above results we have determined one region ("Region $A$" in the graph) where $f(x,y)$ is convex, in particular $$\tag{Region A} α\ge 0 \qquad \text{ and } \qquad 1 \le β \le 1-α$$ (Note that the name of the Region is selected in order to fit with the given graph). Similarly we will proceed with the case $α \le -1$ to determine the remaining regions where $f$ is convex.
Second case $α \le -1$. Then $(\star)$ is satisfied and for $(\dagger)$ we have that $$\underbrace{(α+1)}_{\text{negative}}(β-1)(1-α-β)$$ Therefore in order for the whole term to be positive (so that $(\dagger)$ is satisfied) we need that $$(β-1)(1-α-β)\le 0$$ The roots of this polynomial are $β_1=1$ and $β_2=1-α$, where the second, due to the selection of $α$, is equal or bigger than $2$ ($α\le-1 \implies 1-a \ge 2$). Obviously this polynomial (in $β$) is negative when $$β \notin (1, 1-α)$$ therefore there are two admissible regions for $β$ which are $$β\le 1 \qquad $$ and $$β\ge 1-α$$ Combining the above results we have determined two regions ("Regions $B$ and $C$" in the graph)where $f(x,y)$ is convex, in particular $$\tag{Region C} α\le-1 \qquad \text{ and } \qquad β \le 1 \phantom{-α1}$$ and $$\tag{Region B} α\le-1 \qquad \text{ and } \qquad β \ge 1-α$$ which completes the explanation of the reasoning in the above solution.
To sum up, the author of the solution, initially determined the principal minors of the Hessian matrix of $f$ and thus came up with two inequalities $(\star)$ and $(\dagger)$ that depend on the values of the unknown parameters $α$ and $β$. Afterwards he started with the simpler equation of the two (that is $(\star)$ which depends only on $α$ and can thus be easily solved) and discriminated cases for the one parameter $α$. For each case, he plugged the value(s) of $α$ in the second inequality $(\dagger)$ and came up with the respective values of the other parameter $β$. The admissible results formed in total $3$ Regions (depending of course on the values of $α$ and $β$) which are depicted in the given graph with the names $A$, $B$ and $C$ as it was required.
Necessary conditions:
if $y=1$ is fixed, $\alpha+1>1$
if $x=1$ is fixed, $\beta-1>1$
Sufficient conditions :
if $\alpha>0$ and $\beta>-2$, $f$ is the product of 2 convex functions
