The variable is $\mathbf{x}=\bigl[x_1,\ldots,x_n\bigr]^T$ and $a_k$ is given number. The constraint is the following: $$\dfrac{|a_k\cdot x_k|^2}{\sum_{j\neq k}^n|a_j\cdot x_j|^2+1}\geqslant \alpha.$$
After some modifications, we can easily get the constraint in the form:
$$|a_k\cdot x_k|^2-\sum_{j\neq k}^n\alpha\cdot |a_j\cdot x_j|^2-\alpha\geqslant 0.$$
So, how can we prove that this constraint is convex or not.
It is not convex for any $\alpha>0$. Let $x$ be any vector that satisfies the inequality, and consider another vector $\bar{x}=-x$. The absolute values ensure that $\bar{x}$ satisfies the inequality as well. However, the midpoint between these two vectors is simply $\tfrac{1}{2}(x+\bar{x})=0$, and the origin does not satisfy the inequality. Thus the constraint does not describe a convex set.
If $\alpha\leq 0$, it is satisfied for any $x$, so it is trivially convex.
