Is it correct to limit the objective function of an optimization problem sometimes? I heard we shouldn't limit the objective function at all. In other words, a namely problem like this, is possible?
$Min{ \space\space\space } Z $
$Z^2+x+y>3$
$Z-x-y<10$
$Z<5$
x,y,Z are the decision variables of the problem.
The problem you described makes perfect sense. The last three inequalities would be regarded as "constraints", and you are therefore dealing with a "constrained optimization" problem.
In your problem, each constraint inequality describes some region of 3D space. So, for example, the middle one is the set of points $(x,y,z)$ that is bounded by the plane $z-x-y=5$. The set of points that satisfy all three constraints is called the feasible region for the problem. It is the intersection of the three sets defined by the individual constraint inequalities. Lets call this set $S$. Then your problem says "find the point in $S$ that has the smaller $z$-coordinate. That's a perfectly reasonable problem.
An optimization problem doesn't really make sense if there are no feasible points (i.e. if the set $S$ is empty). I haven't checked thoroughly, but I don't think this is the case in your particular problem.
In general, an inequality-constrained minimization problem takes the form: $$ \min_{\mathbf{x} \in S}f(\mathbf{x}) $$ where $$ S = \{\mathbf{x} \in \mathbb{R}^m : g_1(\mathbf{x}), \ldots,g_n(\mathbf{x}) \le 0\} $$ It's OK if one of the constraint functions $g_i$ is the same as $f$.
Where did you hear that "we shouldn't limit the objective function"?
