If I understand your question correctly (see my comment above) something like this can only be true if $Y$ is codimension $1$ in $X$. This is because for a general polynomial $f$, the vanishing set of $f$ in $X$ will be codimension $1$ in $X$. Call this $Z$. Then if $\dim X > 1$, we can take $Y$ to be an arbitrary number of finite points in $Z$. Then $Y$ can have arbitrarily large degree and $f$ vanishes on $Y$ but it doesn't vanish on all of $X$. Similarly, you need $X$ irreducible otherwise $f$ might just vanish on a whole irreducible component but not all of $X$.
Now, let $Y \subset X$ be codimension $1$ in $X$ irreducible and let $f$ be a polynomial of degree $h$ such that $f$ vanishes on $Y$. Let $H$ be the hypersurface cut out by $f$ and suppose $X \not \subset H$. Letting the $Z_i$ for $i = 1$ to $r$ be the irreducible components of $H \cap X$ (which is codimension $1$ by assumption), then $Y \subseteq H \cap X$ and $Y$ is codimension $1$ so $Y$ is a union of some of the $Z_i$, say $Y = Z_1 \cup \ldots \cup Z_s$. We can ignore any lower dimensional components of $Y$ since they don't change the degree. We have
$$
\deg Y = \sum_{j = 1}^s \deg Z_j
$$
(see this question). By generalized Bezout's theorem,
$$
\sum_{j = 1}^ ri(X,H; Z_j)\deg Z_j = h(\deg X).
$$
where $i(X,H;Z_j)$ is the intersection multiplicity of $Z_j$ in $X \cap H$. See Hartshorne Theorem $I.7.7$.
Suppose $\deg Y > h(\deg X)$. Then
$$
\sum_{j = 1}^ ri(X,H; Z_j)\deg Z_j \geq \sum_{j = 1}^s i(X,H; Z_j)\deg Z_j \geq
$$
$$
\sum_{j = 1}^s \deg Z_j = \deg Y > h (\deg X).
$$
This contradicts Bezout's theorem. Therefore, if $\deg Y > h(\deg X)$ and $f$ vanishes on $Y$, then $f$ must vanish on all of $X$.
This is a higher dimensional generalization of the following fact: suppose we have $f$ and $g$ of degree $d$ and $h$ defining irreducible curves in $\mathbb{P}^2$ that intersect properly. Then the number of intersection points up to multiplicity is $dh$. In particular, if $C$ is the curve defined by $g$ and $f$ vanishes at more than $dh$ points on $C$, then $f$ must vanish on all of $C$.