Let $f \in C[a,b]$ with $$\int_{a}^{b}f(x)x^{n}dx=0$$ for all $n\in \mathbb{N}$. Prove $f=0$.
I got the intuition to prove this with induction over $n \in \mathbb{N}$, for $n=0$, I have $\int_{a}^{b}f(x)dx=0$. So how I got that $f=0$? Also, how I end up the proof? Any help will be appreciated. Thanks
Use the fact that polynomials are dense in $C[a,b]$ with the supremum norm (see the Stone–Weierstrass theorem).
Due to the linearity of integration, your assumption implies that
$$\int_a^b f(x)P(x) \mathrm{d}x = 0 \hspace{10px}(\star)$$
for any polynomial $P(x)$.
By the Stone-Weierstrass theorem, you can find a sequence of polynomials $\{P_n\}_{n=1}^{\infty}$ such that $\lim_{n\to\infty}P_n(x) = f(x)$ uniformly. Since the convergence is uniform, we can exchange limit and integration operators. Therefore, we have
$$\lim_{n\to\infty}\underbrace{\int_a^b f(x)P_n(x)\mathrm{d}x}_{\text{this is zero by }(\star)} = \int_a^bf(x)\lim_{n\to\infty}P_n(x)\mathrm{d}x= \int_a^b f(x)^2 \mathrm{d}x = 0$$
Since $f^2(x) \geq 0$, we conclude that $f=0$ on $[a,b]$.
Based on the context set in the text of the question, I assume we can take
$0 \in \Bbb N; \tag 1$
I also take it that
$a \le b. \tag 2$
We have the following
Lemma: Suppose
$g_n \in C[a, b] \tag 3$
is a sequence such that
$g_n \to f \; \text{as} \; n \to \infty \tag 4$
then
$\displaystyle \lim_{n \to \infty} \int_a^b fg_n \; dx = \int_a^b f^2 \; dx. \tag 5$
Proof of Lemma:
Observe that
$\displaystyle \int_a^b f^2 \; dx - \int_a^b f g_n \; dx = \int_a^b (f^2 - fg_n) \; dx = \int_a^b f(f - g_n) \; dx; \tag 7$
thus
$\left \vert \displaystyle \int_a^b f^2 \; dx - \int_a^b f g_n \; dx \right \vert = \left \vert \displaystyle \int_a^b f(f - g_n) \; dx \right \vert$ $\le \displaystyle \int_a^b \vert f (f - g_n) \vert \; dx = \displaystyle \int_a^b \vert f \vert \vert f - g_n \vert \; dx \le \int_a^b \vert f \vert \Vert f - g_n \Vert \; dx = \Vert f - g_n \Vert \int_a^b \vert f \vert \; dx; \tag 8$
passing to the limit,
$\lim_{n \to \infty} \left \vert \displaystyle \int_a^b f^2 \; dx - \int_a^b f g_n \; dx \right \vert \le \lim_{n \to \infty} \Vert f - g_n \Vert \displaystyle \int_a^b \vert f \vert \; dx = 0; \tag 9$
thus (5) binds. End: Proof of Lemma.
With this lemma in hand we proceed with the observation that the function
$x:[a, b] \to \Bbb R \tag{10}$
separates points in $[a, b]$; that is, for any
$c_1, c_2 \in [a, b], \; c_1 \ne c_2, \tag{11}$
we have
$x(c_1) = c_1 \ne c_2 = x(c_2); \tag{12}$
it then follows from the Stone-Weierstrass theoremthat the sub-algebra of $C[a, b]$ generated by $x$ and the constant functions is dense; since this sub-algebra is the set of real polynomials in $x$, it follows that there exists a sequence of polynomials $p_n(x)$ such that
$p_n(x) \to f(x) \; \text{in} \; C[a, b]; \tag{13}$
that is,
$\displaystyle \lim_{n \to \infty} \Vert p_n(x) - f(x) \Vert = 0, \tag{14}$
and from the hypothesis that
$\displaystyle \int_a^b x^m f(x) \; dx = 0, \; \forall m \in \Bbb N, \tag{15}$
the linearity of the integral allows us to conclude that
$\displaystyle \int_a^b p_n(x) f(x) \; dx = 0, \forall n \in \Bbb N; \tag{16}$
now applying the lemma we find
$\left \vert \displaystyle \int_a^b f^2(x) \; dx \right \vert = \lim_{n \to \infty} \left \vert \displaystyle \int_a^b f^2 \; dx - \underbrace{\int_a^b p_n f \; dx}_{0} \right \vert \le \lim_{n \to \infty} \Vert f - p_n \Vert \displaystyle \int_a^b \vert f \vert \; dx = 0, \tag{17}$
which forces
$\displaystyle \int_a^b f^2(x) \; dx = 0; \tag{18}$
now since $f^2(x)$ is a non-negative continuous function it follows that
$f(x) = 0, \forall x \in [a, b]. \tag{19}$
$OE\Delta$.
