What is $$\int_0^{\pi} \sin^4(x)\cos^6(x)\,dx $$
Putting $\sin(x) = t$, then $\cos(x) = \sqrt{1-t^2}$ and $\cos(x)\,dx = dt$
At $x = 0$, $\sin(x) = 0$, $\therefore\ t = 0$
At $x = \pi$, $\sin(x) = 0$, $\therefore\ t = 0$
Integral becomes
$$\int_0^0 t^4\cdot\sqrt[5]{1-t^2}\,dt = 0$$
Integral should not evaluate to zero
What is going on here, i cannot find any mistake....
The $π-$ periodicity of the integrand does not allow you to use your change of variables.But rather inserting the change of variables $u=π-x$ we get
$$\int_{0}^{\pi} \sin^4(x)\cos^6(x) dx =\int_{0}^{\pi/2} \sin^4(x)\cos^6(x) dx+ \int_{π/2}^{\pi} \sin^4(x)\cos^6(x) dx\\= 2\int_{0}^{\pi/2} \sin^4(x)\cos^6(x) dx=2\int_{0}^{\pi/2} \sin^{5-1}(x)\cos^{7-1}(x) dx \\=B(5/2,7/2) = \frac{\Gamma(5/2)\Gamma(7/2)}{\Gamma(6)} =\frac{3π}{256} $$
see here for the Beta and Gamma function and use the fact that $\Gamma(n+1)=n!$, $\Gamma(1/2)=\sqrt{π}$,
Others have already presented several approaches to find this integral, but let me offer a very short answer to your actual question:
What is going on here, i cannot find any mistake....
Your main mistake is here:
Putting $\sin(x)=t$ then $\color{red}{\cos(x)=\sqrt{1−t^2}}$.
The statement in red is not true. Since the integral ranges from $x=0$ to $x=\pi$, you have values of $x$ in quadrants I and II. But in quadrant II, i.e. for $x\in(\frac{\pi}{2},\pi)$, the values of $\cos(x)$ are negative. In other words, for $x\in(\frac{\pi}{2},\pi)$, we have $$\cos(x)=-\sqrt{1−t^2}\neq\sqrt{1−t^2},$$ invalidating your proposed substitution.
$$\sin^4x\cos^6x=\frac{1}{16}\cos^2x\sin^42x=\frac{1}{128}(1+\cos2x)(1-\cos4x)^2=$$ $$=\frac{1}{128}(1+\cos2x)\left(1-2\cos4x+\frac{1+\cos8x}{2}\right)=$$ $$=\frac{1}{256}(1+\cos2x)(3-4\cos4x+\cos8x),$$ which says that our integral it's $$\int\limits_0^{\pi}\frac{3}{256}dx=\frac{3\pi}{256}.$$
This is an answer both to the OP and to @GuyFsone, who does not understand my objection. And it's too long for a comment.
The original question is
What is going on here, I cannot find any mistake...
Let's have a look at a substitution in an integral. Wikipedia tells us that
$$\int_{\phi(a)}^{\phi(b)} f(x)\,dx = \int_a^b f(\phi(t))\phi'(t)\, dt$$
Where $\phi$ is a differentiable function defined on $[a,b]$. Actually, it needs only be defined on $]a,b[$ (you may then get an interval on an infinite interval as a result of the substitution).
Notice, the variable in your original integral is $x$, and for the substitution one has to write $x=\phi(t)$. Usually, one thinks of substitution backwards, and you would write $t=\psi(x)$, but this is only valid if you can invert the function $\psi$, to write the substitution $x=\phi(t)$ as needed. To invert a function, it has to be a bijection. Here you have chosen $t=\sin(x)$, with $x\in[0,\pi]$, and you can't invert the sine on $[0,\pi]$.
Notice that $\phi$, on the other hand, needs not be bijective: it's because you wrote $t=\psi(x)$ in the first place that you have to invert the function $\psi$ to get $\phi$.
Now, how could we do that properly, even with a periodic function? The cotangent comes to mind, as it's defined and bijective on $]0,\pi[$. The inverse is arccotangent, or $x=\mathrm{arccot}(t)$.
A note about definitions: the cotangent is $\pi$-periodic, thus has no inverse on $\Bbb R$. You have to pick an interval where it's bijective. Here, the obvious interval is $]0,\pi[$. If you want a plot of this arccotangent, have a look at this. The plot shown by Wolfram Alpha is another possibility, but it's not the one we want here. The former is the same, with a constant $\pi$ added for $t<0$. The derivative is thus the same where they are both defined. The former, that we will use here, has the advantage of having image $]0,\pi[$ as we want, and also of being differentiable on $]-\infty,+\infty[$.
Now, for $t\in]0,\pi[$, $\dfrac{\cos^2 \mathrm{arccot}(t)}{\sin^2 \mathrm{arccot}(t)}=t^2$, hence $1=(1+t^2)\sin^2\mathrm{arccot}(t)$ and
$$\sin^2\mathrm{arccot}(t)=\frac1{1+t^2}$$
Likewise,
$$\cos^2\mathrm{arccot}(t)=\frac{t^2}{1+t^2}$$
And of course, $\dfrac{\mathrm d}{\mathrm dt}\mathrm{arccot}(t)=\dfrac{-1}{1+t^2}$
The substitution is therefore:
$$\int_0^\pi \sin^4(x)\cos^6x\mathrm dx=-\int_{\infty}^{-\infty} \frac{t^6}{(1+t^2)^6}\mathrm dt=\int_{-\infty}^{\infty} \frac{t^6}{(1+t^2)^6}\mathrm dt$$
This integral isn't very easy. You first have to use partial fractions:
$$\frac{t^6}{(1+t^2)^6}=\frac{1}{(1+t^2)^3}-\frac{3}{(1+t^2)^4}+\frac{3}{(1+t^2)^5}-\frac{1}{(1+t^2)^6}$$
But the integrals of each terms get quite long (see this). It was merely an example to show how substitution can be made correctly here.
Another way to use substitution, still with $t=\sin(x)$, is to use it on an interval where the sine is bijective. If you integrate on $[0,\pi/2]$ and $[\pi/2,\pi]$ separately, it will work. See zipirovich's answer.
Your substitution can be made to work with the following procedure (that works for all integrals of $\sin^nx\cos^mx$ over an interval that is the union of several "full quadrants", i.e. integral multiples and translates of $[0,\pi/2]$.
Step 1. Use the identity $\sin^2x+\cos^2x=1$ to turn all the powers of sine (or cosine, makes no difference) into either the first or the zeroth power depending on the parity of $n$.
Step 2. Study the interval. Take advantage of the symmetries $\sin(\pi-x)=\sin x$, $\cos(-x)=\cos x$, $\sin(x+\pi)=-\sin x$, $\cos(x-\pi)=-\cos x$ et cetera to write everything over the interval $[0,\pi/2]$. Observe that many cancellations are possible here.
Step 3. If $n$ was odd (does not apply here) you are left with terms of the form $\sin x\cos^\ell x$, and can use the formula $$\int f'f^\ell=\frac1{\ell+1}f^{n+1}$$ to get the indefinite integral. If $n$ is even you have a linear combination of integrals like $$ \int_0^{\pi/2}\cos^\ell x\,dx. $$ At this point my students can refer to a table of definite integrals (provided for them in the exams), and look up the value. That table formula was derived during the course. They are welcome to memorize it, but that is optional.
Let's see.
The first step would be to rewrite the integral as $$ \int_0^\pi\sin^4x\cos^6x\,dx=\int_0^\pi(\cos^6x-2\cos^8x+\cos^{10}x)\,dx. $$
Here all the terms are even powers of cosine (no sign changes), and we know at a glance that the integrals over $[0,\pi/2]$ and $[\pi/2,\pi]$ are equal (two full quarter waves). Therefore the integral equals $$2\int_0^{\pi/2}(\cos^6x-2\cos^8x+\cos^{10}x)\,dx. $$ As promised, here you can use your substitution. Actually, you could have skipped Step 1 here, and just observe at a glance that your integral is $$2\int_0^{\pi/2}\sin^4x\cos^6x\,dx.$$
Then we look up those definite integrals $\int_0^{\pi/2}\cos^{2k}x\,dx$ from the table and we are done.
Concluding remarks:
$$\int_{0}^{\pi} \sin^4 x\cos^6 x dx =\int_{0}^{\pi} (\frac{1-\cos 2x}{2})^2 (\frac{1+\cos 2x}{2})^3 dx =$$ $$=\frac{1}{32}\int_{0}^{\pi} (1-\cos 2x)^2 (1+\cos 2x)^3 dx =$$ $$=\frac{1}{32}\int_{0}^{\pi} (1-2\cos 2x+\cos^2 2x) (1+3\cos 2x+3\cos^2 2x+\cos^3 2x) dx =$$ $$=\frac{1}{32}\int_{0}^{\pi} (1+\cos 2x-2\cos^2 2x-2\cos^3 2x+\cos^4 2x+\cos^5 2x) dx $$
$$\int_{0}^{\pi} (1) dx =\pi$$ $$\int_{0}^{\pi} (\cos 2x) dx =0$$ $$\int_{0}^{\pi} (\cos^2 2x) dx =\frac{\pi}{2}$$ $$\int_{0}^{\pi} (\cos^3 2x) dx =0$$ $$\int_{0}^{\pi} (\cos^4 2x) dx =\frac{3\pi}{8}$$ $$\int_{0}^{\pi} (\cos^5 2x) dx =0$$ $$=\frac{1}{32}(\pi -\pi+\frac{3\pi}{8})=\frac{3\pi}{256}$$
