Why $f^n x = a^nx + b\frac{a^n-1}{a-1}$ when known :
$f(x)=ax + b$
$f^n(x) = f(f^{n-1}x)$
$f^1(x)=fx$
$f(x) = f(f^{n-1}x)$ means $f^2(x)=f(f(x)) = a(ax + b) + b = a^2x + ab + b$
$f^3(x)=f(f^2(x)) = a(a(ax + b) + b) + b = a^3x + a^2 b + ab + b$.
Why $f^n x = a^nx + b\frac{a^n-1}{a-1}$ when known :
$f(x)=ax + b$
$f^n(x) = f(f^{n-1}x)$
$f^1(x)=fx$
$f(x) = f(f^{n-1}x)$ means $f^2(x)=f(f(x)) = a(ax + b) + b = a^2x + ab + b$
$f^3(x)=f(f^2(x)) = a(a(ax + b) + b) + b = a^3x + a^2 b + ab + b$.
Note that this only holds if $a \neq 1$ (think about what the right formula would be if $a=1$).
You can prove this by induction on $n$. You have already checked the first several cases, but the $n=1$ case $f^1(x)=f(x)=ax+b$ is all you need for the base case.
Now suppose $f^{n-1}(x)$ follows the given formula, i.e., $f^{n-1}(x)=a^{n-1}x + b\frac{a^{n-1}-1}{a-1}$. Then, using the formula $f^n(x)=f(f^{n-1}(x))$, we have: $$f^n(x) = f\left(a^{n-1}x+ b\frac{a^{n-1}-1}{a-1} \right)= a\left(a^{n-1}x+ b\frac{a^{n-1}-1}{a-1} \right)+b. $$ Can you simplify this to show it is equal to $a^nx+b\frac{a^n-1}{a-1}$?
Something that might or might not be useful to you is to recall that $1+a+a^2+\cdots+a^{n-1}=\frac{a^n-1}{a-1}$ if $a \neq 1$.
$f^n(x)=a^n.x+b.(\sum_{i=0}^{n-1} a^i)$
By induction:
$n=1:f^1(x)=a^1.x+b.a^0=a.x+b$
$n=k\Rightarrow n=k+1:f^{k+1}(x)=f(f^k(x))=a.[a^k.x+b.(\sum_{i=0}^{k-1} a^i)]+b=a^{k+1}.x+b.(\sum_{i=0}^{k-1} a^{i+1})+b.a^0=a^{k+1}.x+b.(\sum_{i=-1}^{k-1} a^{i+1})=a^{k+1}.x+b.(\sum_{i=0}^{k} a^{i})$.
$\sum_{i=0}^{n-1} a^i=1+a+a^2+...+a^{n-1}=\frac{a^n-1}{a-1}$ ($a\neq1$)
By induction:
$n=1:1=\frac{a-1}{a-1}=\frac{a^1-1}{a-1}$
$n=k\Rightarrow n=k+1:1+a+...+a^{k-1}+a^k=\frac{a^{k}-1}{a-1}+a^k=\frac{a^k-1+a^{k+1}-a^k}{a-1}=\frac{a^{k+1}-1}{a-1}$
Therefore, we have that $f^n(x)=a^n.x+b.(\frac{a^n-1}{a-1})$