You may assume $M$ is a Riemannian manifold with some metric $g_0$. First, observe that $M$ admits a new Riemannian metric $g$ that is invariant under the group action: for example define
$$
g=\sum_{\tau\in G} \tau^*g_0;
$$
this can be defined since $G$ is a finite group.
In general, for any finitely many elements $\tau_1, \tau_2, ..., \tau_n$ assume $p$ is a fixed point of them, i.e. $\tau_j p=p$ for $j=1, ..., n$. Note that since $g$ is an invariant metric, the differential $d\tau_j$ is an orthogonal linear transformation on the tangent space $T_p$.
If there is no tangent vector $X$ at $p$ that is a common eigenvector of all $d\tau_j$ with eigenvalue $1$, then $p$ is an isolated fixed point: in fact, $\tau_j$ maps any geodesic from $p$ to a geodesic from $p$, so if a tangent vector $X$ is not an eigenvector with eigenvalue $1$ for some $d\tau_j$, then $\tau_j$ moves the geodesic $\exp_p(tX)$ (here $\exp_p$ is the exponential map from $p$) to a different geodesic, so there is no fixed point on $\exp_p(tX)$ for $0<t<\delta$.
Now assume the common eigenvectors of all $d\tau_j$ at $p$ with eigenvector $1$ form a linear subspace $V$ in $T_p$. Then as above we see the manifold $\exp_p V$ are all fixed points; on the other hand, any other point $q$ near $p$ can be written as $\exp_ptX$ so that $X\notin V$ and $t$ small, as above we see this point is moved to some other point by some $\tau_j$. So near $p$ the fixed point set is exactly $\exp_tV$.