Perhaps you are confused because Stanley just previously defines $[n]=\{1,2,\dots,n\},$ and this statement about $[t^k]\chi(t)$ is broken out of the list of notations, so it looks like an application of the prior notation.
Stanley here is defining another notation, separate from the previous $[n].$ This notation also uses square brackets, so it might be confusing to see the two defined right next to each other.
From a analytic point of view, we might write:
$$[t^n]f(t)=\frac1{n!}f^{(n)}(0),$$ where $f^{(n)}$ is the $n$th derivative of $f.$
Now, technically, this only applies to analytic functions - functions $f$ whose Taylor series converges to $f$ in some open region around $0.$ But mathematicians also deal with abstract power series, where we don't care about convergence for particular values $x.$
For example, $$f(x)=\sum_{n=0}^{\infty} n^nx^n$$ is a power series that does not converge for any non-zero real $x,$ but it is still an abstract power series, and we still say $[x^k]f(x)=k^k.$
This is most useful if you can write your polynomial or power series in a closed form. For example:
$$[x^k]\frac1{(1-x)^{n+1}}=\binom{n+k}{n}$$
or $$[y^k]e^{ay}=\frac{a^k}{k!}$$