One way to visualize the minimal polynomial (and characteristic polynomial) is to first ask why they are annihilating polynomials. This intuition is probably easiest for diagonalizable operators so let us start there.
If $A: V\rightarrow V$ is diagonalizable, then we can write
$$V=\bigoplus_{i=1}^k E_{\lambda_i}$$
where each $E_{\lambda_i}$ is the eigenspace of $A$ corresponding to eigenvalue $\lambda$. To annihilate $A$, it is therefore sufficient to annihilate $A$ on each of the eigenspaces and the annihilator for each $E_{\lambda_i}$ is just $A-\lambda I$. Now as far as invariant subspaces go, eigenspaces are pretty much as simple as possible. This means that a single iteration of $(x-\lambda_i)$ on $A$ is enough to eliminate the space $E_{\lambda_i}$. Therefore the minimal polynomial is
$$p(x) = (x-\lambda_1)\cdots (x-\lambda_k)$$
and intuitively this is why the minimal polynomial of a diagonalizable operator splits into linear terms.
However, not all operators are diagonalizable and so we need to talk about generalized eigenspaces instead. If you are familiar with the Jordan form, then generalized eigenspaces need no introduction. If you are not (which I think might be the case since Jordan forms are chapter 7 in Hoffman) then we can summarize as follows: Just as a diagonalizable operator "decomposes" into eigenspaces, any operator (diagonalizable or not) decomposes into what are called generalized eigenspaces.
Normal eigenspaces $E_{\lambda_i}$ are defined as the set of vectors $\mathbf{v}\in V$ for which
$$(A-\lambda_i I)\mathbf{v} = \mathbf{0}$$
generalized eigenspaces $W_{\lambda_i}$ are defined as the set of vectors $\mathbf{v} \in V$ for which there exists some $m\in \mathbb{N}$ such that
$$(A-\lambda_i I)^m \mathbf{v} = \mathbf{0}$$
And we then have
$$V = \bigoplus_{i=1}^kW_{\lambda_i}$$
The representation of the operator $A$ corresponding to the above decomposition is then the Jordan Normal Form of the matrix.
With each generalized eigenspace, we can make a conceptual distinction. We separate the space into sets
$$S_j = \left\{\mathbf{v}\in V \big|\ j\ \text{is the smallest integer such that } (A-\lambda_i)^j\mathbf{v} = \mathbf{0}\right\}$$
for example, $S_1 \cup \{\mathbf{0}\} = E_{\lambda_i}$. In this notation, we can view the action of the operator $A-\lambda I$ as shifting the sets
$$S_m \rightarrow S_{m-1} \rightarrow \cdots \rightarrow S_1 = E_{\lambda_i} \rightarrow S_0 = \{\mathbf{0}\}$$
In this way, the exponent of $e_i$ each factor $(x-\lambda_i)^{e_i}$ in the minimal polynomial can be seen as the "depth" of the generalized eigenspace $W_{\lambda_i}$.
I am not sure if the above explanation is satisfying, but this gives us a very rough geometrical intuition of the minimal polynomial. The minimal polynomial can be seen as a sequence of steps taken to eliminate the vectors of $V$ which reflects the structure of $A$. We decompose $V$ into invariant generalized eigenspaces $V = \bigoplus_{i=1}^kW_{\lambda_i}$. For each generalized eigenspace, we apply the operator $(A-\lambda_i)$ enough times so that the space is eliminated. Each application of the operator corresponds to a shift of the form $S_{i+1} \rightarrow S_i$ in the space. Doing this for all the generalized eigenspaces then eliminates the entire the vector space.
