This question is quite old, but some non-trivial things can be said. The case of K3 surfaces has already been treated, so let me restrict to the case of Calabi-Yau threefolds. By the Calabi-Bogomolov-Beauville decomposition theorem, a compact Kähler manifold with holomorphically torsion canonical bundle admits a finite étale cover that is the product of a complex torus, a product of simply connected compact Kähler manifolds of complex dimension at least 3, and hyperkähler manifolds (which have complex dimension divisible by 4). Hence, if $X$ is a compact Kähler threefold with holomorphically torsion canonical bundle, there is a finite étale cover of $X$ by a three-dimensional complex torus or a compact simply connected Kähler manifold with holomorphically trivial canonical bundle. Moreover, the algebra of holomorphic forms is non-zero only in the extreme degrees (i.e., $0$ and $3$). Hence, $h^{p,0} =0$ for $p=1$ and $p=2$. I will understand this to mean Calabi-Yau, here.
For a Calabi-Yau threefold, the intersection form allows us to view the second Chern class $c_2$ as an integral linear form on the closure of the Kähler cone. Miyaoka (Math. Ann. 1998) showed that this integral linear form $c_2 : \overline{\mathscr{K}} \to \mathbf{Z}$ is nonnnegative. Hence, there are three cases: If $c_2 \equiv 0$, then $X$ admits a finite étale cover by a complex torus. If $c_2$ is not strictly positive, but not identically zero, then Wilson (Contemp Math 162) showed that the Picard number $\rho \leq 5$. If $\rho=4$ or $\rho=5$, then the cubic hypersurface $W \subset \mathbf{RP}^{\rho-1}$ (associated to the cubic cone $W^{\ast} : = \{ D : D^3 \equiv 0 \}$) decomposes as $W = \Lambda \cup Q$, where $\Lambda$ is a hyperplane and $Q$ is a quadric of rank $\rho-1 \leq r \leq 4$. If $Q$ is a quadric cone, the vertex $p \not \in \Lambda$. If $\Lambda \neq \{ c_2 =0 \}$, then $X$ is an elliptic fiber space. If $\Lambda$ is the hyperplane and $Q$ is a quadric cone, then $X$ is a K3 fiber space over $\mathbf{P}^1$.
If $c_2>0$, then Wilson (Contemp Math 162) showed that the automorphism group of $X$ is finite. Oguiso and Peternell (CAG, 1998) showed that Calabi--Yau threefolds with positive second Chern class admit only finitely many proper algebraic fiber space structures. To my understanding, their results do not give any indication as to whether this finite number is non-zero.