Actually, we've already seen a sheaf on projective space which satisfies some of the properties we're after! That is, we've seen a sheaf $\mathcal{E}$ on $\Bbb P^n_k$ and a morphism $t:H^n(\Bbb P^n_k,\mathcal{E})\to k$ which for some class of coherent sheaves $\mathcal{F}$ on $\Bbb P^n$ satisfies the properties of a dualizing sheaf, namely that $$\operatorname{Hom}(\mathcal{F},\mathcal{E})\times H^n(\Bbb P^n_k,\mathcal{F})\to H^n(\Bbb P^n_k,\mathcal{E})$$ followed by $t:H^n(\Bbb P^n_k,\mathcal{E})\to k$ gives an isomorphism $\operatorname{Hom}(\mathcal{F},\mathcal{E})\to H^n(\Bbb P^n_k,\mathcal{F})'$.
In the computation of the cohomology of sheaves of the form $\mathcal{O}(d)$ on projective space $\Bbb P^n_A$ (theorem III.5.1), we showed that the natural map $$H^0(\Bbb P^n_A,\mathcal{O}(d))\times H^n(\Bbb P^n_A,\mathcal{O}(-n-d-1))\to H^n(\Bbb P^n_A,\mathcal{O}(-n-1))\cong A$$ is a perfect pairing of finitely generated free $A$-modules. Reindexing slightly, we can rewrite this as $$H^0(\Bbb P^n_A,\mathcal{O}(-n-d-1))\times H^n(\Bbb P^n_k,\mathcal{O}(d))$$ and recognizing that $\operatorname{Hom}(\mathcal{O}(d),\mathcal{O}(-n-1))\cong \operatorname{Hom}(\mathcal{O},\mathcal{O}(-n-d-1))\cong H^0(\Bbb P^n_A,\mathcal{O}(-n-d-1))$, we've exactly found that $\mathcal{E}=\mathcal{O}(-n-1)$ and the isomorphism $H^n(\Bbb P^n_k,\mathcal{O}(-n-1))\cong k$ satisfy what we ask of a dualizing sheaf when $\mathcal{F}$ is among the sheaves of the form $\mathcal{O}(d)$. As $\mathcal{O}(-n-1)\cong\bigwedge^n\Omega_{\Bbb P^n_k/k}$ is the canonical sheaf of projective space, this would be a sign that we're on the right track if we're trying to invent this theory.
There's also good intuition coming from the manifold case: remember, Serre duality is supposed to be kind of like an analogue of Poincare duality. How does Poincare duality work? Cap with the fundamental class! What is the fundamental class? It's an element of the top-dimensional homology which tracks orientations. Orientations are described by an ordered basis of the tangent space at each point up to equivalence, which can be measured by top-dimensional differential forms. So poking around top-dimensional differential forms to try and invent a duality theory is a natural avenue for experimentation.
Finally, let me point out that I've never actually asked Serre, Hartshorne, or any of the other mathematicians who developed this theory about how they got to the idea or read accounts of their thought processes, so I can't be sure that what I've related here was actually a factor in how they got to these concepts. But I think this material is certainly at least suggestive of a direction to explore - once you have the motivation of "can I prove some sort of duality for varieties?" and some of this evidence in the form of calculations showing some preliminary success, it would be natural to investigate further.