For questions such as this one, I like to distinguish between the (Euclidean) inner product of two vectors $\mathbf a$ and $\mathbf b$, defined by $\langle\mathbf a,\mathbf b\rangle = \lVert\mathbf a\rVert \lVert\mathbf b\rVert\cos\phi$, where $\phi$ is the angle between the vectors, and the dot product of a pair of coordinate tuples: $[\mathbf a]_{\mathcal B}\cdot[\mathbf b]_{\mathcal B}=([\mathbf a]_{\mathcal B})^T[\mathbf b]_{\mathcal B}=\sum_i a_i b_i$. Here I’m also distinguishing between a vector $\mathbf v$ and its representation as a coordinate tuple relative to some basis $\mathcal B$, denoted $[\mathbf v]_{\mathcal B}$. This can get a bit confusing when working in $\mathbb R^n$ because the vectors are themselves tuples of real numbers.
In the standard basis $\mathcal E$, the two are equal, but in other bases the formula for the inner product in terms of coordinates might be something else. We can easily find this formula if the coordinate systems are related by a linear transformation: if we have $[\mathbf v]_{\mathcal E}=M[\mathbf v]_{\mathcal B}$ for some matrix $M$, then $$\langle\mathbf a,\mathbf b\rangle = (M[\mathbf a]_{\mathcal B})^T(M[\mathbf b]_{\mathcal B}) = ([\mathbf a]_{\mathcal B})^T(M^TM)[\mathbf b]_{\mathcal B}.$$ If $M$ is orthogonal, then $M^TM=I$ and the formula reduces to the dot product. So, in any orthonormal basis, the Euclidean inner product of two vectors is equal to the dot product of their coordinates.
When talking about the operators $\operatorname{grad}$ and friends, we’re dealing with vector fields, and it’s important in this context also to distinguish between points and vectors. The basic idea is that attached to each point is a vector space that consists of all possible tangent vectors to curves through that point—the tangent space at that point. A vector field is a function that assigns to each point an element of its tangent space. It doesn’t really make sense to talk about the inner product of two points, nor of the inner product of vectors that belong to different tangent spaces. If the ambient space is $\mathbb R^n$, then every tangent space is a copy of $\mathbb R^n$, which is why we can usually identify points with vectors and blithely move tangent vectors around from point to point.
The basis vectors $\hat{\mathbf r}$, $\hat{\mathbf\theta}$ and $\hat{\mathbf z}$ live in these tangent spaces, as do the standard basis vectors $\hat{\mathbf i}$, $\hat{\mathbf j}$ and $\hat{\mathbf k}$, and the transformation between them isn’t the cylindrical-to-Cartesian point coordinate transformation that you used. These cylindrical basis vectors for the tangent space at a point are unit normals to the surfaces $r=\text{const}$, $\theta=\text{const}$ and $z=\text{const}$ that pass through the point. Clearly, these vectors vary from one point to another. It should be easy to see that these unit vectors are pairwise orthogonal, so in cylindrical coordinates the inner product of two vectors is the dot product of the coordinates, just as it is in the standard basis.
You can verify this directly. With a little bit of work we can find that $$\begin{bmatrix}\hat{\mathbf r}\\\hat{\mathbf\theta}\\\hat{\mathbf z}\end{bmatrix} = \begin{bmatrix}\cos\theta&\sin\theta&0\\-\sin\theta&\cos\theta&0\\0&0&1\end{bmatrix} \begin{bmatrix}\hat{\mathbf i}\\\hat{\mathbf j}\\\hat{\mathbf k}\end{bmatrix}$$ so that $$\hat{\mathbf A} = A_r\hat{\mathbf r}+A_\theta\hat{\mathbf\theta}+A_z\hat{\mathbf z} = (A_r\cos\theta-A_\theta\sin\theta)\,\hat{\mathbf i} + (A_r\sin\theta+A_\theta\cos\theta)\,\hat{\mathbf j}+A_z\,\hat{\mathbf k}$$ and similarly for $\hat{\mathbf B}$. Note that the $\theta$ here is the cylindrical coordinate of the point at which our tangent space lives, not of the vector we’re transforming. If you multiply out their dot product, you’ll find that the cross terms in $\cos\theta\sin\theta$ cancel and the other terms that involve sines and cosines simplify to unity, leaving $\langle\hat{\mathbf A},\hat{\mathbf B}\rangle = A_rB_r+A_\theta B_\theta+A_zB_z$.
Of course, it’s also possible to view the cylindrical-to-Cartesian point coordinate transformation as a map $\varphi$ from one copy of $\mathbb R^3$ to another. Considering the inner product as a function $f:\mathbb R^3\times\mathbb R^3\to\mathbb R$, we can always define $\alpha:(\mathbf u,\mathbf v)\mapsto(\varphi(\mathbf u),\varphi(\mathbf v))$ and compute the pullback $\alpha^*f$ of the scalar product to this other copy of $\mathbb R^3$. This is basically what you did, but that’s not what’s going on when you’re dealing with vector fields on $\mathbb R^3$.
