When you specify a vector space, you always specify two things: a set and a field. When we take our set to be the complex numbers $ \mathbf{C} $, two possible choices for our field are $ \mathbf{R} $ and $ \mathbf{C} $. So we may think of $ \mathbf{C} $ as a vector space with either real scalars or complex scalars. The choice is important when defining an inner product, since an inner product always maps into whichever field you took your vector space over.
If we choose complex scalars, then one inner product on $ \mathbf{C} $ is $ \langle z, w \rangle = z \overline{w} $. As you rightly point out, $ \langle 2 + i, -1 + 2i \rangle \neq 0 $. So $ 2 + i $ and $ -1 + 2i$ are not orthogonal in this inner product space. That shouldn't be surprising, because as vectors they are linearly dependent; we have $ 2 + i = -i(-1 + 2i) $. We really needed complex scalars for this to be true; no real number $ \alpha $ satisfies $ 2 + i = \alpha (-1 + 2i) $.
If we choose real scalars, then the map $ \langle z, w \rangle = z \overline{w} $ is no longer an inner product on our vector space, since in general $ z \overline{w} $ is not a real number. You can still define an inner product
$$ \langle z, w \rangle = \text{Re}(z) \text{Re}(w) + \text{Im}(z) \text{Im}(w). $$
This is just the dot product on $ \mathbf{R}^2 $ in disguise. With this inner product, you can check that $ \langle 2 + i, -1 + 2i \rangle = 0 $, i.e. $ 2 + i $ and $ -1 + 2i $ are orthogonal.
So even though the set we were working with was the same in both cases, we defined two different vector spaces and two different inner product spaces - the notion of orthogonality in each of them is not the same.