Yes -- using a vector to write the cross product is just a 3D convention to remind you it has 3 independent components. It's more naturally represented as a bivector or as a rank-2 tensor, but there's a duality between these and vectors in 3D.
Scale invariance/tensors in physics
The formal way of expressing your concern -- that the cross product has different "units" from other vectors in your space -- is that the cross product doesn't behave like a vector under scaling. Under scaling, a vector is supposed to scale as $v\to\lambda v$, because lengths scale that way, and areas scale as $t\to\lambda^2t$. The cross product scales as an area, so it makes a sucky vector.
This is sort of related to the definition of tensors in physics (specifically in relativity) -- a tensor is an object that transforms as a tensor under specific transformations, specifically Lorentz transformations (skews in the t-x/t-y/t-z plane and rotations in the other three, x-y/y-z/z-x). So scalars are invariant under Lorentz transformations, vectors transform as $\Lambda_\mu^{\bar\mu}v^\mu$, rank-2 tensors transform as $\Lambda_\mu^{\bar\mu}\Lambda_\nu^{\bar\nu}t^{\mu\nu}$, etc. So it's not enough to have the right number of components -- because this can be gamed with symmetries, like it is for the cross product (which has nine components, but only three independent one) -- you need to transform in the right way.
It's a bit more complicated with scaling, because different scalars transform differently (lengths scale like vectors, areas scale like rank-2 tensors), but the idea is the same -- the cross product is not a vector in the physics sense, although it is in the math sense. But nobody in math cares about the cross product anyway.