Here's an answer to your first question. It's also rather topical right now:
We work over $\mathbb{R}$ throughout.
A Normed Division Algebra is a "number system" which generalizes $\mathbb{C}$ in a similar way that $\mathbb{C}$ generalizes $\mathbb{R}$. More precisely:
A division algebra is a a vector space over $\mathbb{R}$ (so it has addition and subtraction, among other things) which also admits multiplication and division (as long as you don't divide by the $0$ vector).
So you might think of a division algebra as a "number system" (since we can add/subtract/multiply/divide) which contains the reals $\mathbb{R}$ in a nice way. Indeed, $\mathbb{R}$ itself and $\mathbb{C}$ are two good examples of division algebras.
A normed division algebra is a division algebra which also admits a norm $x \mapsto |x|$ which tells you how "big" a number is. Again, $\mathbb{R}$ with the absolute value function, and $\mathbb{C}$ with its standard norm $a + bi \mapsto \sqrt{a^2 + b^2}$ are both examples of this object too.
The drama:
There is a theorem due to Hurwitz which says that if you restrict your attention to "finite dimensional" systems, the only such number systems are
So if you are interested in "number systems" which contain $\mathbb{R}$, admit a notion of "size", and "aren't too big", you are FORCED into picking one of these systems.
Rather interestingly, you lose something on each stage.
- Passing from $\mathbb{R}$ to $\mathbb{C}$, you lose the ability to order your numbers (is $1 < i$?)
- Passing from $\mathbb{C}$ to $\mathbb{H}$, you lose commutativity (so $ab \neq ba$ in general)
- Passing from $\mathbb{H}$ to $\mathbb{O}$, you lose associativity! That is,
$(ab)c \neq a(bc)$ in general!
What is most interesting, though, is that a few months ago, somebody published an article saying they had found a new algebra, which was like $\mathbb{O}$, but managed to preserve associativity!
That is, they claimed the existence of an $8$-dimensional, associative, normed division algebra.
This is certainly a "number system" of the kind you might be interested in. It is also nonexistent by Hurwitz' Theorem! The paper has since been retracted from the journal, though I believe the author is still fighting the decision, claiming their result is actually true.
So you see, there are cases where mathematicians accidentally study objects which don't exist. There are lots of apocryphal examples as well, but most of them don't fall under the heading of "number systems".
As a rather quick reply to your other questions:
- Can a number system fail to exist even when its axioms are consistent?
My instinct says "no". Certainly in the case of a First Order set of axioms, a system exists if and only if the axioms are consistent. This is Godel's Completeness Theorem.
I'm notably less well versed in higher order logics, such as those needed to formalize the completeness of $\mathbb{R}$. In general, completeness/soundness/effectiveness cannot all happen at the same time for higher order logics, and I'm far from an expert in the subtleties of what is and isn't possible. Rather funnily, this is a corollary of Godel's Incompleteness Theorem.
I'm sure if you post another question on this site asking for specifics of completeness of second order logic, you'll find some more specialized logicians than me who are willing to help :)
- What does it mean for a system to "not exist"
I think this is better asked to a room of philosophers than a room of mathematicians. For me, it "doesn't exist" if there isn't a model. That is, there is no set equipped with some functions that actually satisfy the axioms.
For instance, there is no noncyclic group of order $2$. Why does this not exist? Because you can look at a set of size $2$, and try all possible group structures, and see for yourself that none of them work. It's the same idea for other objects that "don't exist", we just need cleverer arguments since we can't check by hand that all (possibly infinitely many) options fail.
I hope this helps ^_^