Our experience shows that all finitary mathematical objects could be encoded using the natural numbers, and all operations on those objects could be expressed in terms of a few basic operations on numbers: the successor operation, addition and multiplication.
As far as I understand, this is not a theorem in the strict sense (because the notion of a finitary mathematical object is somewhat vague), but is rather a principle with a status similar to the Church–Turing thesis. Nevertheless, the essence of this principle can be expressed by a precise statement, for example, that the theory of hereditary finite sets and the Peano arithmetic are mutually interpretable.
As it turns out, other familiar operations on integers — the exponentiation, prime factorization and in fact (invoking the Church–Turing thesis), every effectively computable operation — can be expressed using the few aforementioned basic operations.
Now, the set theory encompassing infinite sets and the powerset operations is much richer than the Peano arithmetic. But we can observe that the universe of sets contains a subclass similar in some of its properties to the natural numbers — the class of ordinals. So, here is my question:
Is it possible to reduce studying of (finite and infinite) mathematical objects to studying of ordinals, and to build a formal theory free of the concept of an arbitrary set, the theory whose universe consists of ordinals only, that uses only a few basic operations on them, but the theory that is still powerful enough to serve as a foundational theory of all mathematics — in the same sense as it is possible to use the Peano arithmetic as a foundational theory for finitary mathematics? And if so, what could be its language and its axioms?
To clarify, I'm not asking if it would be useful or convenient to use such a theory, but I'm interested in a mere possibility to do that.