This answer really demands illustrations, but I'm too lazy to draw them. I encourage you to draw your own illustrations as you read. If you like them enough to share, leave a comment—I'll make the answer community wiki so you can add them!
The outermost points
Take an invertible linear map $A$ from one two-dimensional space to another. This map turns the unit circle $\mathcal{S}$ into some mystery shape $A\mathcal{S}$, which we want to understand. If we take a unit vector $\mathbf{u}$ and spin it all the way around the unit circle, $\|A\mathbf{u}\|^2$ can't increase forever, because $\mathbf{u}$ will eventually come back to where it started. That means there must be places where $A\mathcal{S}$ reaches a maximum distance from the origin. Let's focus on those outermost points.
Suppose $A\mathbf{u}_0$ is one of the outermost points of $A\mathcal{S}$. Choose a moving unit vector $\mathbf{u}_t$ that travels at unit speed along the unit circle, passing through $\mathbf{u}_0$ at time $t = 0$. If we graph $\|A\mathbf{u}_t\|^2$ with respect to $t$, we get a smooth curve. At $t = 0$, the graph has a maximum, so its slope must be zero. That's a clue about what $A\mathcal{S}$ looks like near $A\mathbf{u}_0$. Let's study it more carefully.
Over short distances, moving at unit speed along a circle is very close to moving at unit speed along a straight line [1]. In symbols,
$$\mathbf{u}_t \approx \mathbf{u}_0 + t\mathbf{w},$$
where $\mathbf{w}$ is a unit vector perpendicular to $\mathbf{u}_0$. This means that when $t$ is near zero, the graph of $\|A\mathbf{u}_t\|^2$ is very close to the graph of $\|A(\mathbf{u}_0 + t\mathbf{w})\|^2$. In particular, at $t = 0$, the slope of the graph of $\|A(\mathbf{u}_0 + t\mathbf{w})\|^2$ matches the slope of the graph of $\|A\mathbf{u}_t\|$—which is zero.
The function $\|A(\mathbf{u}_0 + t\mathbf{w})\|^2$ is convenient, because we can break it down using dot product algebra:
$$\begin{align*}
\|A(\mathbf{u}_0 + t\mathbf{w})\|^2 & = \|A\mathbf{u}_0 + tA\mathbf{w}\|^2 \\
& = (A\mathbf{u}_0 + tA\mathbf{w}) \cdot (A\mathbf{u}_0 + tA\mathbf{w}) \\
& = (A\mathbf{u}_0 \cdot A\mathbf{u}_0) + 2(A\mathbf{u}_0 \cdot A\mathbf{w})\,t + (A\mathbf{w} \cdot A\mathbf{w})\,t^2.
\end{align*}$$
Now we see that $\|A(\mathbf{u} + t\mathbf{w})\|^2$ is a quadratic function. Its graph is a parabola. Since know the graph has slope zero at $t = 0$, the coefficient of $t$ must be zero. In other words, $A\mathbf{u}_0 \cdot A\mathbf{w} = 0$, which means $A\mathbf{u}_0$ and $A\mathbf{w}$ are perpendicular.
Let's review. We started by looking at $A\mathbf{u}_0$, one of the outermost points of $A\mathcal{S}$. Then we chose [2] a unit vector $\mathbf{w}$ perpendicular to $\mathbf{u}_0$. By thinking about what a maximum looks like, we concluded that $A\mathbf{w}$ was perpendicular to $A\mathbf{u}_0$. In summary:
If $A\mathbf{u}_0$ is one of the outermost points of $A\mathcal{S}$, then $A$ sends every unit vector perpendicular to $\mathbf{u}_0$ to a unit vector perpendicular to $A\mathbf{u}_0$.
The longest axes
Each unit vector $\mathbf{u}$ has a counterpart $-\mathbf{u}$ on the opposite side of the unit circle. Since $A(-\mathbf{u}) = -A\mathbf{u}$, it follows that every point $A\mathbf{u}$ on $A\mathcal{S}$ has a counterpart $-A\mathbf{u}$ on the opposite side of $A\mathcal{S}$. If $A\mathbf{u}$ is an outermost point, then $-A\mathbf{u}$ is too, because a vector and its opposite always have the same length. We've discovered that the outermost points of $A\mathcal{S}$ come in opposing pairs. Each pair sits on a line through the origin: a longest axis of $A\mathcal{S}$.
This point of view is convenient, because linear maps send lines to lines. If $\mathcal{L}$ is the line along a vector $\mathbf{v}$, then $A\mathcal{L}$ is the line along the vector $A\mathbf{v}$. We can use that idea to make our summary statement even more geometric. For brevity, I'll write $\text{line}^0$ to mean a line through the origin, and I'll write $\mathcal{L}^\perp$ to mean the $\text{line}^0$ perpendicular to $\mathcal{L}$.
If $A\mathcal{L}$ is one of the longest axes of $A\mathcal{S}$, then $\mathcal{L}$ and $\mathcal{L}^\perp$ stay perpendicular when you apply $A$. In other words, $A\mathcal{L}$ and $A\mathcal{L}^\perp$ are perpendicular.
We can use this idea to draw a picture of $A\mathcal{S}$. First, pick a $\text{line}^0$ $A\mathcal{L}$ which is one of the longest axes of $A\mathcal{S}$. Since we're working in two dimensions, we can use $\mathcal{L}$ and $\mathcal{L}^\perp$ as the axes of a coordinate grid. When we apply $A$ to this grid, the axes stay perpendicular; the only kind of distortion we can get is stretching along the axes. If you draw a circle on a coordinate grid and then distort it by stretching along the axes, you get an ellipse! That means $A\mathcal{S}$ is an ellipse.
We now see that in most cases, $A\mathcal{S}$ only has one longest axis: the major axis of the ellipse. The exception is when the stretch factors along both axes happen to be the same, making $A\mathcal{S}$ a circle.
[1] If you've ridden a bike or driven a car, imagine locking the handlebars or steering wheel just a tiny bit off center. If you travel a long distance this way, you'll go in a huge circle, but over short distances it will be hard to tell you're not going straight.
[2] In our original reasoning, $\mathbf{w}$ was determined by the direction $\mathbf{u}_t$ was moving at $t = 0$. However, we're the ones who chose $\mathbf{u}_t$, and we could make it move in any direction.
The next dimension
The same reasoning works in three dimensions, where it shows that applying an invertible linear map $A$ to the unit sphere $\mathcal{S}$ turns it into an ellipsoid. Like before, we see that if $A\mathcal{L}_1$ is one of the longest axes of $A\mathcal{S}$, then $\mathcal{L}_1$ and $\mathcal{L}_1^\perp$ stay perpendicular when you apply $A$. The only complication is that $\mathcal{L}_1^\perp$ is a $\text{plane}^0$ instead of a $\text{line}^0$. The trick is to realize that $A$ is a linear map from $\mathcal{L}_1^\perp$ to $A\mathcal{L}_1^\perp$. In other words, it's a map from one two-dimensional space to another—so our two-dimensional reasoning applies. If $A\mathcal{L}_2$ is one of the longest axes of $A\mathcal{S}$ within $A\mathcal{L}_1^\perp$, and $\mathcal{L}_3$ is the $\text{line}^0$ perpendicular to $\mathcal{L}_2$ within $\mathcal{L}_1^\perp$, then $A$ keeps $\mathcal{L}_2$ and $\mathcal{L}_3$ perpendicular. We end up with three perpendicular $\text{lines}^0$: the longest axis $\mathcal{L}_1$, the second-longest axis $\mathcal{L}_2$, and the third-longest axis $\mathcal{L}_3$. These lines stay perpendicular when we apply $A$.
We can extend this reasoning to any finite number of dimensions. Given an invertible linear map $A$ from one $n$-dimensional space to another, we pick one of the $\text{lines}^0$ that $A$ stretches the most, focus on the $(n-1)\text{-plane}^0$ perpendicular to it, and repeat. We end up with perpendicular $\text{lines}^0$ $\mathcal{L}_1, \mathcal{L}_2, \ldots, \mathcal{L}_n$ which stay perpendicular when we apply $A$, and are ordered from most to least stretched. You asked for an answer that doesn't use singular value decomposition, but the joke's on you: if you try to understand why applying a linear map to a sphere turns it into an ellipsoid, you'll inevitably end up understanding singular value decomposition.
