Here's a expanion of the discussion in the comments.
The quoted passage is missing some important information. First, an equation like $U=(u,h)$ means that $U$ is the horocycle in the upper half plane tangent to $u$ on the real axis and of diameter $h$. Second, I'm guessing that $\lambda(U,V)$ refers to some measurement of the "signed hyperbolic distance" between the horocycles $U$ and $V$, in any case there should be an isometry taking one pair $(U,V)$ to another pair $(U',V')$ if and only if $\lambda(U,V)=\lambda(U',V')$. Third, the question of how to apply a matrix like $D$ having negative determinant seems to be fishy.
To answer the explicit question, there's a miscalculation of the fractional linear transformation as applied to points on the real line:
$$\begin{pmatrix} \mu & 0 \\ 0 & -1/\mu \end{pmatrix}(u) = \frac{\mu u + 0}{0u - 1/\mu} = -\mu^2 u
$$
But there's also an implicit question regarding how the fractional linear transformation $D$ transforms diameters. In general, given a hyperbolic isometry acting on a horocycle in the upper half plane, the diameter of the image horocycle is not a well-defined function of the diameter of the given horocycle. On the other hand, this is true for the specific fractional linear transformation $D = \begin{pmatrix} \mu & 0 \\ 0 & -1/\mu \end{pmatrix}$: a calculation shows that given any complex number $x+iy$, the imaginary part of the image $D(x+iy)$ is a well-defined function of $y$. To put it another way, this $D$ takes horizontal lines to horizontal lines. The formula for how $D$ acts on the $y$-coordinate can be obtained by plugging in the pure imaginary number $yi$:
$$\begin{pmatrix} \mu & 0 \\ 0 & -1/\mu \end{pmatrix}(iy) = \frac{\mu iy + 0}{0iy - 1/\mu} = - \mu^2 i y
$$
BUT, how can the imaginary part by negative? I think maybe the formula for applying a negative determinant matrix such as $D$ to a point in the upper half plane should not be $z \mapsto D(z)$ but instead $z \mapsto \overline{D(z)}$, but I cannot tell from what is pasted in the question.
Assuming something like that is appropriate, the summary is that $D$ takes points with imaginary coordinate $y$ go to points with imaginary coordinate $\mu^2 y$, and it takes horocycles of diameter $y$ to horocycles of diameter $\mu^2 y$.
$\lambda(U,V)$ is the lambda length of the two horocycles $U$ and $V$.
– Ricky Blake Jul 04 '14 at 12:50