Background information
My question is mostly mathematic but requires a bit of knowledge, which I will introduce here.
I am currently working through a microsoft paper (p. 20) about the evaluation of recommender systems. There is one specific block of formulas that either is erroneous or, which could be more probable, I do not understand.
The author introduces the Normalized Distance-based Performance Measure (NDPM). When you evaluate the quality of a recommender system, you do not want to punish it for explicitly ordering all items, even though some items might be equally relevant. In other words, when a result is forced to be an ordered list, equally relevant items have to be ordered as well, and this should not lower the quality of the system. The NDPM itself is not a formula of relevance, which is why I will leave it out for simplicity.
Suppose you have $n$ products rated by a certain user. I omit the subscript $u$ for the user, as we always talk about a specific user. A rating is called $r$ for the real data and $\hat{r}$ is the corresponding rating estimation from the recommender system.
Then the sum $C$ (:= $\sum_{ij}\ldots$) iterates over all rating pairs, and it denotes the number of rating pairs for which an order is given. It does not iterate over $n^2$ products. If there were 3 products $X$, $Y$, $Z$, then the sum would iterate over $(X,Y)$, $(X,Z)$, and $(Y,Z)$. This can be written as $\frac{1}{2}n(n-1)$, as explicitly stated in the paper. As in my example there are 3 products, this also results in 3 combinations.
In the paper, $C$ is defined as (it is a bit imprecise, as one can only guess the meaning of $i$ and $j$): \begin{equation} C=\sum_{ij}sgn^2(r_i-r_j) \end{equation} It is obvious here that only those rating pairs account for $C$ in which the ratings of two products differ. Therefore, you get the number of ordered pairs. $sgn$ seems to be the $sign$ function that converts any difference to $-1$, $0$, or $1$. As the square is used, $-1$ always translates into $1$, thus increasing the number of differing item pairs.
The Question
Now the author states:
$C^{+}$ is the number of these [not tied = differing] pairs that the system ranking asserts the correct order [...].
He defines it as follows:
\begin{equation} C^{+}=\sum_{ij}sgn(r_i-r_j)sgn(\hat{r}_i-\hat{r}_j) \end{equation}
Here I wondered whether the statement is correct.
If $sgn(r_i-r_j)$ is $+1$ and $sgn(\hat{r}_i-\hat{r}_j)$ is also $+1$, the sum will be increased by $1$, which is correct, as there is one more pair in which the correct order has been assumed by the estimation system (whose ratings are denoted by $\hat{r})$.
If $sgn(r_i-r_j)$ is $-1$ and $sgn(\hat{r}_i-\hat{r}_j)$ is also $-1$, the sum will be increased by $1$, as $-1\cdot-1$ results in $+1$, which is correct as well.
If $sgn(r_i-r_j)$ differs from $sgn(\hat{r}_i-\hat{r}_j)$, e.g., we get $+1$ and $-1$, then the result will be $-1$. This would ultimately result in a decrease of the sum. In my opinion, it does not make sense that the number of ordered pairs is reduced by an additional unordered pair. This should evaluate to $0$ as well and not modify the sum.
Did I make a mathematical mistake? Did I misinterpret anything? Could the author have been imprecise? Thank you for looking at this issue.
