How to estimate CTR (click-through rate)?

Question

How many times banner should be shown to estimate CTR? For example, a banner was shown x times, and was clicked y times.

CTR = y/x;

How to evaluate incaccuracy of this value?

Answer 1

As a rule of thumb, errors for a number $y$ have the order of magnitude $\sqrt y$. Thus you may be relatively confident that the "true" CTR (i.e. what can be expected as $x$ becaomes larger over time) is probably between $\frac{y-\sqrt y}x$ and $\frac{y+\sqrt y}x$

Answer 2

Here's one way to do that. Assuming that there's a probability $p$ with which a user clicks on a banner that he sees. Then if $n$ banners are shown, the probability of there being $k$ clicks is $$ p(k) = \binom{n}{k}p^k(1-p)^{(n-k)} \text{.} $$ To derive this formula, observe that to observe $k$ clicks, the event CLICKED which has probability $p$ has to occur $k$ times, and the event $NOT(CLICKED)$ which has probability $(1-p)$ must occur the other $n-k$ times. The factor $\binom{n}{k}$ is required because you do not care about the precise sequence of events, i.e. about whether $k$ clicks all occur in a row, followed by $n-k$ not-clicks, or are interleaved in some way.

From this, you get that the probability of observing at least $k$ clicks out of $n$ views given $p$ is $$ P(K \geq k|p) = \sum_{i=k}^n \binom{n}{k}p^k(1-p)^{(n-k)} $$ and similarly the the probability of observing at most $k$ clicks is $$ P(K \leq k|p) = \sum_{i=1}^k \binom{n}{k}p^k(1-p)^{(n-k)} $$

You can now pick an $\alpha \in [0,1]$ and find all $p$ for which $$ P(K \leq k|p) > \frac{\alpha}{2} \text{ and } P(K \geq k|p) > \frac{\alpha}{2} \text{.} $$ The idea being basically that you rule out all values of $p$ for which the probability of observing either fewer or more than $k$ clicks out of $n$ views is larger than $1-\frac{\alpha}{2}$. Thus, the smaller you make $\alpha$, the fewer values of $p$ you rule out. This, btw, is called the Clopper-Pearson interval.