Lets say that when you point the compass north $(0^{\circ})$, the compass values have the distribution $X_{0^{\circ}}\sim F_{0^{\circ}}$. Hopefully, $E[X_{0^{\circ}}]=0^{\circ}$, otherwise, your compass is biased, so you will need to remove that first.
Once you have an unbiased $X_{0^{\circ}}$, you can begin to characterize the typical variability of the readings when it points north.
This can be set up as a hypothesis test of a population mean.
Let $X_o$ be the population of possible readings from the compass when it is pointing in some unknown direction. We can formulate the following hypothesis test:
$H_0:=E[X_o]=0$ vs. $H_a:=E[X_o]\neq 0$
Let the test statistic be the sample mean: $\bar X_n$, where $n$ is a pre-determined number of measurements.
The next step is to determine the null distribution of $\bar X_n$ when $H_0$ is true. We can do this nonparametrically by resampling from the values you got for $X_{0^{\circ}}$ (i.e., sample $n$ values from $X_{0^{\circ}}$ with replacement, calculate the sample mean, record the value, and repeat about 5000 times...the distribution of these values iks your bootstrap null distribution). I would check to see if this distribution is reasonably normal...if it is, then you can approximate the null distrbution using a Normal distribution with mean $0^{\circ}$ and standard deviation $\frac{s(X_{0^{\circ}})}{\sqrt{n}}$
Next, you need to determine your willingness to accept two types of errors:
1.The probability that you conclude it is not pointing north when it actually is (Type I). This probabiliy is called $\alpha$
2.The probability that you conclude it is pointing north when it is actually not (Type II). This probability is called $\beta$
Related to the Type II error, you need to determine what range of directions around $0^{\circ}$ you are willing to accept as north (e.g., $350^{\circ} - 10^{\circ}$).
Once you answer the above questions, you are in a position to determine the correct value of $n$ and the rejection region $R_{\alpha,n}$ of $\bar X_n$ (which will simply be the upper and lower $\frac{\alpha}{2}$ of the distribution of $\bar X_n$:
You'll need to determine the sample size $n$ that will ensure that $P(\bar X_{n,0^{\circ}}-10^{\circ} \in R_{\alpha,n})\leq \beta$ and $P(\bar X_{n,0^{\circ}}+10^{\circ} \in R_{\alpha,n})\leq \beta$ so that you achieve the proper sensitivity and the associated $\alpha$-level rejection region.
Sorry for the long-winded answer, but I think if you take enough samples, you can rely on the normal approximation and then determine the sample size and rejection region using the theory associated with the $Z-test$.
