2

I have 37 files of different sizes, containing a total of 7 million words.

I want to estimate the frequency of each word in the corpus.

Let's say I find the word "Cranberry" only twice in the corpus, then it would be naive to state the frequency of this word as 2/7000000. This is because the odds are very high that I could, for another corpus establish a frequency of 0/7000000 or 4/7000000 for the very same word. So in other words, the statistical error for estimating the frequency increases as the frequency of a word decreases. Is there a way of estimating how great the statistical error is? For example, lets say I was to say that "the" occurs 405139 times in the corpus, how accurate is it to say that the frequency of the word "the" in the English language is 405139/7000000? Does it help, if I present the frequency of "the" in each of my 37 files?

219/3293
992/15072
321/4792
358/6736
147/1819
388/4047
312/6567
349/7182
735/11233
5789/118655
18359/300927
236/2677
91/1135
1664/26761
585/9709
477/10414
248/3253
238/3857
41634/670576
1703/34261
246/3080
69971/1005119
195/5372
1362/25915
896/22999
1186/25515
109/3360
72/1832
203/4261
78673/1363843
67210/1229371
69277/1330254
93/2542
1888/66501
1451/27679
32382/514278
5080/87831

If I only have 2 cranberries in my corpus, how large a corpus would I need to get a reasonably accurate indication as to its frequency?

user111322
  • 137
  • Generally, I would say that most statistics I have seen rely on getting $~95%$ accuracy (approximately one standard deviation worth IIRC), which would say that you would want to find enough material that the ratio $2/7000000$ appears among $~95%$ of the groups of text you assemble. – abiessu Sep 12 '13 at 13:44
  • @abiessu What is IIRC? – user111322 Sep 12 '13 at 14:18
  • Sorry, old abbreviation for if I recall correctly... – abiessu Sep 12 '13 at 14:24
  • Am i very wrong to think that both cranberries are in the same file? how did you select the files ? (can hardly be random) – Willemien Sep 12 '13 at 16:38
  • @Willemien Mentioned twice in the same novel. I have created the English corpus myself by taking films and books and some other smaller free corpora. – user111322 Sep 12 '13 at 19:02

1 Answers1

1

Let $p$ be the proportion of words in written English everywhere that are "cranberry". You are trying to estimate $p$ based on a sample of size 7000000, and you want to know something about the accuracy of your estimate. This is standard statistics stuff. For example. take a look at this page.

The simplistic estimates are based on the Central Limit theorem and an assumption of normal distribution. As noted on the page cited above, these simplistic methods don't work very well when $p$ is close to zero or one, and you need to use more complex methods.

In the techniques that I'm familiar with, frequencies in the individual files are not relevant -- it's only the overall frequency in the entire sample that matters. This should make sense, intutitively. Obviously you could split or join the files in arbitrary ways, and there is no reason that this should affect your estimate.

Some answers to specific questions:

how accurate is it to say that the frequency of the word "the" in the English language is 405139/7000000?

Look at the confidence intervals derived from the Central Limit Theorem on the Wkipedia page I cited.

Does it help, if I present the frequency of "the" in each of my 37 files?

No. Not as far as I know. See above.

If I only have 2 cranberries in my corpus, how large a corpus would I need to get a reasonably accurate indication as to its frequency?

Again, the Central Limit Theorem will tell you how large a sample you need in order to get a given level of confidence in your estimate.

bubba
  • 43,483
  • 3
  • 61
  • 122
  • @abiessu The word count is 7 million. – user111322 Sep 12 '13 at 13:53
  • I guess my concern would be if the name "Bubba" were to appear in a certain film (say Forest Gump) which is included in my corpus. This will give an exaggerated frequency for Bubba for just one of the files. Can I not take advantage of this fact when estimating how accurate my frequency estimation for Bubba is? – user111322 Sep 12 '13 at 13:59
  • The problem you describe is universal. Whenever you sample a population, there is always some chance that the sample is biassed in some way, and not truly representative of the population. As far as I know, the only ways to solve this problem are: (1) be careful how you choose the samples, and (2) make the sample as large as possible. – bubba Sep 13 '13 at 00:26
  • Also, as I mentioned, you shouldn't be estimating based on individual files, you should use the entire set of text from all files. So, maybe the Forest Gump file has a "Bubba" bias, but the entire set of text will not have this bias, provided it's much larger than the Forest Gump file. I recommend you find a local statistician to talk to. – bubba Sep 13 '13 at 00:31