I'm doing linguistics work and I'm trying to make a measure of linguistic diversity in a text. A simple calculation seems to be
linguistic uniformity (diversity) = number of words / unique words
But this must "privilege" shorter texts like poetry for diversity, since the author is less likely to repeat themselves in so short a space. For example, Blake's poetry gets a score of 5, whereas the KJV bible scores 79 and this has a lot to do with the size of the bible compared to Blake's poetry. If we analysed more of the Blakes's corpus, his score would definitely go higher.
How can I normalize the calculation, taking into account the inevitable repetition that comes with longer texts, and the fact that shorter texts are so small that we just haven't seen an appreciable part of an author's lexicon? I know that there's a name for this, I just can't remember what it is. I've looked at the wiki article on diversity index but it hasnt helped me.
Apologise if my question seems stupid. I'm a linguist, not a mathematician. Tagged as homework even though it's not because I'm unsure what category to put it under otherwise.
