3

I was looking at this Facts and Figures page for Mount San Antonio College and saw this note for the Age breakdown of the school:

*note: Numbers do not add up to 100 because some students selected "unknown" or chose not to respond.

This is the data:

Age
24.55% 19 or less
34.77% 20 to 24
13.88% 25 to 29
6.04% 30 to 34
3.65% 35 to 39
5.78% 40 to 49
12.30% 50+

The sum of these percentages is 100.97%. I tried to scale them back down by dividing through by 1.0097 but I don't think this "undoes" the voodoo the original researches did.

My question is, what were they doing to the data that would cause these percentages to be inflated? I could guess but I can't rationalize it and I definitely can't test it because I don't have the data. It wouldn't be hard to experiment with a fake survey (i.e. the same question, but fewer respondents with 1 non-response), but what makes an adjustment the "correct" adjustment? How would we know?

EDIT: Ok, what it looks like this is a situation where the survey results were "weighted" to account for non-responses. I found a really nasty PDF writeup called Designing Surveys to Account for Endogenous Non-Response. I can't understand (can't apply) this at all. I dropped out of grad school and this writeup seems to assume I know a lot about Stats or complicated use-cases that I don't. Can somebody explain in this context? Is a model this complex even relevant? Are survey weights non-reversible without the original data (different, less-important question)?

EDIT2: HOLY-POOP, THEY'RE TALKING ABOUT THIS ON NPR RIGHT NOW! (not the nitty-gritty calculations though, just the social cost)

  • 2
    Good observation ... I can't think of any explanation other than a mistake of some sort. – Greg Martin Aug 07 '21 at 07:19
  • We see notes like this sometimes in survey results, I just couldn't find an explanation online. Maybe I didn't ask the question in a way Google understood – user391838 Aug 07 '21 at 07:44
  • In the real world, almost all surveys encounter subjects who answer 'don't know' or 'decline to answer' and some who simply do not answer or who do not return the survey. Usually, percentages reported are percentages based on those who gave responses. The general public might have more faith in surveys if more of them admitted their 'non-response' rate. // If I've been answering questions on this site until 2am and some eager pollster calls me at 7am, one of those non-respondents might be me. I'll bet the survey gets tabulated and published anyhow--better off without my 7am opinion. – BruceET Aug 10 '21 at 03:13

3 Answers3

1

I think mathematics is completely innocent here.

Contents of that webpage had been changed quite a few times over the course of past six years and some of the historical versions can be viewed using the Internet Archive service. While I have not looked at each and every version, I believe the bits I present below to be correct:

  • Between 2015-10-22 and 2016-12-19, the website was showing only two age groups; "24 or younger" and "25+", whose percentages add up to $100\%$ perfectly. It also included a link to a PDF file dating back to 2014 which is still available on the original location (but it is also present in the snapshot taken by the Internet Archive). This PDF file contains more detailed age groups and also explicitly mentions the Unknown age group as $0.1\%$. The ethnicity percentages also include the Unknown group explicitly and they sum up to $100.02\%$, which is not really surprising due to adding up nine different numbers, each of which was rounded to two decimal places. Finally, the percentages for the genders leave out $2.2\%$, corresponding to the unmentioned Unknown category. The data in the PDF also match the information shown on the website.
  • We don't know when exactly the website got updated, but on 2017-03-09, it was already showing the same statistical data on diversity as it is showing today. It also included a link to a new PDF, supposedly created in February 2017 (again, available from the original location or from the Internet Archive). We will get to the actual data in a bit, but let's mention some additional information first.
  • The PDF link apparently remained the same until 2018-10-07 and on 2018-11-07 it was already replaced by a third PDF, dated May 2018.
  • Finally the link to the PDF was last seen in 2020-01-08; it was no longer present on 2020-02-08.

Now, when it comes to the content of those PDFs from 2016 and 2018, both of them show gender percentages different from those displayed on the website. This is not too surprising; it would rather be surprising if they managed to attain the exact same (well, up to two decimal places) percentages of males and females for six years. In other words, the website gender percentages were not updated to correspond to the second or third PDF... and neither were the ethnicity or age groups on the website when the third PDF was released.

When it comes to the second PDF file, its ethnicity data corresponds to what the website is showing (and adding up to $100.01\%$ is perfectly fine) but the age-groups data shows two important differences: It explicitly lists the Unknown age group as being $0.03\%$ and, more importantly, the 40-49 group is shown as $4.78\%$ rather than $5.78\%$ (as the website says). All of a sudden, everything adds up to the exact $100\%$.

Having said that, even the data in the third PDF look like they are affected by some poor rounding decisions: The age groups add up to $99.84\%$ but all of them apart from the Unknown group look like they are rounded to one decimal place rather than two; which could very well explain the greater gap. The ethnicity data looks even more strange; with more than half of them possibly rounded to just one decimal place rather than two.

All in all, I strongly suspect the culprit to be just sloppiness with editing the website (this is pretty much confirmed by the discrepancies between the website and the PDFs linked from it) that introduced that extra percentage point due to a typo, rather than any kind of intentional-statistical-correction shenanigans.

0

The following case is possible-

  • There might be some error during the calculation of data. It might be possible that someone might be age of $49$ years and $6$ months and so this might had been considered as $40-49$ by some because they approximately it $49$ whereas other would have put it in $50+$ since the number rounds to $50$
  • In that case their response would be in one bucket or the other, regardless of their intention, and a conventional calculation should go as expected. The page says this discrepancy is caused by non-responses. So what kind of adjustment(s) could the surveyors have made? – user391838 Aug 07 '21 at 07:43
  • @user391838 Does this and this answer your question?\ –  Aug 07 '21 at 07:47
  • 1
    No, my question already includes the explanation on Survey Monkey: that the weighting is because of non-responses. – user391838 Aug 07 '21 at 08:04
0

I suspect that roundoff errors in the manipulations that attempt to account for nonresponses can account for the fact that the sum of these seven percentages exceeds $100\%$ - by less than one percent.

There is no way to tell without access to the raw data and the manipulation algorithm.

You could probably make up some data and some algorithms that would lead to a similar outcome.

Ethan Bolker
  • 95,224
  • 7
  • 108
  • 199