Value-Addedness

It struck me the other day, as I was reviewing the statistics on school performance, that something was going rather wrong with certain calculations. Let me explain.

Here is how we calculate something called value-addedness in a nationwide high-stakes standardized examination:

1. You get everyone in Grade 6 to be tested in Math, Science, English and a second language. This gives you a score for the entire country.
   
2. Then you test them again at Grade 10 and Grade 12 (well some skip the Grade 10 test).
   
3. This gives you a statistical measure of what kind of performance at Grade 10 or Grade 12 students ought to produce, relative to their score at Grade 6.
   
4. If a student has got a significantly higher score than this expectation, this is called 'value-addedness'.

For example, if Jack scores 250/300 at Grade 6, he is in the top 10% and is expected to get straight As at Grade 10, and so on. The method is a bit more well-refined than that, and a little worse. For a start, they don't actually factor how well Jack did in individual subjects at Grade 6, but they do track how well Jack's seniors did in individual subjects at Grades 10 and 12. They use a rolling average over n years to computed the expected score, which is only useful if there is no clear upward or downward trend.

If the trend is upward, then the rolling average is actually lower than the current expectation; if the trend is downward, then the rolling average is actually higher. If a school policy means that everyone must take subject X, the chances are that X will regress towards a mean; if a school policy means that only the elite can take subject Y, the chances are that Y will have so high a mean score that it becomes meaningless to think about what value-addedness means.

What if, however, the results for the whole country are said to be better than before, every year? Well, then that's very odd. It means that value-addedness should be declining. Actually, if the statistic is distributed normally, then half the population should be above the mean and half below. If the country is doing better AND the value-addedness is rising, then there must be a very long tail or something.

But the interesting thing is, what if the entire national population is in the top 50% of the world at Grade 6 (or 10, or 12), and it isn't anymore after Grade 12? Does that mean the system is screwy (or screwed)? Something to think about...