December 5, 2013

The danger of data drones

Sam Smith

One of the problems with test scores such as PISA is that they assume  the test is accurate enough to show significance despite statistically minor differences. But, as Dana Goldstein writes in Slate:

The truth is that the lessons of PISA for our school reform movement are not as simple as they are often made out to be. PISA results aren’t just about K–12 test scores and curricula—they are also about academic ability tracking, income inequality, health care, child care, and how schools are organized as workplaces for adults.
To figure out what PISA results really tell us, let’s first look at what’s on the test. PISA is quite different from the mostly multiple-choice, fact-driven state exams American kids take annually. The idea of PISA is to test students’ ability to handle words and numbers in real-world situations. One math activity asked students to compare the value of four cars, using a chart showing the mileage, engine capacity, and price of each one. American kids were especially bad at problems like this, in which they were not provided with a formula, but had to figure out how to manipulate the numbers on their own.
A reading activity asked test takers to read a short play, and then write about what the characters were doing before the curtain went up. The challenge is that the question prompts students to envision and describe a scene not actually included in the text itself. These are good questions that most of our kids should be able to tackle—we want analytical, creative children, not just kids who are good at memorization.
Let’s look at some of the PISA figures. Here are some the media didn’t report:
There was only a 13% difference in the score for the top country in 2012 and America’s. In 2000 there had been only an 8% difference. 

Thanks to a 3% improvement in scores, Japan came in 4th place in 2012. It was in 8th place in 2000.
Despite Race to the Top and Common Core, if you’re going to play this game then you have to say both are failures because America’s test scores went down 1% between 2000 and 2012.

Our data obsession teaches us to ignore the obvious, but because we don’t know how to test for that we stick with the most easily obtained data. For example, if the wealthiest schools in New York are spending 80% more per student than the poorest ones, then it is likely this will have some effect on test scores.  If we teach students to use the numbers they learn in a different way than, say, Finland, it is likely to have an effect. But what test can tell you that Finland’s approach is better or worse than ours?

And it’s not just in education. For example, the Obamacare disaster is not one of bad data but of the unwise use of data. It assumed that people would react more like programmed robots than like quirky human beings. And that just establishing the goals for a web site would produce the web site one wanted.

Similarly, our medical industry blindly uses BMI as a major health measure, despite the fact that we are three, and not two, dimensional people and that muscle and fat have different health implications.

When you’re dealing with millions of people, it’s so much easier to come up with such cozy formulas than it is to admit honestly the complexities of the situation. And the beauty of it is that those who do it never have to take a test that might prove them to be in error.


greg gerritt said...

I went to a a meeting on economic development and basically everyone there agreed that the testing regime schools now use, the Common Co9re, produces robots for industry that are not as good as real robots. We ought to be teaching for creativity, which means no more standardized testing

Anonymous said...

My methodology prof gave us a heuristic for deciding what's important:

- if it's statistically significant, it's good for publication.

- if it requires statistics to detect, it's not significant outside academia.

Genuinely important effects can be seen with the naked eye.