Open Letter to my Professors

by Izaak

I.

One of the problems educators face is understanding which methods are good for teaching, and which methods are useless, or even worse, harmful. For a lot of them, it works like this: try a lot of different things, get a gut reaction for what works well, and do the things that feel right.

Most of my professors are actually following this method, but they try and follow up this method with data. They attempt to justify their strategies to their students. And worst of all are science and math teachers, because they think that they know what they are doing.

They will have some sort of optional work, but some work where they can track who is doing the work. And then they’ll do an analysis of test results, find that the students who did the optional work did better on the test, and conclude that the optional work is beneficial.

So the next day when they stroll helpfully into class, they say with a smile on their face that screams I Told You So, Remember to do the optional work! Students who do that have an XX% higher score than those who didn’t! And everyone who did it smiled to themselves because they did the optional work, and that’s why their test grade was so great. Everyone who didn’t do it thinks about their test grade and thinks, if only I had done the optional work, I would have gotten a good grade.

II.

Why didn’t they do the optional homework in the first place? Probably because they were overworked, or sick, or lazy. Who did the optional homework the first time? Those who had already finished their other work, or were healthy, or who worked hard.

But these traits that correlate to who did the homework also seem like very possible traits that would correlate to who did well on the test, even if the optional homework hadn’t been presented as an option. So how can we tell if it’s those traits that caused people to both do the homework and excel on the tests, or if doing the homework helped people do the tests?

That’s what randomized controlled trials are for. For some reason, people (at least in my social groups) complain about the lack of blinding in trails far more than the lack of randomization, even though randomization seems a lot more important most of the time. In this instance, by letting students choose whether or not to do the homework, the students in the test group were significantly different from the students in the control group—the test group had a lot of smart/motivated/healthy people, and the control group had very few.

You cannot let participants choose which test group to be in. You cannot let participants choose which test group to be in. It means the test can be boiled down to “Do smart/motivated/healthy people do better on tests than people who are not?”

And the answer is “Why the hell did you bother running a test on that? It’s obvious.”

III.

Not that I can blame my professors for being the only ones who don’t realize that you can’t let people choose which trial group to be in. The same effect can be seen in many, many psychological studies. This is called confounding—you think you’re finding that variable X causes Y, when you’re actually finding that variable Z causes both X and Y.

You’ll see this type of result in a lot of pop-health and pop-nutrition articles. My favorite is whenever people draw a line between Vitamin D and… well, anything health related. Heart disease, cancer, Alzheimer’s. There are two ways most people get Vitamin D. People can either spend time in the sun, or if they don’t want to do that, they can take vitamin supplements.

If you look at how much Vitamin D people are getting, it almost always correlates extremely strongly with good health. Most likely this is because people who spend time outside are athletes; they run marathons or go hiking or swim or something active, and people who take vitamins care about their health in other ways; they diet, eat salmon instead of steak, and they know how to pronounce quinoa.

When you run randomized controlled trials, every positive effect of Vitamin D drops away, except for bone health. Worried about Osteoarthritis? Vitamin D. Worried about cancer? You’re barking up the wrong tree.

In most actual psychological and medical studies where they don’t do a randomized controlled trial, they attempt to measure all of the possible “Z” variables, and adjust the data to control for them.

This doesn’t always work. But at least these people are trying. My professors either don’t notice, or don’t care that their data is blatantly wrong. They should care.

In every case so far, the thing that supposedly helps students is homework. Except that homework rarely helps anyone. So all that this extra work is doing is increasing the stress, decreasing the health, and worsening the lives of the students who end up doing it.

It’s less than helpful; it’s potentially harmful.

Advertisements