Calling Bullshit 7.6: Science Is Not Bullshit

Calling Bullshit 7.6: Science Is Not Bullshit


[MUSIC PLAYING] LECTURER: So is most
of science bullshit? I think not, and
I think there are a bunch of reasons why we can
be pretty confident that science is nevertheless
working pretty well. One of them is simply that
most of the hypotheses that people test, it’s an
argument against Ioannidis. Most of the hypotheses
that people test are not super unlikely. I’ve got a limited
amount of money. I’m not going to go chasing
after wild geese in my lab with my limited funds. I’m going to pick
something that I think is relatively likely to
give me a publishable result. And so imagine, if you test 400
patients for a common disease. Let’s take herpes simplex
one, 54% of Americans have herpes simplex
one, cold sores. If you test for that, then
what you’re going to find is– oh I’m missing a
little, we should have some false positives over here. There they are, we should have
a couple of false positives as well. So I’m going to get a whole
bunch of true positives. And I’m going to pick up a
couple of false positives. And the same thing happens if
I test 400 likely hypotheses. So if these hypotheses are
50/50 likely to be true, then same thing,
I’m going to pick up a whole bunch of true things. And if I’ve got a
5% confidence level, then I’ll have some that
aren’t true as well. But fine, most of
them are still true. So that’s one bit
of reassurance. Another thing is that
science and replication studies are cumulative. When I write a paper,
no one goes and sets out to replicate the paper exactly. You can’t get
funding to do that. But if my papers interesting,
people try to build on it. So when someone publishes
a paper and says, hey, there’s this
thing, CRISPR that allows us to insert DNA
into sort of targeted places and basically
rewrite the genome. Even if people don’t
sit down and try to replicate the
initial experiments, you’ve got research groups,
and biotech companies, and stuff around the world
trying to get this thing to work in their own systems. And so if it really
doesn’t work, I mean it’s such a useful thing
and so far really doesn’t work. People are going to find out
quite quickly because everyone is trying to get it
to work and they’re going to say, damnit,
it doesn’t work. And so even though no
one’s really setting out to directly replicate
most findings, people do build
on those findings. And if they’re wrong
then that comes out in the process of trying
to build on these things. So that’s another of
cause for comfort. There’s multiple ways to
deviate from an null hypothesis and still get an
interesting result. I gave you this kind of
hypothesis testing framework where I either accept or
reject a null hypothesis. But I might measure
the effect direction. Maybe I say OK,
this antidepressant made depression better, made
it worse, or didn’t change it. And if I do that– I mean suppose there’s
really no effect, and I can publish any of the
results that are statistically significant, then of the
statistically significant results half are going
to make it better, and half are going
to make it worse. And so what I’ll
have is a collection of papers that get published
that are kind of split. And from that we can see
the signature of this, they’re really actually
being no effect. So this is this can be helpful. Contrarian results
become interesting. Once everybody believes in
some particular psychological phenomenon or
whatever, if I come up with this psychological
phenomenon no one’s ever thought
about it, and I test it, and it doesn’t
work, nobody cares. But once everybody
believes that it works, if I do a carefully
controlled experiment that shows that it doesn’t
work, that is now big news. It’s not boring anymore. It’s very interesting. People love conflicts
and arguments and that kind of thing. So people care a lot. And you can publish that stuff. It’s still tricky, but
you can publish it. It gets attention. It gets exciting. And so science– if
people start to build up a series of false positives
that are indicating that one sort of result is
true even though it’s not, now it becomes
publishable to say, oh and actually that doesn’t work. And so that’s kind
of self-correcting. And then finally, look, to
heck with all of this theory, empirically science works. Science just bloody works. And that’s one of the
most important things. I mean as we said at the
start, science gives us these lifesaving vaccines. Science lets us fly across the
country at 500 miles per hour. And you guys are lucky
enough to live in an era where you can go online. You can order a
Bluetooth salt shaker or a water bottle that will
remind you that you’re thirsty. And what could possibly
be better than that. Thank you. [MUSIC PLAYING]

Leave a Reply

Your email address will not be published. Required fields are marked *