Suppose I suggest a possible fact about the cards: "Statement#1 - A card that has a vowel on one side must have a prime number on the other."
The four cards are lying on a table, and the faces you can see are:
E K 4 7
Which of the cards should you turn over in order to determine whether Statement#1 is indeed true? (ok, you could turn them all over, but suppose you are extremely fatigued, what would be the minimal amount of work you would have to do.)
You need only turn over two cards, the E and the 4. If you got it right, pat yourself on the back. Apparently, only about 10% of college students get it right.
It's pretty obvious you gotta flip the E, and check to see whether it's got a prime on the other side (as it is supposed to). That's a pretty direct test of what Statement#1 says in plain English.
Probably you didn't flip the K, since Statement#1 doesn't say anything about consonants.
However, there seems to be a strong pull to flip over the 7 -- after all, primes are mentioned in Statement#1. But Statement#1 doesn't say that primes have vowels on the other side... it says that vowels have primes on the other side. Statement#1 doesn't predict anything about the 7 card. We don't need to flip it.
But we do need to flip the 4. Imagine if we did, and there were an A on the other side. That would disprove the idea that all vowels have primes on the other side. So we do need to flip the 4.
Going back to what was wrong with flipping the 7, I said "Statement#1 doesn't say that primes have vowels on the other side... it says that vowels have primes on the other side." Our brains are trying to suck some extra knowledge out of Statement#1, but it hasn't quite done it right. It turned Statement#1 around the wrong way. In logic-speak, we could write Statement#1 as...
If a card has a vowel on one side, then it has a prime on the other.
If V, then P.
Flipping the 7 is like testing "If P, then V", but this statement (the converse of Statement#1) is not logically identical to Statement#1. So testing it does not provide evidence for the truth of Statement#1.
However, the contrapositive of a statement is equivalent to the original. The contrapositive is:
If not-P, then not-V.
If it's not a prime, then it's not a vowel on the other side.
So flipping the 4 clearly constitutes a test of the contrapositive, and the contrapositive is equivalent to Statement#1.
Okay, I've belabored this point in maybe more detail that it deserves, but I think it reveals something about the kind of mistakes that people make.
Flipping the E is obvious. Flipping the 4 is not. Flipping the 7 may make us feel like we're testing something, when it isn't. If, when we are evaluating the truth of some claim, we do the equivalent of flipping the E and the 7, we may be led to accept as true a statement that isn't.
I was actually spurred to post this by something in Phil Plait's Bad Astronomy, which I have finally cracked open. He was discussing the old 'you can only balance eggs on the equinox' urban legend, which provides a different wrinkle on testing a statement. The 'only' in there makes rephrasing it a little tricky, but it comes out as:
Statement#2: If and only if it's the equinox, then eggs can be balanced on end.
This is equivalent to the combination of:
2A: If it's the equinox, then eggs can be balanced on end.
2B: If eggs can be balanced on end, then it's the equinox.
Hypothetical Frank, maybe led astray by a TV news report showing kids balancing eggs on the equinox, goes to his fridge and pulls out some eggs. Frank makes a direct test of statement 2A. It's the equinox, and he works at it and manages to balance an egg. Success. Statement #2 is supported. And then he plays with 2B. Frank asks himself, "When have I ever seen eggs balanced on end?" The eggs on his countertop were balanced on the equinox. The eggs he saw on TV were balanced by kids on the equinox. Hey-ho, case closed. Statement#2 is proved. At least in Frank's mind.
As usual, he tested the easy ones. He tested the direct reading of 2A, and the direct reading of 2B.
But if we look at the contrapositive of 2B:
If it is not the equinox, then eggs cannot be balanced.
We need to do some testing on a day other than the equinox in order to completely test the logical implications of Statement#2. And what we find is that there is nothing special about the equinox. Yes you can balance eggs on the equinox, but you can also balance eggs on any day ending in y.
By only testing the direct meaning of the statement, we become liable to fall into superstition for lack of a better term. Frank accepted the superstitious belief that only on the equinox can eggs be balanced.
Another wrinkle combines truth with superstition. If the second half of the statement is always true, you can't test the contrapositive. It's like saying
Statement#3: "If I drink herbal tea made from bull scrota, then 2 + 2 = 4."
I try to test the contrapositive, tapping endlessly at my calculator waiting to find 2+2 equalling something other than 4, so I can record whether I had the tea of not. But that never happens. And no disproof of statement#3 is forthcoming.
Statement#3 is logically true, but seems to express a superstitious connection between the two halves of the statement that does not exist.
Similarly, "if it is the equinox, then eggs can be balanced" is also logically true. But if we try to sneak in the "if and only if" that would make the statement more useful, then it proves false.