Before You Believe Any 'Study'...

Scientific studies can reveal powerfully important information. Often, however, they are deliberately built (or perverted) to advance some political purpose. Sowell shares some tips for learning to tell the difference.
Thomas-sowell

Whenever I hear the phrase “studies prove” this or that, it makes me think back to the beginning of my career as an economist at the Labor Department in Washington.

Secretary of Labor Arthur Goldberg was scheduled to appear before Congress to argue in favor of some policy that the Labor Department wanted enacted into law.

Down at the bottom of the chain of command, I was given four sets of census data that had not yet been published and was told to analyze these data for a report to go to the Secretary of Labor.

Two of these sets of data seemed to support the Labor Department’s position but the other two went counter to it. When I wrote up a paper explaining why this was so and concluded that the statistics overall were inconclusive, there was much dismay among those in the hierarchy between me and the Secretary.

They were also puzzled as to why anyone would write up such a paper, knowing what the Department’s position was on the issues. They took my paper, edited and rewrote it before passing it up the chain of command.

Secretary Goldberg then made his usual confident presentation of the rewritten study to Congress, probably unaware of the contradictory data that had been left out.

It was a valuable experience so early in my career to learn that what “studies prove” is often whatever those who did the studies wanted to prove. Labor Department studies “prove” whatever serves the interest of the Labor Department, just as Agriculture Department studies “prove” whatever serves the Department of Agriculture’s interests.

It is the same story on the other side of the Atlantic, where a new book about Britain’s criminal justice system exposes the fraudulent methods used to generate statistics about the “success” of various programs of alternatives to imprisonment. The book is titled “A Land Fit for Criminals” by David Fraser.

The numbers may be accurate but the definition of “success” makes them meaningless. When a criminal is put on probation and the probation is not revoked for a violation, that is “success.”

Unfortunately, the British criminal justice system does not automatically revoke probation when a criminal commits a new crime.

A criminal on two years’ probation can commit a crime after six months, be convicted and sentenced — and, after serving his sentence, go back to completing the remaining 18 months of his probation, producing statistical “success” for the probation program. That is the whole point of the “study.”

On either side of the Atlantic, it is a terminal case of naivete to put statistical studies under the control of the same government agencies whose policies are being studied.

Nor will it do any good to let those agencies farm out these studies to “independent” researchers in academia or think tanks because they will obviously farm them out to people whose track record virtually guarantees that they will reach the conclusions that the agency wants.

Climate expert Richard S. Lindzen of M.I.T. has indicated that the vast amount of government research money available for studies of “global warming” can discourage skeptics from being vocal about their skepticism.

This is not peculiar to studies of “global warming.” Many people who complain about the corrupting influence of money never seem to apply that to government money.

If high government officials were serious about wanting to know the facts, they could set up an independent statistical agency, along the lines of the General Accounting Office, to do studies of the effects of the policies of the operating agencies.

That would mean that the fox would no longer be in charge of the hen house, whether the fox was the Labor Department, the Commerce Department, or any of the other departments and agencies.

It would also mean that various bright ideas originating in Congress or the White House would now be exposed to the risk of being shown to be costly failures or even counterproductive. Whole careers could be ruined among both elected officials and bureaucrats.

Don’t hold your breath waiting for it to happen. But do keep that in mind when someone says “studies prove...”

My late mentor, Nobel Prize-winning economist George Stigler, used to say that it could be very instructive to spend a few hours in a library checking up on studies that had been cited. When I began doing that, I found it not only instructive but disillusioning.

A footnote in a textbook on labor economics cited six studies to back up a conclusion it reached. But, after I went to the library and looked at those six studies, it turned out that they each cited some other study — the same other study in all six cases.

Now that the six studies had shrunk to one, I got that one study — and found that it was a study of a very different situation from the one discussed in the labor economics textbook.

Some years back, there was a great flurry in the liberal media because a study showed that (1) black pregnant women received prenatal care less often than white pregnant women and that (2) infant mortality rates were higher among blacks.

There were indignant editorials in the New York Times and the Washington Post blaming the government for not providing greater access to prenatal care in order to stop preventable deaths of infants.

After getting a copy of the original study, I discovered that in the same study — on the very same page — statistics showed that (1) Mexican American women received even less prenatal care than black women and that (2) infant mortality rates among Mexican Americans were no higher than among whites.

A few pages further on, statistics showed that American women of Chinese, Japanese and Filipino ancestry also received less prenatal care than white women — and had lower infant mortality rates than whites.

Apparently prenatal care was not the answer, though it was the kind of answer that suited the mindset of the liberal media and provided an occasion for them to wax indignant.

More recently, the National Academy of Sciences came out with a study that supposedly proved beyond a doubt that human activities were responsible for “global warming.” A chorus of voices in the media, in politics and in academia proclaimed that this was no longer an issue but a scientific fact, proven with hard data.

The NAS report not only had statistics, it had an impressive list of scientists, which supposedly put the icing on the cake.

The only problem was that the scientists had not written the report and in fact had not even seen it before it was published, even though they had some affiliation with the National Academy of Sciences.

At least one of those scientists, meteorologist Richard S. Lindzen of M.I.T., publicly opposed the conclusion and has continued to do so. But that fact was largely lost in the midst of the media hoopla.

Besides, what is a mere meteorologist at M.I.T. compared to Al Gore and his movie?

Nobody can afford the time to check out every claim of what “studies prove.” Even with the help of outstanding research assistants, I can only check out some.

However, the big television and print media have ample financial resources to check out claims before they present them to the public as “news.” But when “60 Minutes” didn’t bother before basing a story about President Bush’s national guard service on a forged document, do not look for a lot of zeal for facts when that could kill a juicy story or the political spin accompanying it.

Let’s face it. There is not much pay-off to checking original sources.

Once a minister was explaining to me the structure of his funeral orations. He said, “At this point, you are expected to say something good about the deceased. Now, Tom, if I were preaching your funeral, what would I say good about you at that point?”

He thought and thought — for an embarrassingly long time. Finally, he said gravely: “In his research, he always used original sources.”

I’ll take that.

Thomas Sowell is a Senior Fellow at The Hoover Institution at Stanford University in California. He has published dozens of books on economics, education, race, and other topics. His most recent book is Black Rednecks and White Liberals, in which he argues that "internal" cultural habits of industriousness, thriftiness, family solidarity, and reverence for education often play a greater role in the success of ethnic minorities than do civil-rights laws or majority prejudices.

Log in to post comments »  

To post comments, please log in first. The Atlasphere is a social networking site for admirers of Ayn Rand's novels, most notably The Fountainhead and Atlas Shrugged. In addition to our online magazine, we offer a member directory and a dating service. If you share our enjoyment of Ayn Rand's novels, please sign up or log in to post comments.
To post comments, please log in first. The Atlasphere is a social networking site for admirers of Ayn Rand's novels, most notably The Fountainhead and Atlas Shrugged. In addition to our online magazine, we offer a member directory and a dating service. If you share our enjoyment of Ayn Rand's novels, please sign up or log in to post comments.