Fudging War Data at The New York Times

There is a good reason why my fellow scientists and I follow strict rules for interpreting data: it helps prevent biased conclusions. For the same reason, journalists would do well to adhere to those rules when they write articles involving the interpretation of data. But they don’t always do so, often at the cost of accuracy. A February 2010 article in The New York Times about the increased use of drones in Afghanistan illustrates my point.

The print version of the article included a sidebar (reproduced in full below) with three graphs and their corresponding captions.

ThreeGraphs –

The middle graph of the Times‘ sidebar shows the number of missiles fired by drones during each month of 2009, while the bottom graph shows the total number of bombs and missiles from all types of aircraft for the same period.

The similarity in the patterns in the two graphs is apparent to me (and presumably to you). Both data sets show an initial rise in the use of munitions, then a decrease around mid-year, followed by a peak in the fall. In both cases, the total for the last month of 2009 exceeds that for the first month. The slopes of trend lines through the data of each graph are weakly positive and indistinguishable from each other within the uncertainty. I note, however, that plotting trend lines through these data sets, with their two distinct peaks, is not particularly informative.

I assume that the Times’s staff responsible for the article also recognized that the two data sets look alike. But the conclusion that the newspaper drew from these data discounted that obvious similarity. Instead, the Times reported that the two graphs show different trends, specifically, that the use of munitions from drones increased during 2009 (the middle graph in the sidebar), while those from all aircraft decreased (the bottom graph).

How can the Times conclude something so at odds with what simple inspection indicates? Answer: by violating one of the rules in science regarding data analysis. The paper decided, without any stated justification, to ignore the largest peak (what is calls the “spike in October”) on the graph of munitions from all aircraft (the bottom one), as this caption with that graph demonstrates:

But the total number of bombs and missiles from all aircraft has gone down, except for a spike in October, because of Gen. Stanley A. McChrystal’s order to reduce civilian casualties.

Thus, the difference in the trends that the Times claims for the two graphs is manufactured, that is, it only exists as the result of the paper’s selective exclusion of data. While scientists may also find occasion to exclude data, they do so based on well-established statistical tests for detecting spurious results, or for some other stated justification. The Times‘s arbitrary exclusion of data here does not meet that standard and results in an unjustified conclusion.

Fudging data about war strategy is particularly egregious. In the case at hand, General McChrystal’s goal, according to the graph’s caption, was to reduce civilian casualties by reducing the use of munitions from all aircraft. But if the reported decrease in munitions use is just an artifact of data manipulation, the relationship between munitions use and civilian casualties cannot be accurately established (or conveyed to those who read the article).

In sum, there is a lesson to be learned from the example I present here. Journalists who ignore the rules of science when interpreting data risk misleading the public.