From Gateway Pundit (with a curtsy to Crooked Timber):
What universe are these people living in?
This weeks disgraceful liberal surprise- Another outlandish Lancet Iraqi Death Estimate reporting 660,000 have died in the Iraq War.
To give that some perspective, that’s like:
* 3-10 Hiroshima atomic blasts
* 6-20 Nagasaki atomic blasts
* Or 10 Dresden bombing campaigns
I’ve seen a lot of “it’s soooo biiig” critiques from right-wing bloggers this week – but also from a couple of left-wingers. The most intelligent version I’ve seen is the “reality check” press release from Iraq Body Count. All of these critiques exhibit what I think of as “The Great Wall Of China Fallacy.”
Sherri: Some statisticians calculated that the great wall of China is made out of 3,873,000,000 bricks.
Michelle: That’s an absurd number of bricks! Why, if you combined the Empire State Building with Lenin’s Tomb and added on the Pyramid of King Tut, you still wouldn’t have 3,873,000,000 bricks! Clearly, the methodology used by that study is flawed beyond belief.
Sherri: What flaws are those?
Michelle: Weren’t you listening? More bricks than the Empire State Building, Lenin’s Tomb and a pyramid combined!
I hear again and again that it’s absurd to think that the Iraq government could be undercounting deaths by a large degree, but never an explanation of why this is absurd. It seems to me that if you want to measure how well official statistics measure death rates, the way to do it is to conduct a random representative sample study of the population. The foundations of statistics have not, contrary to what many conservatives believe, collapsed; but the Iraqi government has collapsed; surely the former is therefore a more reliable source.
Meanwhile, in Iraq itself, it’s clear that there are a hell of a lot more bodies than have been counted.
Speaking of Lenin’s Tomb, Lenin’s Tomb has a lengthy response to the Iraq Body Count press release, which is well worth reading. Here’s a sample:
The next implication is that “Over 7% of the entire adult male population of Iraq has already been killed in violence, with no less than 10% in the worst affected areas covering most of central Iraq.” IBC doesn’t like this because – well, in fact, they do not say. They simply tell us that this is what is implied. One assumes that they think this is inherently unlikely, but surely they were supposed to be testing that implication? Another version of this argument would be: “one implication of the report is that close to 655,000 people may well have died in Iraq. Furthermore, as if that wasn’t bad enough, a further implication is that close to 600,000 of those have died violently. In Iraq! In one of the most violent societies in the world right now! Could you credit it?”
There is no inherent reason it’s impossible for an incredibly large number of people to have been killed in Iraq, any more than it’s inherently impossible that nearly 4 billion bricks were used to build The Great Wall of China. Large events sometimes happen, and to date the best-conducted study of mortality in Iraq indicates that, measure by mortality, Iraq is one such event.
More links:
An epidemiologist discusses the Lancet study.
Tim Lambert responds to the Iraq Body Count critique (scroll down).
Echidne explains the Lancet study’s methodology. And a similar post from health blogger Stayin’ Alive.
Iraqi blogger Zeyad reacts to the study: “I have personally witnessed dozens of people killed in my neighbourhood over the last few months (15 people in the nearby vicinity of our house alone, over 4 months), and virtually none of them were mentioned in any media report while I was there. And that was in Baghdad where there is the highest density of journalists and media agencies. Don’t you think this is a common situation all over the country?”
Fred at Stone Court points out that the death rate suggested by the Lancet study is hardly unprecedented in history.
[Crossposted at Creative Destruction. If your comments fail to be recorded by official statistics here, try there.]
Pingback: Three Wise Men
Pingback: Pacific Views
Pingback: The Truth Laid Bear
Bush apparently dismissed this study as using a method that is “discredited”. He’s right in one sense. The method they use is notorious–for UNDERCOUNTING. The totals they got are probably on the low side. Of course, passive surveillance methods such as Iraq Body Count uses are even more notorious for undercounting. Passive surveillance in a situation this disorganized probably counts no more than 5-10% of events, possibly fewer. So really, the corrected IBC number (ie about 440,000-900,000) and the Lancet figure are pretty similar. The fact that the people running IBC don’t seem to realize this is disturbing. Don’t they have any statisticians working there.
Hello. I tried to post a comment and it didn’t take. Have I been banned or was it a blip?
Ok, the previous comment took so I’m going to assume that the first was lost to a glitch. If it doesn’t work this time I’ll wander over to CD.
So what I said was…Bush dismissed the most recent study of deaths in Iraq as using a “discredited” methodology. He’s right in one sense: the methodology used is notorious for one type of error–undercounting. So the numbers they obtained are probably on the low side. On the other hand, the method used by Iraq Body Count, passive surveillance, always gives undercounts. Under the best of circumstances, one might pick up as many as 50% of events with a passive surveillance method. Under the sort of circumstances IBC is working under (ie third world country, no national system for tracking data, high level of proof demanded, etc), a 5-10% detection rate is more realistic–expect a true total of 440,000-960,000 deaths based on a count of 44-48K. So if one corrects for the expected undercount, the IBC numbers aren’t really so different from the Lancet numbers. What’s odd is that no one at IBC seems to realize this problem. Don’t they have any statisticians working with them?
Pingback: Deltoid
I have some problems with the lancet study that perhaps you can help me with.
First, Death Certificates. The Johns Hopkins/Lancet Study reports that 87% of all fatalities were recorded by official government death certificates. So, if their data cluster method of extrapolating total casualty figures is correct, then there must be 570,000 death certificates issued by the Iraqi government over the last three years. This is over 10 times the number the Iraqi government actually claims to have issued. Are we to believe that somehow 9 victims in 10 counted randomly by Johns Hopkins were issued certificates, but nationwide only 1 casualty in 10 is so documented? Does that make sense to you? How is that JH was so fantastically lucky to find such a high percentage of certificates, if such certificates are issued only 10 percent of the time? Maybe the researchers from JH should enter a lottery pool. If they do, get me five bucks worth.
Car Bombs: The JH/Lancet study calculated deaths from car bombs at a rate that indicated HUGE numbers of un-reported car bomb attacks. As IBC points out, the average car bomb kills 7-8 people, wounds a number several times that large, and is reported by 6 independent news organizations. Since the average “documented” car bombing is covered by 6 different sources, is it reasonable to believe that a huge majority of bombings go completely unreported? And what of the injured? JH is already claiming that 90% of the dead receive no death certificates. They are also expecting us to believe that 90% of the wounded DO NOT SEEK MEDICAL ATTENTION FOR THEIR WOUNDS. I find that statistic in particular to be completely unbelievable.
population: the population of Basra governorate in 2003 was 2,600,000 according to Wikipedia, and 2,443,000 in 2006 according to this link . The JH study has it as roughly 1,800,000 in 2004, a difference of 6-8 hundred thousand. And actually, I now see additional problems. They have Baghdad as 6,554,000, while my last link there has it as just barely over 6 million, though Wikipedia has it as 6.4 million. They also overestimate Anbar’s population by 100,000, Babil’s (another violent province) by 300,000, Salaheddin’s (yet another violent province) by almost 100,000. The difference of the total population of Iraq between the two sources is fairly small: about half a million, so there is clearly something wrong. There isn’t any one definitive estimate of population (I don’t think a real census has been done in a little while), but I find it interesting that the study systematically overestimated the populations of the more violent areas.
And I could go on with similar inconsistencies, but I want to close with one point that disqualifies the entire study, purely from a mathematical perspective. The JH group conducted interviews in every province, or at least in almost every province. Now, as we know, some provinces in Iraq are experiencing considerably more unrest than other provinces. Baghdad, for example, would be very dangerous, but cities in Kurdistan are fairly quiet. The JH/Lancet study undercounted the population of (relatively) peaceful Basra by 800,000 people, while possibly over-reporting the populations of some of the more volatile regions, such as the Sunni Triangle. Now, it could be that JH/Lancet relied on pre-war census information, that either exaggerated the Sunni population or failed to account for possible migration patterns as people tried to get out of the warzone and into more peaceful areas of the country. Or maybe they deliberately fudged the numbers to get a more sensational result. Either way, by overcounting the populations in “hot zones” and vastly undercounting the populations in less violent areas of the country, the group from JH extrapolated national numbers from faulty – and highly exaggerated – local death rates.
In short, I think the JH numbers are off by around a factor of 8 to 10. This still leaves 60K-100K for all of us to be genuinely upset about. We don’t need to resort to imaginary and grotesquely inflated numbers. There are plenty of real dead people in Iraq as it is without inventing more.
Frank,
The death certificate question is very old hat. It is not reasonable to expect the centralized system for collecting death certificates to function particularly well in the middle of a civil war. If I bring my dead sister to the hospital and the doctor gives me a death certificate so I can take her body to the cemetery, and I keep the certificate in my box of important documents, it is easy for me to show it to the interviewers in my living room. There are a thousand reasons and a dozen steps where the record that the doctor wrote a death certificate might get lost before they get to the final report from the central government.
Naively disaggregating the results as point values is unwise. The error bars on car bomb fatalities are huge, much larger than for the total result. Also, what portion of the car bomb news reports are for Baghdad, and how many are for other areas? News coverage is much better in Baghdad than elsewhere, so the fact that some car bombs are getting reported repeatedly does not in any way mean that other car bombs are not getting reported at all. The IBC can’t talk about the average car bomb, they can only talk about the average car bomb attack that gets reported in at least one news source. They can say nothing about the number that don’t get reported at all. As you know from reading the Lancet study (you have read it, right?), other civil wars have had press reporting rates as low as 5%, with the rate of reporting dropping as the level of overall violence goes up.
Also, just as death certificates are likely to be underreported, I see no reason why hospital visits may not also be underreported, nor why people would be failing to visit hospitals that have been over run by death squads. Likewise, while car bombs have high injury rates, execution style gun violence and torturing kidnap victims to death have very low injury rates.
The population question I can’t answer, but the differences you are citing seem to be on the order of 10-25%, with the source you are using and the source the authors are using disagreeing (which you mistakenly call the authors over-estimating). This 10-25% bias, even if we pretend that wikipedia must be more authoritative than the study authors’ source, is nothing like the factor of 8-10 that you then pull out of nowhere.
If you’d like better and more extensive answers to your questions, assuming you are actually puzzled and not merely meeting your trolling quota, the track back above your comment will take you to one of the best sources for coverage of these questions.
Neither. It’s most likely been taken by the spam fairy, and will be restored as soon as Amp gets round to it. the same thing happens at CD, though with more of us with the magic buttons, they perhaps get restored more quickly.
Thanks Charles, no I am not trolling. But I am still particularly bothered by the death certificates. Much has been made of how many deaths go unreported in Iraq and I accept that as a reason why IBC would have such a low number for their estimate. But how does the fact that deaths *don’t* get reported (and so are not issued death certificates) account for the fact that in 87% of the respondants who were asked for death certifocates were able to produce them?
This would seem to say that there is a much high percentage of deaths being reported to officials than is being claimed or the the Johns Hopkins group got unbelievably lucky.
Deaths do get reported, and death certificates issued by local registrars, but the process of filing, forwarding to the central government, and collating these reports is inefficient or corrupt.
A lot of the protests–and responses–that I see are discussing untestable issues.
IBC, for example, notes that it is essentially ludicrous to claim a number of ijuries that suggests only 10% of folks are being treated in hospitals.
IBC gives no particular factual explanation of why this is ludicrous. Nor (though they DO acknowledge that some folks won’t seek treatment) do they set an upper limit of what is/isnot ludicrous.
The responses note that it is perfectly normal that many people won’t seek treatment. The responses note–probably correctly–that hospitals in a war zone have better things to do than track their patients.
Like IBC, the responses use a lot of words like “many”, “often”, “some”, “obvious” (my least favorite, on either side) and so on. None of these words have a quantized or exact meaning and thus none of them are a particularly good defense.
So then we are left in the middle. Both sides are reduced to posting statements which have little exact value. Many arguments on both side sound logically sound but have no particular grounding in the data and/or are based on assumptions which may well be incorrect. That actually leaves me tending not to believe the exact findings of the Lancet study quite so much.
But then does it really matter anyway? Even if they’ve listed a number that’s 200,000 too high, or one that was 200,000 too low, I wouldn’t change my views on the war. I already don’t like the war.
What I am curious to know is if there is anyone out there whose opinions on the war are seriously affected by the accuracy of the body count.
But how does the fact that deaths *don’t* get reported (and so are not issued death certificates) account for the fact that in 87% of the respondants who were asked for death certifocates were able to produce them?
I think the problem is that you’re using “reported” to mean two different things. IBC only counts deaths that are reported in the media by two different sources. This is something very different from a death that is reported to local authorities, such as a hospital or even police station and has a death certificate made out. As others have pointed out, given the level of chaos in Iraq right now it is not at all suprising that many of the deaths reported to and registered by local authorities never get recorded nationally.
Death certificates can be issued by a hospital or even doctor’s office. The hospital is then supposed to report the death to the national registery, but how a hospital in, say, Fallujah, would do that I can’t imagine. So the family might have a copy of a certificate saying “Mr Soandso died on X/Y/ZZ. Cause of death: exsanguination from multiple gun shot wounds” or something like that but the national government might still know nothing of Soandso’s death.
Incidently, one thing that struck me on reading the article was that the rate of non-violent deaths was said to be stable compared with the pre-invasion rate. Given the number of hospitals and clinics destroyed in the invasion–and the lack of rebuilding–this seems highly unlikely. Admittedly, the rate of death was probably already high due to the embargo, random bombings, and Saddam Hussein’s government, but how can the death rate from disease NOT go up when a country’s water purification system, electricity (and therefore refrigeration), hospitals, and transportation system (and therefore supplies ranging from fresh food to chemotherapy) is destroyed I can’t imagine. Again, I conclude that this is further evidence that the estimates given are undercounts. Possibly quite serious undercounts.
frank: The Roberts study used the population estimates of the Iraqi Ministry of Planning in their study. It is not impossible that the M of P has less reliable statistics on population than Wikipedia, but I’d like to know what wiki’s source was before concluding for certain that that was so. Also I doubt that the Iraqi government would willfully overestimate the population of areas with high conflict.
Sailorman,
I haven’t seen anything from the “It can’t really be that bad” school that convinces me that the study is defective. I certainly haven’t written or read anything that would convince me that things were that bad without the existence of the study. I consider the “maybe” and “seems reasonable” claims that I have made in defense of the study to simply be a response to the “that can’t be” arguments against the study. Neither category of arguments carries much weight, certainly not as much weight as the study itself, but I feel the “It could be” arguments are necessary to counter the seeming weight of the “It can’t be” arguments. I hope that my weak tea arguments will help people like frank who look at the study, read the “It can’t be” arguments, and assume that it really can’t be- that there must be some invisible methodological error- to realize that the “It can’t be” arguments are not necessarily sound, and certainly not sound enough to require the assumption of serious methodological errors in the study. I’ve seen a few people temporarily swayed by the “It can’t be” arguments come back to considering the study to be valid, so I believe that my “It could be” arguments are worth making.
I certainly haven’t run across anyone who claims or admits that their opinion of the war was changed by the Lancet study. I can’t imagine anyone coming to likethe war as a result of the study, so it would presumably only be a matter of war supporters becoming disaffected.
The study certainly reminded me that the war is worse than I can easily imagine. Even to someone who reads juancole.com once a week, with his daily summary of 100 deaths from the day’s news, the numbers from the Lancet study are larger than I would have guessed.
Before the study came out, I assumed that the horrific 3000/month number was accurate.
I don’t any longer.
Sailorman,
An additional note (although one I’m sure you already understand).
If the point value from the study is 200,000 too high or too low, then the study is still on target, since their confidence interval is around that size.
The study is only designed to be able to distinguish the approximate factor by which the death rate has increased (doubled or tripled). It isn’t intended to give anything close to an exact number.
If it turned out that they were wrong by several factors, and the death rate had either decreased or stayed neutral relative to the pre-war period, it might make me waiver a bit in my opposition to the war. Back before the war, Amp wrote a piece in which he argued that the war might be for the good, give the death rate from sanctions before the war. He opposed the war anyway, but the idea was out there (along with its variant that the war was for the good because it would end Saddam Hussein’s killing of his own people).
Charles,
yes; i was referring to a 200,000 change in the center of the CI (which would reduce both 95% intervals by about the same amount).
I am still really not sure how I feel about this study. I read a lot of studies, and (hmm, how to put this) something just seems a little “off” here though i can’t put my finger on it. I think it’s mostly a bias concern. But I tend to think that about many studies other than ones which are insanely detailed in their disclosure.
Nonetheless, I’m well aware that my feelings don’t really mean much in the fact of statistics. And even if the study has some errors, I suspect the “real” number is MUCH closer to that measured by Lancet than to any other currently existing study. So I’ll withold judgment for a few weeks until more is written and disclosed and argued. And of course, it doesn’t change my views anyway.
But I tend to think that about many studies other than ones which are insanely detailed in their disclosure.
And this study is a particularly difficult one to disclose properly. I can’t imagine that any of the authors stand to gain financially from anything they report in the study. There are no new drugs or devices under study, no drug companies involved, etc. On the other hand, it is an issue that many people, probably including the authors, have very strong feelings on and those feelings can–conciously or unconciously, bias the results. If I had this paper to review, I would go over it very carefully and skeptically even though–especially because–the results tend to agree with my pre-test guesses (prejudices?) about what the results “should” be. It’s not a double blinded randomized controlled trial of overthrow versus no overthrow of previous government by the US*. Biases are likely to be present. But the methods seem sound and the results unequivocol so I’m inclinded to believe the study, always keeping in mind my own biases.
*The image of a randomized double blinded controlled trial of overthrow versus no overthrow amused me for a good five minutes, but it doesn’t seem a very practical method for dealing with this problem.