Over the years, I’ve frequently referred to the National Violence Against Women Survey’s statistics regarding rape prevalence. ((A relatively brief summary of the NVAW survey’s findings can be read here.)) And I’ll probably continue to refer to NVAW Survey: it’s probably the best measure that’s ever been taken of national rape prevalence in the USA. It used a large, random, nationally-representative sample; the survey included people “from a variety of racial and ethnic backgrounds”; and it asked about some things no previous national survey had effectively asked about, such as rape by intimate partners.
Nonetheless, there are important flaws in the NVAW survey, and in particular with how it measures rape prevalence. This is a first of a series of posts which will describe the flaws in the NVAW survey. This post will look at some general shortcomings with the survey; future posts will look at different ways the NVAW study may be both undercounting and overcounting rape prevalence, and at how the NVAW study does a particularly poor job of measuring rape prevalence among men.
1. The NVAW study needs to be repeated at regular intervals. ((Shorter Amp: “The NVAW study is deeply flawed, and we need to do it more often.” :-P))
Although there have been multiple good studies of rape prevalence done over the last twenty years, they each use different methodologies, making it impossible to use these studies to get a reliable estimate of how rape prevalence has changed over the years. Not all the studies are structured the same way; not all of them use nationally representative samples; not all of them use exactly comparable questions. As a result, we can’t use these studies to see if rape is increasing or decreasing over the years.
Right now, there are only two national, large scale measures of rape prevalence which are repeated every year, and both of them significantly underestimate rape prevalence. One, the National Crime Victims Survey, is believed by most experts to radically underestimate intimate violence, including rape committed by acquaintances, friends and spouses. ((From an NVAW survey research report: “For years, researchers have attributed the low rate of intimate partner violence uncovered by the NCVS to the fact that it is administered in the context of a crime survey. Because they reflect only violence perpetrated by intimates that victims are willing to label as criminal and report to interviewers, estimates of intimate partner violence generated from the NCVS are thought to underestimate the true amount of intimate partner violence.”))
The second annual measure of rape prevalence, the FBI’s Uniform Crime Reports, includes only rapes reported to police. This is problematic, because surveys of rape victims show that the majority of rape victims never report their rape to the police. ((For example, according to this federal study, 95% of college women who have been raped never report the rape to police. According to the NVAW study, 81% of all women who have been raped never report the rape to police.)) So although both of these studies can be used to measure how rape prevalence is changing over time (and they both indicate that rape has been decreasing in recent years), it’s difficult to have confidence in these results, particularly when it comes to rapes committed by acquaintances, friends and spouses.
What’s needed is a state-of-the study of sexual and intimate violence that is repeated on a regular basis: Every year would be ideal, but even every three or five years would be useful. This would give us a much more reliable look at how rape prevalence changes over time.
2.) Ask respondents which state the rape took place in.
Many intriguing studies of rape and intimate violence have compared how rape and intimate violence have changed over time in different states. ((For example, Four Theories Of Rape In American Society: A State Level Analysis by Straus and Baron. For another example, see
Fatal violence among spouses in the United States, 1976-85″ in American Journal of Public Health 79(5), which I discuss in this post.)) When state-level data is available, it is possible for researchers to examine correlations between rape prevalence and other factors. For instance, after controlling for other factors, is intimate rape less prevalent in states which have more help available for battered women? Is rape more common in states with a high male to female ratio? Is rape less common in states in which women have achieved greater social parity? ((According to Straus and Baron’s study, rape is less prevalent in states in which women have achieved greater social parity, even after controlling for a great many other factors. But that study was published in 1993, and used police reports as a measure of rape prevalence; it would be nice to see studies examining this question with better and more recent measures of rape prevalence.)) These are questions that could tell us a lot about the causes and — perhaps — the prevention of rape, but the data needs to be available first.
3. Include demographic questions about disability.
This is a no-brainer. Don Lollar of the Centers for disease control has said “Disability is not an outcome, it’s an input. Disability is a demographic variable –just like age, sex, racial ethnicity, socioeconomic status.” But although the NVAW survey included questions about age, sex, race, and ethnicity, it did not ask respondents any direct questions about their economic status (although it did ask one question — highest level of education — which can be used as a rough proxy for social class). Even worse, it didn’t ask at all about disability.
Approximately 19% of Americans are disabled in some way. Furthermore, some research ((For example.)) indicates that disabled people are at greater risk of sexual assault. For these reasons, it is inexcusable for a nationally representative survey of sexual violence to not ask questions about disability.
Probably the demographics section of the survey should ask about immigrant status, as well. It certainly seems plausible that immigrant women may be especially vulnerable to sexual assault. However, language barriers ((The NVAW survey is conducted in English and in Spanish.)) — and whether or not immigrant women have phones — may be a barrier to effectively surveying immigrant women.
I’ll post more about the NVAW survey later this week.
Amp, I have followed this site for a while and I finally feel like there is something that I am qualified to comment on. Being in the field of survey research, it would be ideal to be able to study this every year — just like every study that we try to conduct. But, because there are limits on resources (primarily money), one is often forced to trade the number of respondents for the frequency of the study. One thing that I would be interested in seeing is a study that is conducted every year, but only covers a third of the country — that way you have nationally representative data for three-year intervals, but you can also measure the changes on an annual basis as well.
Another option would be to add questions about sexual assault to pre-existing national surveys — I think that while depth might be sacrificed in terms of the questions that can be asked, at least there would be frequent data on the prevalence of victimization and better demographic data on the individuals
Thanks for the blog!
Mike, thanks for your comment. I hope you’ll keep watching; I’m planning to post at least a couple more times on this subject, and I’d appreciate your responses.
That’s a really great idea.
While usually this might be a good idea, for a survey to make it safe for victims of intimate violence to report their experiences, it needs a different study design than any current pre-existing national survey I know of. For instance, the National Crime Victims Survey not only has the problem of setting things up as “crime,” which discourages victims from reporting assaults that they might not think of as crimes (such as if their husband has raped them); the surveys often take place without adequate privacy controls (i.e., the respondent’s spouse, parents or children might be in the room while the survey is taking place).
This could obviously be a problem. The NVAW study was especially designed to make it safe for respondents to answer questions; for other surveys to adequately do the same job the NVAW study did, would require not just adding on some new questions but also substantially redesigning how those other surveys are administered.
“the surveys often take place without adequate privacy controls”
Do you know what kind of privacy controls exist in the NVAW survey? I was surprised to hear it was phone-based, though I shouldn’t be… for some reason, I was thinking it would be like many anthropological studies, with the interviewer meeting the interviewee on neutral ground somewhere.
There are a couple of national surveys that are designed for different puposes that may be appropriate.
The one that immediately comes to mind is the National Survey for Family Growth which, in the description of the data says: “The Survey’s main function is to collect data on factors affecting pregnancy and women’s health in the United States. ” This is the one that I am most familiar with and I would argue that this topic fits squarely inside of woment’s health. The other advantage is that the survey has started interviewing men recently as well — it would be interesting to track perceptions of rape and sexual assault among men and women.
I think that there are others that might be appropriate — if I come across them, I will definitely pass them along.
That sounds great – I can’t wait =)
I like the group; I like the conclusions; I like the goals; I like the general methods; but I have some minor quibbles with some specifics of the study itself. I won’t post them (not interested in starting the you’re just a “rape apologist” shitstorm) but from a general study-pickiness standpoint I wish I could.
So a question for some folks (Amp, let me know if this is an unwanted derail)
What with all the logic/emotion stuff in the “moderation” threads, I’m confused. What now? Given my interpretation of some of the comments, what happens when something like this study gets posted (by Amp)?
The study is “pure” logic, so to speak, and the subject is obviously not. Can it even be openly discussed?
I can’t figure out a way for someone–especially a man–to criticize any aspect of the study based on a logical or scientific protest without getting lumped into the group of people (MRAs) who criticize the study based on its goals. Am I missing something? What?
Probably by making really clear up front, like Amp did, that you support the goals of the study, and that your criticisms are to make the study stronger and further those goals.
—Myca
Well Myca, let’s see if you’re right ;) My improvement suggestions were so long I wrote my own post on it:
http://moderatelyinsane.blogspot.com/2007/04/how-to-improve-nvaw-study.html
Sailorman, I think you are thinking that logic and emotion are mutually exclusive. I argue that they are not.
I personally think no one can make progress and make advances in policy without solid science (ok, so I know this isn’t actually true, but I WISH it was the case).
I ABSOLUTELY think it’s possible to critique the science and methodology of a study without being lumped as a MRA, rape apologist or general asshole.
Science and knowledge in general is not furthered without some dispute/critique about how to improve it. It’s common practice for instance at the end of an article in most top peer reviewed journals in sociology to admit the faults and limitations of the study as you see them and discuss possible avenues to further the research. Sometimes, it’s just part of the formula and is totally meaningless, other times it’s an admittance that data is flawed and though it’s not perfect, this is the best we could do at this point and perhaps in the future we can work through some of the problems.
The problem is not about critiquing the science – it’s when you do so solely to undermine the purpose or goals of the studyand do so in a dishonest way, or if you lack the skills and knowledge to actually capably do so. Bad science is bad science and no one gets the answers they want unless people are willing to say that and deal with it.
I’m heading over to read your critique now – I’ll be sure to let you know what I think ;)
I have to say, I think you’re setting up a false dichotomy here, Sailorman. There are a lot of reasons why one would object to arguments without being opposed to all logic or analysis. For instance, in your post about rewriting rape law, I objected to your tone and your positioning of yourself as an objective and (it seemed to me) all-knowledgeable writer.
I only point this out because I think it will be possible for people to criticize you, and to criticize the tone you take, without meaning that it’s “impossible for people to talk logically about this stuff without being called an MRA.” Once you’ve set that up as a proposition, it’s pretty chilling for anyone to criticize you, because if we do, we’re showing we can’t hack logic.
I know that probably isn’t your intent, but it’s what comes across.
oh no, mandolin, that wasn’t my intent at all–sorry if that’s what I suggested. Criticize away.
To use a different example: there exist logical arguments against, say, illegal immigration that are not based on racism, but it is extremely hard to make those arguments without getting called a racist, because many/most people who argue against II are in fact racially motivated.
Similarly, there exist arguments relating to rape that are not based on a desire to sustain the patriarchy. And some of those points disagree with some anti-rape activists. But since many/most people who get into arguments with anti-rape activists are in fact MRAs, it is exceedingly difficult to make those arguments, either, without getting lumped in that category.
I was merely worried. hope that’s more clear.
Sailorman, I think you are conflating 2 different things. It’s one thing to criticize research based on its merits it’s another thing to be involved in a heated discussion where numerous people feel offended/attacked or whatever and to keep going with the argument despite pleas to retrain your language/argument whatever.
I’m not saying the latter happens all the time or that it hasn’t even happened unfairly, but it’s not the same thing as the former.
Am I making any sense (for the record, I think you’re a good guy who in general is pro feminist and not a rape apologist but I’ll admit I’ve read things periodically that made me raise my eyebrows a bit – though I can’t quote anything off the top of my head – ya know how you just get a feel for certain frequent commenters?)?
Sorry – enough thread derailment though – I’ll stop now.
Pingback: Moderately Insane: How to improve the NVAW study.
Mandolin wrote:
Sorry it took me so long to respond to this; I meant to reply, but I got busy. You know how it goes…
Anyhow, I think it would have been hard (and prohibitively expensive) to conduct 16,000 interviews of randomly-selected subjects in the way you describe. :-)
Also, the survey designers believed that the random-digit dialing method gave subjects greater assurance of confidentiality; the interviewer can honestly tell the subject that they have no idea of who the subject is, and that after the interview is over there will be no records connecting the subject to their answers. (Subjects were coded using ID numbers, not their names or phone numbers; once the interview process was over, the last four digits of each subject’s telephone number were eliminated from all records). “The interviewer knows neither the respondent’s name, address, nor what the respondent looks like.”
At the start of interviews, subjects were given a toll-free number, and it was explained to them that if they needed to, they could hang up at any time and use the number to complete the interview later; subjects were explicitly given permission to hang up suddenly if they needed to. They could also schedule callback interviews at times they expected to have privacy. “In addition, interviewers were instructed to schedule a callback interview if they thought someone was listening to the interview on another line or was simply in the room with the respondent.”
Finally, local hot line numbers for counselors were kept on hand for both subjects and interviewers who were distressed by the interview and wanted to talk to someone.