[Trigger Warning! This post includes a sample question from a study which asked graphic questions about rape.]
(I know that some “Alas” readers looked at the title of this blog post and said “well, no kidding: isn’t that obvious?” But I still think it’s valuable to be able to point to proof. Perhaps this research will lead to the NCVS improving its design in the future.)
I just read an interesting new report about the design of surveys measuring rape prevalence.1
The author, Bonnie Fisher, conducted two almost identical surveys of women in college. (Both surveys were conducted in 1997). In the first survey, respondents were asked a series of 12 behaviorally specific sexual victimization screening questions, such as “Since school began in the Fall 1996, has anyone made you have oral sex by force or threat of harm? By oral sex, I mean did someone’s mouth or tongue make contact with your vagina or anus or did your mouth or tongue make contact with someone else’s genitals or anus.” These screening questions are built on the approach developed by Mary Koss in her influential rape prevalence studies in the 1980s, which have been oft-criticized by conservatives.
In the second survey, respondents were asked the sexual violence screening questions from the governments National Crime Victimization Survey (NCVS). The NCVS, along with the FBI’s compilations of police data, is the Federal government’s primary way of measuring crime in the US.
A bit of history: The NCVS used to be called the NCS (National Crime Survey), and didn’t ask about rape or sexual victimization at all; rapes were only counted if respondents brought up their rapes after being asked if they had been “attacked.” In the 1980s and 1990s, in response to feminist criticism, the government redesigned the NCS into the NCVS. Now, along with dozens of other crime-related questions, respondents are asked if they’ve experienced “Any rape, attempted rape, or other type of sexual attack.” They’re also asked:
Incidents involving forced or unwanted sexual acts are often difficult to talk about. Have you been forced or coerced to engage in unwanted sexual activity by—(a) Someone you didn’t know before—(b) A casual acquaintance—OR (c) Someone you know well?
This is a big improvement from not asking about rape or sexual violence at all — but it’s still pretty vague, compared to the behaviorally-specific questions.
In both surveys, each screening question “the respondent reports (answers “yes” to) is then followed up with a detailed incident report that contains multiple questions about what occurred during the incident. The responses to these questions are then used to classify the type of victimization that occurred.”2
But even though the two studies were mostly identical, apart from the questions used,3 the results were very different. Using the behaviorally specific screening questions, Fisher found that 19 out of 1000 female students had been raped since the beginning of the Fall term of school.4 In contrast, using the Federal government’s screening questions from the National Crime Victimization Survey, two out of 1000 female students had been raped since the beginning of the Fall term.
This makes it seem likely that the National Crime Victimization Survey, the USA’s primary method of measuring crime, severely underestimates how often rape occurs.
- Bonnie S. Fisher, “The Effects of Survey Question Wording on Rape Estimates: Evidence From a Quasi-Experimental Design,” in Violence Against Women Volume 15 Number 2 February 2009 pages 133-147. Link. [↩]
- Quoted from “Measuring the Sexual Victimization of Women: Evolution, Current Controversies, and Future Research”, by Bonnie S. Fisher and Francis T. Cullen. Link. [↩]
- There were two more significant differences between the two surveys. The first survey was described to respondents as a study of “unwanted sexual experiences that women may experience during college.” The second survey, like the National Crime Victimization Survey, was given a crime context, and described as a study of “criminal victimization that women many experience during college.”
The last difference is that the second survey, modeled after the NCVS, actually used a slightly broader definition of rape, including incidents of “psychological coercion as well as physical force.” [↩]
- The surveys were conducted in February through May 1997. [↩]
“Since school began in the Fall 1996, has anyone made you have oral sex by force or threat of harm? By oral sex, I mean did someone’s mouth or tongue make contact with your vagina or anus or did your mouth or tongue make contact with someone else’s genitals or anus.”
Please put a trigger warning before those words.
Good point. Trigger warning added. I’m really sorry if this post was a trigger for you.
Cue rape apologists who want to say forcing sex isn’t rape if the victim doesn’t describe it that way.
I went to the link for the 2nd reference but that only gives an abstract. What I’m looking for is the actual questions asked on each of the two surveys so that I can compare/contrast.
I haven’t been able to read it, as it is subscription-only. Do you know of a free link?
Based on your summary, I agree with the study’s apparent conclusions–does anyone these days seriously think that rape is not underreported?–but I still worry about the presentation.
Basically, it’s an issue of semantics. “Criminal” means “legal” means “a particular, extremely limited definition of ‘rape’ which fails to include an enormous amount of actions which are morally bad and which case a lot of problems for women.”
The legal definition of rape, as we all know, fails to cover many many things which are bad and which should stop. My own personal view is that the law covers only ~25% of the things which would make people believe they were raped. There are a variety of reasons for that discrepancy (some good and some bad) and a variety of ideas about how to change the laws (some good and some bad.) But in any case, if you look at the current status of the law, most folks are in agreement that “not illegal” is in no way equivalent to “not wrong.”
Outside the legal world, ‘rape’ can mean pretty much anything you define it to mean. It’s just a word.
So there are two very different spheres of discussion–legal and nonlegal–across which the word “rape” has an enormous diversity of meanings. This, unsurprisingly, leads to miscommunication, pointless arguments, and a lot of political fervor. It happens on both sides: you get folks marching into discussions on moral rape, telling people (incorrectly) that something “isn’t really rape because it’s not illegal.” And you get folks marching into discussions on legal rape, telling people (incorrectly) that they are rape apologists because they fail to treat legal and moral rape as equivalent.
If you are studying criminal action (which puts you in the limited arena of legal rape) then you need to use language which comports with the legal definition of rape. As a result, you will be using a definition of rape which severely conflicts with many women’s perceived experience. Your results may be useful in discussing conviction and reporting rates, but they will hugely underreport the number of women who feel that they have been raped, because the questions will exclude many of those women whose experiences do not match the legal definition.
If you are studying noncriminal (moral) rape, then you can capture information with no regard to whether or not the actions comport with the legal definition of rape. But while the data will be highly relevant w/r/t women’s experience, the data will not be generalizable to legal rape unless you do a really good job of study design. That is because the data will fail to exclude (or provide a method for identifying in analysis)many of those women whose experiences do not match the legal definition. If you use language which includes noncriminal action in the captured data, then you are getting valuable information but you are not getting data which would necessarily be relevant to (for example) rape convictions or reports of criminal rape behavior.
Koss’ study was attacked not because it was internally inaccurate, but because it (a) included some questions which were viewed as capturing moral rape; and (b) was commonly presented by third parties as being applicable to legal rape. It’s not an issue of conservatism so much as it is an issue of study design.
I don’t know if this study is different. Ihope so!
I don’t think the misunderstanding about what constitutes legal rape only runs one way, i.e., people sometimes think something isn’t legally prohibited when it actually is. Take for example Colorado’s sexual assault statute:
Circumstances (c) and (d) from the second list are exactly the sorts of things that will make people feel horrible and violated, but not necessarily aware that they have been raped.
For example, if a spouse or significant other demands sex and says if s/he doesn’t get it, s/he’s leaving, and this causes the victim to submit to sex that otherwise is against her will, that fits into (c), executable retaliation, yet how many people would be aware that they had been sexually assaulted rather than just manipulated against their will? How many people who are given drugs that will change their behavior — including alcohol — and have sex that they wouldn’t have had when sober (d – impairment) consider themselves to have been sexually assaulted within the legal definition?
Or consider the situations deemed sexual assault in the first list; how many people, upon discovering that their vaginal or anal examination had been for the physician’s personal amusement or gratification, would think of the criminal charge of sexual assault (as opposed to filing a civil malpractice claim or complaint to the state medical board)?
The law in some ways is broader than the common understanding of rape.
PG, good point. I don’t know how good that law is, though: I once read a similar law, and it was so qualified as to make some of the issues you raise moot. Have you checked the whole statutory section?
For example
may seem at first glance to mean “if the accused offers to buy a victim 3 shots of tequila, and she accepts and gets shitfaced as a result and “agrees” to have sex with the accused, then it is rape.”
So it may look like it is a legal codification of “drunk sex is rape sex.”
But I doubt it is. Other statutes I have read had qualifying language in other sections, which made that apparently-protective language essentially meaningless. IIRC, it is fairly common to define things in such a way that the substantial impairment only applies if it was done without knowledge or consent (e.g. a pill in someone’s drink;) if the victim consents to getting drunk then they do not get the benefit of that clause.
That is actually fairly standard in the law, as you know. But it goes to show how difficult it is to interpret rape statutes.
PG,
On my blog, I once did a detailed analysis of an Oregon rape law. The law seemed (on its face) to be fairly protective, but had so many qualifications that most of the protections ended up being worthless. Many of the qualifications were not at all obvious and were in different sections of the statutes.
I have since closed my blog to the public, but the post still exists. If you are interested in analyzing rape laws, you will probably find it a very helpful place to start. Let me know and I can either invite you or temporarily reopen it.
But generally speaking, they are rarely as protective as they seem.
Sailorman,
I’d be interested in seeing your analysis — please invite me through my hotmail account, pgofhsm.
I don’t have time to double-check this right now, but I think the link in footnote two will give you the exact wording of the questions. Keep in mind, however, that the screening questions are not the same as their definition of rape; respondents experiences are classified as rape based on their response to the incident report that a “yes’ to the screening questions leads them to, not based on the screening questions themselves.
PG: Not the most scholarly thing, but it’s at http://moderatelyinsane.blogspot.com/2006/10/analysis-of-oregon-rape-law.html; invite is en route.
Amp: Thx, reading that now.
That is a very good paper at link #2.
Sailorman’s distinction beween what one could morally regard as rape and what fits the legal definition of rape is worthwhile. It seems to me that a survey entitled “National Crime Victimization Survey” should include only crimes. Now, that’s not to say that certain acts currently not criminalized should not be considered as rape. I can see a reasonable basis for debate on that. But until they are, they shouldn’t be reported as such.
So the title to this post may well be accurate without there being any flaw in either Fisher’s survey or the NCVS. The issue may simply be a difference in what’s being measured. The first measures the incidence of acts that Fisher (or some authority she recognizes as valid) regards as rape, whereas the second measures the incidence of acts understood to be rape that are also defined as crimes.
Or not. Is it specifically contended that the NCVS underreports acts defined as rape under the law? I’m not clear on this from what I’m reading here.
This also makes me think that the difference between the two should be examined in detail, with an eye for stating “Here are acts appearing to be rape that are occurring with this incidence that are not considered as crimes.” That could be the basis for a debate on whether or not those acts should be legislated to be crimes.
The problem with the headline, especially if it appeared as such in the MSM and not in a blog of this nature where people are aware of the difference, is that people generally view rape=crime and would presume from reading it that the NVCS is underreporting a specific sort of crime. That is by no means demonstrated from what I can see here.
I think any acts that do not fit the legal definition of rape would still fit the definition of sexual assualt. So they’re still crimes, but not necessarily the crime the gov. survey was designed to measure. However, based on the incident report recording method, I assume the survey does seek to measure these crimes as well.
The line between legal and illegal is frequently moved. As such, we need data from both sides of the line. I think it makes perfect sense to capture data for actions which may not be illegal, so that the country can thus better decide what changes we should enact in our criminal code.
Geek-out time:
Ron, I highly suggest that you read the article linked in footnote 2.
I will give it my own summary spin here, to try to reduce many pages to few:
There is no “perfect measure” in social sciences, because all measures are subject to bias and error. So what do you do when you measure rape? you make a choice:
You can focus on reducing “false” classifications of rape* (i.e. incidents which were not in fact rape but which are classified as such by the study responses.) If you do so, you are likely to increase the number of “missed” classifications of rape (i.e. incidents which were rape, but which fail to meet the study criteria for classification as rape.)
The reverse is also true; you can focus on reducing “missed” classifications of rape. If you do so, you are likely to increase the number of “false” classifications of rape.
Often (though not always) the total number of errors is going to be higher if you try to reduce a particular result. So there is a third option, which is that you can try to design a study which contains the smallest total number of incorrect classifications for either side. Such a study would use a design located somewhere between the two poles described above.**
The paper in footnote #2 talks about that process, and the tradeoffs in “false” and “missed” reports inherent to each kind of study, and why one might or might not want to choose various study designs. It is really a well written paper.
Let’s put that into practice, using a question from the paper in question:
You could alter those questions so a “yes” answer would only be possible if the behavior was clearly legally rape. Then, nobody would be able to say that a respondent who said “yes” was not legally raped. but by so altering the response, you would make it more likely that someone who WAS legally raped would fail to answer yes. And so on.
There is also an ongoing objective/subjective debate which is addressed in the paper. The one sentence summary is simple: If a women does not classify a certain incident as rape or sexual assault, and if her report of the facts underlying the experience do support objectively classifying it as rape or sexual assault, how should the study treat the incident? As you might expect, there is a lot of debate about that as well. I’m on the “yes” side of that one, incidentally.
*However you happen to define it for that study.
**I am assuming for the moment that the errors are equivalent. The balance between type 1 and type 2 error is actually more complex than that, and should include an analysis of the relative cost of each type of error, and the relative cost of avoiding it, among other things.
Looks like a version of this reworked for publication in a journal.
http://www.ncjrs.gov/pdffiles1/nij/199705.pdf
That’s not true. You want to reduce the absolute inaccuracy of the estimate, not the total number of misclassifications. There are obviously situations where reducing misclassifications increases the inaccuracy of your estimate. That’s why the whole defence of Koss on the basis that behaviourally specific screening questions increase the accuracy of correct rape diagnosis is misguided.
james,
Could you explain this? I am not getting the statistical/ probability point you’re making. It’s not because you’re making it poorly; I just need this stuff explained in detail.
But it seems like there would be a difference between the accuracy of an estimate about a population, and the accuracy of correct rape diagnosis (since diagnoses are for individuals, not populations). That is, I can see how reducing misclassifications doesn’t necessarily increase the accuracy of one’s extrapolation from the surveyed group to the population as a whole, but I don’t see how reducing misclassifications doesn’t necessarily increase the accuracy of one’s diagnosing that the individuals surveyed have been raped.
That’s probably b/c people assume we live in a free country and only assume things are criminal that involve physical force, fraud, etc. there are bunch of laws on the books, like laws against sodomy, adultery, defaming religion etc that are probably unconstitutional but since no one ever enforces tham they never go to court.
would love to see this Colorado law in show up on scotus’s steps.
James, did you see my “**” note? Going after what you term “absolute inaccuracy” is simply another way of saying what I already said: in real life, the costs of type 1 and type 2 error may be different. When that is the case, you won’t go for a numerical reduction of error, but will go for a reduction in the total cost of error.
A good example is health care, where the consequences of a failure to treat can have hugely greater consequences than a treatment which was unwarranted. In those cases, the default becomes “treat.” IOW, even though it is known to create a large number of errors (people who are unnecessarily treated), the cost of those errors is smaller than the alternative.
Same in our criminal system: we consider it “worse” to convict an innocent than to free a perpetrator, so the system is biased towards the defendant. That said, I don’t want to side track into the question of whether, for the purposes of studies like this, it is “worse” to have Type 1 or 2 error. So i explained it using equivalent cost.
PG also raises a good point: the accuracy of your study and analysis (i.e. how well you reflect the objective truth about your actual sample) is really a separate issue from how good a job you do in extrapolating your sample to a large population with different characteristics.
Sailorman, you make good points. And certainly there are good and valid reasons for doing a survey of the incidence of acts that are a) not legally rape but b) may well still involve some kind of coercion or deception. But if as you discussed upthread the survey’s design does not permit an analysis that differentiates between acts that are legally defined as rape and those that are not, then the results of such a survey should not be taken as proof that a survey such as the NVCS is invalid.
You make some assumptions regarding how the structure of the study and the means of classifying acts as legal or not might affect the reporting of a given kind of act. I’m not going to tell you that those assumptions are invalid, but AFAIK those are assumptions, not proven facts. It seems to me that these two studies could be used to test those assumptions if the Fisher study (or a similar one) was designed in such a fashion so as to permit an analysis of the data that can differentiate among the different acts being reported. If it’s possible to take the data from the surveys and comprare the incidence of reports of act “x” in one survey vs. the incidence of reports of that same act in the other survey then at that point you can compare the methodologies, etc., and start to try to figure out what would account for the difference (if any!) between the two.
And yes, you’re right about the line sometimes being fuzzy. I have no personal knowledge of the fact, but I’d imagine that at least for certain acts (e.g., buying someone drinks and then having sex with them) the same act in the same juristiction being adjudicated by two different juries could get you two different findings. All I can say there is that you do the best you can to eliminate such effects and then let people know that there’s a certain type of sampling error involved.
Let me make it clear, folks, that the concept that the government has screwed up something is not a concept I have a whole lot of problem with. But if you want me to support making policy/legislative changes on that basis I’d like to see a better analysis that we seem to have right now.
I am happy to discuss why i am making the assumptions I am, but you have to read that paper first!
Cool.
As I explained on the “Christianity Is the Problem” thread, adultery is enforced when someone brings the charge. It has been enforced in Virginia and frequently is part of charges brought in the military against a soldier.
Laws against sodomy are unconstitutional under Lawrence v. Texas, but Lawrence specifies that laws that are put in place to prevent abuse of a state institution (such as marriage) are not affected by the ruling. Since adultery is a pretty significant cause of divorce, the state has a rational interest in prohibiting it.
And most people know we don’t live in a “free,” in the sense of libertarian, country. People know that they can’t have multiple spouses at once, for example.
Adultery offenses are dealt with pretty strictly in the military because of the risk of a breakdown of discipline, which is pretty important when people are depending on each other for their lives. If the person who’s watching your back is pissed because you’ve been boinking their spouse it could cause bad problems not only for the two people involved but also the overall mission, which can get a whole bunch of people killed.
RonF,
I don’t think so. These adultery cases rarely deal with soldiers’ boinking each other’s spouses — Kelly Flynn was rather exceptional, and got into trouble mainly because she first refused the order to desist in the affair, and then lied about it. More commonly, soldiers will be charged for cheating on their spouses with other soldiers (particularly with the increased presence of women in the armed forces serving alongside men, and which when done with a power differential also bring the charge of fraternization), or with civilians. The rationale is that such conduct “was to the prejudice of good order and discipline in the armed forces or was of a nature to bring discredit upon the armed forces.” It also applies to soldiers who are themselves single but committing adultery with a married soldier or civilian, as in Kelly Flynn’s case. The concern about “bringing discredit” is why Bush reformed the procedure so that severe punishment tends to be reserved now for higher ranking officers, whose misdeeds are more likely to garner media attention.
I am making it poorly. Imagine you’re a doctor, if you say 10 people have a disease when they don’t and wrongly say 10 people don’t have a disease when they do, then you’ve screwed up rather badly and have 20 very upset patients. Now imagine you’re an epidemiologist, and rather than being interested in diagnosing individuals you want to know the prevalence of a disease in a population. If you say 10 people have a disease when they don’t and wrongly say 10 people don’t have a disease when they do, then you’ve done your job and can go home happy. You’ve said 10 people have the disease and you’re right. The errors cancel each other out, they make no difference to your estimate.
If you’re diagnosing individuals you want to minimise misclassifications (false negatives plus false positives). If you’re estimating prevalence you want to balance misclassifications (get false negatives to equal false positives, i.e. minimise absolute inaccuracy).