Yet Another Koss Critic Can't Get His Facts Right

[Crossposted on “Alas” and on “TADA.” Non-feminists aren’t allowed to leave comments on this post on “Alas,” but anyone can leave a comment on “TADA.”]

Jim Hopper makes a good point about Mary Koss’ rape prevalence research: ((I haven’t read the rest of Hopper’s essay, so can’t say anything about it one way or the other.))

Major publications like The New York Times Magazine have given cover-story treatment to people who have minimal understanding of social science methodology, and apparently even less interest in the truth about rates of abuse and assault in our country. These people have claimed that researchers “make up” abuse that never happened by labeling subjects’ experiences as abusive even though the subjects might not.

This charge has been made against Mary Koss, an accomplished researcher who has conducted studies on prevalence rates of rape among college women (and has found that one in four have experienced rape or attempted rape since age 14). In constructing her questionnaire items, Koss made a good faith effort to use language that fit the legal definition of rape in the state where she lived when she conducted the research. Yet she has been accused of irresponsibly mislabeling her subjects’ experiences and exaggerating rates of rape. […]

One way that Koss has answered this critique is by referring to an analogous situation. I will paraphrase her argument. Imagine yourself questioning an alcoholic: Do you have more than six alcoholic drinks in one sitting several times a week? Yes. Do you often wake up with such a hangover that you can’t go to work? Yes. Have friends and family members repeatedly tried to help you stop drinking? Yes. Do you suffer from withdrawal symptoms when you stop drinking? Yes. Are you an alcoholic? No.

The point here is that good prevalence research must use behavioral descriptions to which definitions like “alcoholic” or “sexual abuse” may be applied. Researchers should not rely on people defining themselves as alcoholics or defining their sexual experiences as abusive. Such definitions can only be uninterpretable and unreliable.

The Damned Olde Man (TDOM) disagrees with Dr. Hopper. Like so many Koss critics, TDOM is unable to get his facts straight:

For instance, many of Koss’ questions began “Have you ever given in (to some sexual behavior) when you didn’t want to…” (1985, p. 167). In other words, have you ever consented to sex without consenting to sex? Koss states that this meets the legal definition for rape in her state.

Koss doesn’t state that — because those questions weren’t used by Koss to measure rape prevalence.

Two of the ten questions Koss used to measure sexual victimization used the “have you ever given in” wording TDOM quotes. But Koss was measuring a “scale” of sexual victimization — from “sexual contact” to “attempted rape” to “sexual coercion” to “rape.” Neither of the “have you ever given in” questions were used by Koss to measure rape. (( You can read all ten questions yourself (pdf file). The two questions TDOM quoted were questions #1 and #6. From the scoring instructions: “Women are classified as victims of rape if they answered “yes” to items 8, 9, or 10.” ))

This isn’t a minor point, hidden deep in the footnotes of Koss’ study. This is the core of how her survey works. If TDOM had read the study with any attention at all, he’d know this.

TDOM also writes:

Koss could have improved her study by … testing her questionnaire for reliability and validity, ensuring that it measured what it was designed to measure.

Gosh, that would have been a good idea. Which is why Koss did exactly that. She also referred to this several times in her study report, which I sincerely doubt TDOM bothered reading.

TDOM also claims that the “alcoholism” argument, made by Koss and paraphrased by Hopper, is a straw man argument.

The straw man is the comparison (originally made by Koss) of her conclusions to that of questioning an alcoholic about her drinking habits. However, the questions asked by Koss about the alcoholic are quantitative questions. The answers are very clear cut, concrete, and quantifiable. They ask for numbers and frequencies. The questions asked by Koss in her study regarding rape and attempted rape were qualitative questions and answers were very open to interpretation.

Of course, that isn’t what a straw man argument is; a straw man is when “the arguer is attempting to refute his opponent’s position, and in the context is required to do so, but instead attacks a position—the ‘straw man’—not held by his opponent.” (( For example, when TDOM claims that Mary Koss states that answering “yes’ to “Have you ever given in (to some sexual behavior) when you didn’t want to…” “meets the legal definition for rape,” that’s a straw man argument, because he’s falsely attributing a position to Koss and then attacking that position.)) The term TDOM should have used was “weak analogy fallacy.”

Terminology aside, TDOM is wrong on the substance. There were plenty of questions about alcoholism, in the example, that were open to interpretation. If I drink a bottle of wine from the bottle, was that one drink or six? What does it mean that I “can’t go to work” — if I could go to work if my life depended on it, but called in sick because of my pounding headache, does that qualify or not? What does it mean to “help me stop drinking” — if my husband says “don’t forget, you have to drive home” does that qualify as trying to help me stop?

Etc, etc. When dealing with human behavior, there is almost no such thing as a question with no ambiguity at all. Survey authors can, and should, work to make their questions clear and specific, and do validity tests to make sure subjects understand what they’re being asked. But Koss did all that. TDOM’s objections are unreasonable.

Finally, TDOM makes one reasonable argument — he suggests that question 8’s wording (“Have you had sexual intercourse when you didn’t want to because a man gave you alcohol or drugs?”) is too broad.

However, researchers Martin Schwartz and Molly Leggett tested the disputed question empirically back in 1999. ((Schwartz, Martin D. and Molly S. Leggett (1999), “Bad Dates or Emotional Trauma? The Aftermath of Campus Sexual Assault,” Violence Against Women 5(3): 251-271.)) They surveyed students with Koss’ survey, but substituted this question for Koss’ original alcohol and drugs question:

Have you engaged in sexual intercourse when you didn’t want to but were so intoxicated under the influence of alcohol or drugs that you could not stop it or object?

If TDOM is correct to believe that Koss’ question 8 created a significant “false yes” problem, then many fewer students would have answered “yes” to Schwartz and Leggett’s rewritten version. So what actually happened? Rewriting the question made no difference at all. 17% percent of students surveyed by Schwartz and Leggett were found to have been raped, a number basically identical to Koss’ 15%.

Finally, let’s not forget that Koss’ three most important findings — that around 10-15% of women have been victims of rape in their lifetime, that most rapes are never reported to police, and that rape is usually committed by someone known to the victim, not by a stranger — have been replicated again and again by other researchers.

• The NIJ/CDC “National Violence Against Women Study” found that 14.8% of American women experience a completed rape at some time in their lifetime. A typical rape-defining question was worded like this: “Has a man or boy ever made you have sex by using force or threatening to harm you or someone close to you? Just so there is no mistake, by sex we mean putting a penis in your vagina.”
• The Department of Justice’s Sexual Victimization of College Women study included a sub-study in which college women were asked about lifetime incidence of rape (most of the study asked about rape since the beginning of the school year,which isn’t directly comparable to Koss). 10% of the women interviewed reported having been raped at some point in their lifetime. Rape was defined as “unwanted completed penetration by force or the threat of force.”
• There’s also The National Women’s Study (NWS), a large-scale national study which found that 13% of American women have been raped in their lifetime.

At least a dozen more studies have confirmed Koss’ results. It is no longer possible to attribute Koss’ results to the particular wording Koss chose for her questions, because the same results have been found multiple times, regardless of wording.