Remember The Survey In 'National Crime Victimization Survey'

Surveys aren't exact so don't treat them as such.

Did you know that there’s an election coming up in a few weeks? I know, I know, it snuck up on me too, but it’s true! The fun thing about elections is that they produce data in the form of polls that can provide a probabilistic understanding of what will happen in November when the voters cast their votes. The polls are just guesses though and they frequently come with crosstabs that show how different groups within the poll responded.

The New York Times released a survey of the presidential election in Arizona recently showing Donald Trump up 5 points on Kamala Harris. This survey is a bit of an outlier as most surveys to this point show a tighter race. Digging into the survey, however, shows the margin of error in Arizona is 4.4 percentage points for each candidate, so what the poll is really saying is that Donald Trump could be up nearly 14 points or he could be down 4 points.

Sometimes a poll will show your side losing when the rest of the polls show a different trend, and experts are pretty clear that it’s foolish to dig into the crosstabs and attempt to "fix” or “unskewer” the poll to show what you want.

Crime data can be similar at times.

This was true in 2021 when the FBI switched to NIBRS and participation plunged so much that the 2021 report carried large margins of error. And it’s always true with the National Crime Victimization Survey (NCVS) from the Bureau of Justice Statistics.

It has recently become popular to use the NCVS to show that *actually* violent crime was undeniably very much up in 2023, usually by comparing it to 2020. I’ve written previously about what NCVS does and doesn’t do well, and I don’t want to rehash those points here other than to point out that relying on NCVS alone to evaluate US crime trends requires one to completely ignore murder — the violent crime that has undergone the most dramatic change over the last five years.

What I do want to focus on for this post are the pitfalls of using NCVS to confidently assert year on year percent changes in crime. I also want to show how the further you dig into NCVS data to show X crime changed by Y percent between this and that year the more careful you have to be in your assertions.

NCVS is a survey just like that New York Times poll, and as a survey it carries a margin of error. Yes, it surveys over 220,000 people nationally each year, but that’s around 0.7 percent of the country’s population and the vast majority of those people are not the victim of a crime each year.

The rate of violent crime (excluding simple assault) in 2023 was between 7.5 and 10 per 1,000 meaning that between 0.75 and 1 of every 100 people they surveyed said they were the victim of a serious violent crime during the survey period.

Extrapolating that to the nation as a whole provides a whole range of possible outcomes. In other words, there's a uncertainty inherent in the survey which is captured in the confidence intervals provided by NCVS.

Relying solely on NCVS as an exact representation of national crime trends without acknowledging the margin of error is a complete misrepresentation of what the data says and how it was collected. And the problem gets worse the deeper you dig and the larger the margin of error becomes.

BJS has a handy website called N-DASH that enables quick and easy comparisons of all sorts of trends within NCVS. Violent crime (including simple assault) was up 37 percent in 2023 compared to 2020 and up 7 percent compared to 2019. But the confidence intervals mean that violent crime may have been up by just a small amount that barely registers as a change or it may have been up by a much huge amount. And there's no way to say for sure which trend was the right one.

The possibilities for the violent crime rate in 2023 compared to 2019, by contrast, stretch from a sizable decline to an even larger increase. Hence why BJS concluded that “While the 2023 rate was higher than those in 2020 and 2021, it was not statistically different from 5 years ago, in 2019.”

It would be inaccurate and inadvisable, therefore, to confidently assert that violent crime (excluding simple assault) was 55 percent higher in 2023 compared to 2020 without acknowledging the ocean of uncertainty that comes from the survey.

A more accurate statement that compares the differences between what NCVS and UCR reported would say that “BJS reported that the violent crime rate (excluding murder and simple assault) was most likely higher in 2023 than in 2020 while the FBI reported that the violent crime rate (including murder) was around 6 percent lower in 2023 than in 2020. Both BJS and FBI’s reports agree that violent crime was roughly in line with 2019 levels in 2023.”

That’s an accurate representation of what NCVS and UCR are telling us, but it’s unfortunately not a very salacious one for the op-ed page.

The confidence intervals gets bigger while the certainty shrinks the deeper you go into NCVS offenses. The rate of rape/sexual assault was up 42 percent in 2023 compared to 2020, but the lower and upper bounds clearly show that rape/sexual assault could have easily been lower in 2023 than in 2020. The rate of robbery was up 63 percent with a similar uncertainty overlap of the upper and lower bounds. The rate of aggravated assault was up 55 percent with a tiny difference between the 2020 upper bound and the 2023 lower bound.

The urban violent crime rate (excluding simple assault and murder), which former BJS director Jeffrey Anderson discussed in the above linked op-ed in the Wall Street Journal, was up 51 percent in 2023 compared to 2020 with range from a small decline to a huge increase. Urban violent crime was either down a little, up a lot, or somewhere in between. To say nothing of the analytic issue of ignoring murder when describing urban violent crime solely through NCVS.

On the other end of the spectrum, the number of violent victimizations involving a firearm where the victim was injured was down 38 percent in 2023 compared to 2020 — 56,973 in 2023 vs 92,141 in 2020. The rates get really small for some offense types, so I’m using the estimated numbers for this measure. Of course it wouldn’t be accurate to talk about a 38 percent decline in violent victimizations involving a firearm where the victim was injured because the range of possible outcomes is quite wide between an enormous decline and a sizeable increase.

Uncertainty in crime data is not unique to NCVS. The FBI’s report in 2021 also had issues which I’ve written about a number of times. It would be doing a disservice to simply ignore the uncertainty inherent in those issues because the report produced numbers which helps to prove the point I want to make. And not every crime gets reported to the police, so that adds questions as to how precise the FBI's reported figures are for some crimes each year.

When analyzing and communicating crime trends, therefore, it is important to acknowledge uncertainty where it exists. With that in mind, here is what we know — or at least feel highly confident in — about crime trends in the United States:

1. Murder rose at a historic rate in 2020 and it’s coming down at a historic rate in 2023 and 2024. We know this from the CDC, we know it from the Gun Violence Archive, we know it from the FBI, we know it from the Real-Time Crime Index.

   We also know that murder is the one crime that is most likely to be successfully reported by the police. Murder is the crime with the highest societal cost, it’s the crime that tends to make the headlines, and — anecdotally — it’s the crime I get asked the most about when doing interviews and presentations. And murder is not measured by the NCVS.

   Any discussion of US crime trends that ignores or pays lip service to the national murder trend is badly flawed, biased, or both.

2. We know that reported violent crime has fallen in the last few years since rising a bit in 2020, and reported violent crime in 2023 was nearly exactly at the 50 year low reached in 2014. We also know that not every crime gets reported to the police and that the violent crime rate in 2023 was not statistically different from where it has been for most of the last 15 years.

   Not every agency reports data to the FBI every year and not every agency that reports data to the FBI reports in a given year reports 12 full months. That’s why the FBI has an offense estimation procedure spelled out here.

3. We know from both UCR and NCVS that violent crime has fallen a long way nationally since the 1990s regardless of whether it is somewhat higher or lower than where it was in 2020. We also know — shout out to a recent tweet from Justin Wolfers on this — that NCVS has a tendency to bounce around a lot in terms of percent change from year to year.

4. We know that all of our crime reporting system have flaws which can lead to both undercounting and overcounting of offenses. It’s a major theme of this newsletter and one that I’ve discussed many times.

5. We know that BJS reported a dip in violent crime — sans murder — in 2020 and 2021 though the exact size of that dip isn’t entirely clear while FBI reported a slight increase in 2020 — including murder — with unreliable data from the FBI in 2021. We don’t know whether these are major or minor differences and we’ll probably never know that.

Avoiding certainty where it is not warranted is another major theme of this newsletter, and hopefully it's clear from the last few hundred words that it’s a critical aspect of using both FBI and BJS data. Those who want to evaluate crime trends should always strive to avoid confident assertions about things that carry inherent uncertainty, and they should clearly express that uncertainty as best they can (it’s not always easy!).