The UCR vs NCVS Conundrum

Some thoughts on how to square disagreements between the nation's crime measures.

The United States of America is blessed with two systems to measure crime whereas many countries on this planet only have one. Those two systems are the Uniform Crime Report — run by the FBI — and the National Crime Victimization Survey — run by the Bureau of Justice Statistics. UCR — through the National Incident-Based Reporting System — records crimes that were reported to law enforcement while NCVS is a survey which allows for the estimation of crimes that may have occurred but were not successfully reported to police.

NCVS and UCR typically agree on the outline of national crime trends even if the specific ups and downs aren’t always in perfect alignment. Over the last few years, however, NCVS and UCR haven’t played quite as nicely as in previous years. This was particularly problematic in 2020 and 2022. Reported violent crime rose in 2020, including a historic surge in murder, according to UCR while NCVS said violent crime fell that year. Then, in 2022, UCR said violent crime fell slightly while NCVS suggested there was a large increase in violent crime relative to 2020 and 2021.

So how do we reconcile surging violent crime in NCVS with falling violent crime in 2022 and (likely) 2023 in UCR? The answer, in my humble opinion, is that by lowering our expectations of precision we can see pretty similar tales being told by both measures.

Before getting too deep into that, I think it’s important to talk about why NCVS is such a critical tool and some of the challenges with using NCVS — particularly when directly comparing it with UCR.

One of the most important contributions of NCVS in my opinion is the survey’s quantification of the landscape of unreported crimes. We know that crimes regularly don’t get reported, either because of a lack of trust, inability to report right away, long response times, or any number of other reasons. NCVS asks people if they reported incidents to the police and the results are always somewhat shocking if unsurprising.

The survey tells us that auto thefts are mostly reported while rape/sexual assaults are not. The share of crimes that get reported bumps around from year to year, but the overall contours have remained relatively stable over the last few years and quantifying the scale of unreported crime is extremely important to know.

NCVS also provides details about victimization patterns that don't always come through via UCR. This can be especially true with respect to hate crimes which can be badly underreported in official data. And NCVS helps provide validation to the contours of crime as described by UCR over the long term. In sum, NCVS is hugely important because we know that reported crime data can be so flawed.

But what do we do when UCR points to a slight decline in violent crime in 2022 and NCVS says violent crime surged by a lot?

There are several reasons why one should be careful about making an apples-to-apples comparison between the two crime measures and why one may want to be cautious when pointing to NCVS as a counter to the trends reported in UCR.

No Murders

The most obvious reason to avoid year-to-year direct comparisons is that UCR counts murder while NCVS does not. Murder victims cannot be surveyed, so the reason for the crime’s absence makes sense in NCVS, but it’s also the crime that comes with the highest societal cost and I’m guessing it’s usually the crime that people are thinking about when they think about the nation’s violent crime rate. Murder is also the one crime that probably has decently accurate — albeit imperfect — counts each year.

Year of Survey vs Year of Crime

It’s not widely recognized, but NCVS counts crimes by the year a respondent is interviewed rather than year of occurrence. NCVS asks survey respondents throughout the year whether they’ve been the victim of any crime in the last 6 full months. The methodology section of NCVS 2022 spells it out, noting that “Annual NCVS estimates are based on the number and characteristics of crimes that respondents experienced during the prior 6 months, excluding the month in which they were interviewed. Therefore, the 2022 survey covers crimes experienced from July 1, 2021 to November 30, 2022, with March 15, 2022 as the middle of the reference period. Crimes are classified by the year of the survey and not by the year of the crime.”

Let’s say you were robbed in August 2021 and for whatever reason it was not reported to the police. If you are interviewed for NCVS in January 2022 you would say you were the victim of a violent crime over the last six months, but that crime will count in 2022’s report. A change in crime that started in the latter half of 2021 might get picked up in the 2022 report. The reporting overlap is plainly visible in the expertly crafted timeline below.

This isn’t usually a big deal because crime trends tend to change slowly but it does makes it challenging to compare trends from one year to the next using precise language. The problem becomes exacerbated when crime trends are changing quickly (as they are now).

Only 12 and Up

One more reason to be cautious is that NCVS only contains information on people who are 12 years old or older while UCR has all crimes reported to the police regardless of the victim’s age. This isn’t likely to be a huge issue for crime estimates given that the vast majority of crime victims each year are going to be adults, but it’s still a gap that’s worth pointing out.

Confidence Intervals

NCVS is a survey which means that it comes with a margin of error. That margin of error has shrunk over the last few decades as methodologies improve, but it still produces a wide range of possible measurements within its confidence intervals. Using the BJS data tables, I calculated¹ that the nation’s violent crime rate (excluding simple assaults for a mostly apples-to-apples comparison of violent crime) rose 36 percent from 2021 in 2022 though when you include the 95 percent confidence interval then you get a range of possible outcomes of between 11 and 67 percent higher in 2022 than 2021 based on the lower and upper ranges of those two years.

2020 and 2021 Were Weird

You may have blocked 2020 and 2021 from your memory, but those were particularly tough years to survey people. NCVS 2020 notes that there was a change in survey methodology in mid to late 2020 due to the realities of COVID. I am by no means an expert in survey methodology, but it does leave the question of whether this impacted the lower figures reported in NCVS those years at least in my eyes.

It’s Not 2022 Anymore

And, finally, the way the survey is collected makes it impossible to report faster than its current frequency. As such, the 2022 data is the best we’ll have until September 2024 when the 2023 report will come out. But it’s no longer 2022. It’s not even 2023 anymore! In many ways, 2022’s violent crime trend isn’t particularly important relative to the direction implied in 2023’s preliminary reported crime trend sitting here in the spring of 2024. Reported crime data has many flaws, but much of it is often publicly available much faster making it a far superior source for understanding what is currently happening as opposed to what happened in 2022.

Reconciling NCVS and UCR

In my opinion, NCVS and UCR are telling the same story which is obscured by differences in methodology which add confusion to the story. Both measures tell us that the nation’s violent crime rate in 2022 was substantially lower than it was in the 1990s, largely in line than where it was over most of the last 15 years, and likely slightly higher than where it stood in 2019.

NCVS and UCR disagree mostly about what 2020 and 2021 looked like which — in my opinion — is largely irrelevant at this point. This is especially true given that NCVS doesn’t measure murder and the major change occurring in 2020 and 2021 was a historic surge in murder.

NCVS and UCR disagree from time to time about crime’s direction even though the two measures have been in strong agreement over the long term. This is to be expected given the methodological differences in how each metric works and the inherent challenges in tracking crime trends. Crime data is hard and can sometimes tell complicated stories which only serves to highlight the need to always caveat appropriately and use your estimative language when describing it.

1. I grabbed the percent of the population that were the victims of violent crimes minus simple assaults each year from Appendix Table 15 and multiplied that number by the population totals for each year from 1993 to 2022 (https://bjs.ojp.gov/bjs.ojp.gov/personpop.xlsx) using the estimated, upper, and lower bounds provided by BJS.