The 2021 Estimates Aren't Suddenly Reliable

I want to address two points that have popped up this week regarding crime data. First, the 2021 estimates were not reliable and your understanding of whether “crime is rising or falling” should not rely on them.

The FBI publishes estimates each year. They’re estimates because not every agency reports data every year. Anybody suggesting the FBI’s figures are perfect is either naive about the way the data is collected and reported or doesn’t read my newsletter (or probably both). But the FBI’s figures, much like the BJS figures, paint a comprehensive picture of American crime trends over time even if the exact numbers can never be 100 percent correct.

The FBI’s 2021 crime estimates are the exception. The 2021 figures were not reliable when they were first published in 2022 and they are not reliable now.

Around 65 percent of the country was covered by a NIBRS agency when the switch was made in 2021. It’s why the FBI’s 2021 crime report came with very large error bars that were much maligned.

Participation for 2021 rose to around 73 percent when the 2022 estimates were published as a handful of non-NIBRS places like New Orleans added their 2021 totals when they reported 2022, and the 2021 participation rose again to 74.1 percent when 2023’s figures were reported. Some agencies have reported 2021 data to the FBI in later years, but NYPD and LAPD have not while Chicago has reported only a few months of 2021 data. Also not reporting in 2021: Phoenix, Miami-Dade, San Jose, San Francisco, Raleigh, Omaha, Oakland, Pittsburgh, and I could go on.

So the FBI’s estimates for 2021 still are not good which is why I wrote last month:

“Also note that the 2021 estimates were based on substantially lower participation due to the NIBRS switch, so I’m not comparing it to other years before or after in this analysis. The 2021 figures are included in graphs for the sake of continuity though they can be largely ignored.”

And in July 2023 I wrote:

“The national crime estimates for 2021 will likely always require an asterisk, but one skewed year of crime statistics is much easier to accept than three or four skewed years.”

Our assessment of whether crime is rising or falling really shouldn’t include the FBI’s 2021 estimates. People who — wrongly — argued for months about the FBI’s lower participation in 2022 and 2023 really shouldn’t be suddenly relying on 2021’s estimates as an accurate portrayal of crime nationally that year.

Second, yes, the FBI revised its 2022 figures when the 2023 figures were released. The FBI does this EVERY YEAR. Here’s the FBI in 2019 — see footnote 6 — noting that the 2018 figures were revised. Here’s the 2006 report — see footnote 3 — noting that the 2005 figures were revised. And here’s 1995’s report (see footnote 4 on page 58).

Calling these changes “stealth” simply displays one’s lack of awareness that revisions are frequently made to historical FBI estimates. The FBI doesn’t do press releases about the UCR’s methodology, it’s frustrating to those who deal regularly in crime data (at least it is to me!) but it doesn’t imply some malice or effort to hide the revisions.

The revisions released last month were actually considerably smaller than the revisions that the FBI made when it initially released 2022 data last year. I wrote about it in March when I looked at the 1967 violent crime rate rising between 1967 and 1975. I wrote:

“When the FBI published its Crime in the US report for 2022 for example, the US violent crime rate for 2003 was revised up from 475.8 per 100,000 to 501.8 in the table of crime rates in the US over the last 20 years. There were also smaller revisions to more recent years like 2019 (which was at 366.7 in 2019, was revised up to 368.9 in 2020, and then was revised down to 363.9 in 2022) which serve to accentuate highlight the need for caution when applying precision to this information.”

There were 16,740 murders in 2005 in the 2020 FBI report and 17,734 in the 2022 FBI report. The number of murders in 2020 rose from 21,570 when initially reported to 22,414 in 2022 and 22,510 in 2023.

All told, the FBI increased the number of murders that are estimated to have occurred between 2003 and 2020 by 6,404 relative to the 2020 report when it published the 2022 report. The 2023 report lowered the 2021 murder tally by 1,074 (which we’re going to ignore because of the NIBRS switch) and increased the 2022 tally by 625. The murder totals for 2016 through 2020 were also changed a bit, but nowhere near as substantially.

Note in the above table that the 2023 report covered the last 20 years (so not 2003) while the 2022 report covered 2021 and 2022 for the first times.

The revisions to 2023’s figures were large by historical standards though this isn’t the first time we’ve seen somewhat sizable revisions to previous years. There were 825 murders added to the FBI’s originally published tallies for 2006 through 2009 by 2010, for example.

Why were the FBI’s revisions in 2022 and 2023 so large? I don’t really know which is frustrating. I asked about it last year when the 2022 revisions were published but didn’t get any answers back. Large revisions without an explanation isn’t a great practice, but it also wasn’t overly concerning for me personally because it only serves to highlight how these crime estimates should always be taken with a grain of salt.

Do the 2022 and 2023 revisions majorly alter our understanding of national crime trends? In my opinion, not at all.

The 2021 estimate for violent crime right now is so bad that it’s actually outside of the 95 percent confidence interval that FBI and BJS put together when they were first released in 2022.

If our assessment of crime trends rightfully ignores 2021 then we can say that violent crime rose slightly in 2020 while murder rose at the fastest rate ever recorded that year. That was true in the 2020 release, it was true despite revisions in 2022, and it was true in 2023. Again, ignoring 2021, then we can say that the FBI’s 2023 estimates show a continued small decline in violent crime with a historically large decline in murder.

The number of violent crimes in 2022 rose from 1,232,428 when initially published to 1,256,721 after revision, an increase of around 24,000. If you assume a similarly historically large revision will be applied to the 2023 data in 2024 (which I doubt will occur, but certainly can’t rule out) then violent crime still had a small down in 2023 relative to 2022 and was largely in line with the historically low violent crime rates observed from 2013 through 2019.

A similar adjustment to 2023’s murder estimate would put murder down 8.7 percent last year compared to 2022 rather than the 11.6 percent that got reported with an even larger decline likely to be reported for 2024. That would be the second largest decline ever recorded behind a 9.1 percent decline in 1996.

Violent crime rose in 2020 and has fallen since then though the rise in violent crime has always been more muted than people assume. Murder, by contrast, rose a ton in 2020 and is falling a ton right now.

None of these revisions change our understanding of the nation’s crime trends. All the revisions do is highlight the need to avoid precision when talking about national crime trends wherever possible. They also highlight the importance of other data sources such as the Real-Time Crime Index, NORC’s Live Crime Tracker, the CDC, and the Gun Violence Archive which all point in the exact same direction.

More transparency into the revision process and a better understanding of why the revisions were so large in 2022 and 2023 would go a long way to easing the frustrations of people who regularly deal with crime data. But the existence of revisions is certainly nothing new, and the revisions made in 2022 and 2023 shouldn’t alter anyone’s perceptions of the nation’s crime trends.